You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/07/06 04:52:12 UTC
[JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.7.0_60) - Build # 10615
- Failure!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/10615/
Java: 32bit/jdk1.7.0_60 -client -XX:+UseParallelGC
1 tests failed.
REGRESSION: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch
Error Message:
shard1 is not consistent. Got 143 from https://127.0.0.1:36610/xvv/collection1lastClient and got 142 from https://127.0.0.1:33168/xvv/collection1
Stack Trace:
java.lang.AssertionError: shard1 is not consistent. Got 143 from https://127.0.0.1:36610/xvv/collection1lastClient and got 142 from https://127.0.0.1:33168/xvv/collection1
at __randomizedtesting.SeedInfo.seed([3C1FB6EADDDDFE71:BDF938F2AA829E4D]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1139)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1118)
at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:150)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at java.lang.Thread.run(Thread.java:745)
Build Log:
[...truncated 11295 lines...]
[junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
[junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001/init-core-data-001
[junit4] 2> 298061 T1569 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
[junit4] 2> 298061 T1569 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /xvv/
[junit4] 2> 298069 T1569 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> 298070 T1569 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 298070 T1570 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 298170 T1569 oasc.ZkTestServer.run start zk server on port:51853
[junit4] 2> 298171 T1569 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 298174 T1576 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@27b4c2 name:ZooKeeperConnection Watcher:127.0.0.1:51853 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 298174 T1569 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 298174 T1569 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 298177 T1569 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 298178 T1578 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d9bb83 name:ZooKeeperConnection Watcher:127.0.0.1:51853/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 298178 T1569 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 298179 T1569 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 298181 T1569 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 298182 T1569 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 298183 T1569 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 298185 T1569 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 298185 T1569 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 298188 T1569 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
[junit4] 2> 298189 T1569 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 298191 T1569 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 298191 T1569 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 298193 T1569 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 298193 T1569 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 298195 T1569 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 298196 T1569 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 298197 T1569 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 298198 T1569 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 298201 T1569 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 298202 T1569 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
[junit4] 2> 298204 T1569 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 298205 T1569 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 298206 T1569 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 298207 T1569 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 298209 T1569 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 298209 T1569 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 298211 T1569 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 298212 T1569 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 298214 T1569 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 298215 T1580 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@36f065 name:ZooKeeperConnection Watcher:127.0.0.1:51853/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 298216 T1569 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 298469 T1569 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 298471 T1569 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 298475 T1569 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:37674
[junit4] 2> 298476 T1569 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 298476 T1569 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 298477 T1569 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001/tempDir-002
[junit4] 2> 298477 T1569 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001/tempDir-002/'
[junit4] 2> 298490 T1569 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001/tempDir-002/solr.xml
[junit4] 2> 298518 T1569 oasc.CoreContainer.<init> New CoreContainer 13911458
[junit4] 2> 298518 T1569 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001/tempDir-002/]
[junit4] 2> 298519 T1569 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 298520 T1569 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 298520 T1569 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 298520 T1569 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 298520 T1569 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 298521 T1569 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 298521 T1569 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 298521 T1569 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 298522 T1569 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 298524 T1569 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 298524 T1569 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 298524 T1569 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 298524 T1569 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51853/solr
[junit4] 2> 298525 T1569 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 298526 T1569 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 298527 T1591 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e316e1 name:ZooKeeperConnection Watcher:127.0.0.1:51853 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 298527 T1569 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 298531 T1569 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 298533 T1593 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@180bda9 name:ZooKeeperConnection Watcher:127.0.0.1:51853/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 298533 T1569 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 298534 T1569 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 298536 T1569 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 298538 T1569 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 298540 T1569 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 298541 T1569 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 298543 T1569 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 298544 T1569 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37674_xvv
[junit4] 2> 298545 T1569 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:37674_xvv
[junit4] 2> 298546 T1569 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 298547 T1569 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 298556 T1569 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:37674_xvv
[junit4] 2> 298556 T1569 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 298558 T1569 oasc.Overseer.start Overseer (id=92052755457441796-127.0.0.1:37674_xvv-n_0000000000) starting
[junit4] 2> 298563 T1569 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 298569 T1595 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 298569 T1569 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 298571 T1569 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 298572 T1569 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 298573 T1594 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 298576 T1596 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
[junit4] 2> 298576 T1596 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 298577 T1593 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 298577 T1596 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 298578 T1594 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 298578 T1594 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:37674/xvv",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:37674_xvv",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":null}
[junit4] 2> 298578 T1594 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
[junit4] 2> 298579 T1594 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 298580 T1593 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 298581 T1593 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 299577 T1596 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 299578 T1596 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 299578 T1596 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 299579 T1596 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 299579 T1596 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 299580 T1596 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001/tempDir-002/collection1/'
[junit4] 2> 299580 T1596 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001/tempDir-002/collection1/lib/.svn/' to classloader
[junit4] 2> 299581 T1596 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001/tempDir-002/collection1/lib/classes/' to classloader
[junit4] 2> 299581 T1596 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001/tempDir-002/collection1/lib/README' to classloader
[junit4] 2> 299612 T1596 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_10
[junit4] 2> 299641 T1596 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 299642 T1596 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 299649 T1596 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 299890 T1596 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 299892 T1596 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 299893 T1596 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 299896 T1596 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 299909 T1596 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
[junit4] 2> 299909 T1596 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 299910 T1596 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001/tempDir-002/collection1/, dataDir=./solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001/tempDir-001/control/data/
[junit4] 2> 299910 T1596 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@bcda51
[junit4] 2> 299910 T1596 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001/tempDir-001/control/data
[junit4] 2> 299911 T1596 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001/tempDir-001/control/data/index/
[junit4] 2> 299911 T1596 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
[junit4] 2> 299911 T1596 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001/tempDir-001/control/data/index
[junit4] 2> 299912 T1596 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=39, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5821204580873326]
[junit4] 2> 299916 T1596 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 299916 T1596 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 299918 T1596 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 299919 T1596 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 299919 T1596 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 299919 T1596 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 299919 T1596 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 299920 T1596 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 299920 T1596 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 299920 T1596 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 299920 T1596 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 299921 T1596 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 299921 T1596 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 299921 T1596 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 299922 T1596 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 299922 T1596 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 299922 T1596 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 299923 T1596 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 299925 T1596 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 299927 T1596 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 299927 T1596 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 299928 T1596 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=15, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
[junit4] 2> 299928 T1596 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 299929 T1596 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 299929 T1596 oass.SolrIndexSearcher.<init> Opening Searcher@15ab026[collection1] main
[junit4] 2> 299929 T1596 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 299930 T1596 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 299930 T1596 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 299931 T1596 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 299931 T1596 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 299931 T1596 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 299931 T1596 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 299932 T1596 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 299932 T1596 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
[junit4] 2> 299932 T1596 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 299934 T1597 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@15ab026[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 299935 T1596 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 299936 T1600 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:37674/xvv collection:control_collection shard:shard1
[junit4] 2> 299936 T1569 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
[junit4] 2> 299937 T1569 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 299939 T1600 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 299939 T1569 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 299942 T1602 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a354d8 name:ZooKeeperConnection Watcher:127.0.0.1:51853/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 299942 T1569 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 299943 T1569 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 299945 T1600 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 299945 T1569 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 299946 T1600 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 299946 T1600 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 299947 T1593 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> ASYNC NEW_CORE C175 name=collection1 org.apache.solr.core.SolrCore@5181ed url=https://127.0.0.1:37674/xvv/collection1 node=127.0.0.1:37674_xvv C175_STATE=coll:control_collection core:collection1 props:{state=down, base_url=https://127.0.0.1:37674/xvv, core=collection1, node_name=127.0.0.1:37674_xvv}
[junit4] 2> 299947 T1600 C175 P37674 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:37674/xvv/collection1/
[junit4] 2> 299947 T1600 C175 P37674 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 299948 T1594 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 299948 T1600 C175 P37674 oasc.SyncStrategy.syncToMe https://127.0.0.1:37674/xvv/collection1/ has no replicas
[junit4] 2> 299948 T1600 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:37674/xvv/collection1/ shard1
[junit4] 2> 299949 T1600 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 299950 T1593 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 299953 T1593 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 299955 T1593 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 300056 T1602 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 300056 T1593 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 300104 T1600 oasc.ZkController.register We are https://127.0.0.1:37674/xvv/collection1/ and leader is https://127.0.0.1:37674/xvv/collection1/
[junit4] 2> 300104 T1600 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:37674/xvv
[junit4] 2> 300104 T1600 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 300104 T1600 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
[junit4] 2> 300105 T1600 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 300107 T1593 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 300107 T1593 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 300107 T1593 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 300107 T1600 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 300108 T1594 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 300109 T1594 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"https://127.0.0.1:37674/xvv",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:37674_xvv",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 300110 T1593 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 300210 T1569 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
[junit4] 2> 300211 T1569 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 300212 T1593 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 300212 T1602 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 300214 T1569 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 300215 T1569 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:39876
[junit4] 2> 300217 T1569 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 300217 T1569 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 300217 T1569 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001/tempDir-003
[junit4] 2> 300218 T1569 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001/tempDir-003/'
[junit4] 2> 300231 T1569 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001/tempDir-003/solr.xml
[junit4] 2> 300266 T1569 oasc.CoreContainer.<init> New CoreContainer 26554198
[junit4] 2> 300266 T1569 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001/tempDir-003/]
[junit4] 2> 300267 T1569 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 300267 T1569 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 300268 T1569 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 300268 T1569 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 300268 T1569 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 300268 T1569 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 300269 T1569 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 300269 T1569 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 300269 T1569 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 300271 T1569 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 300272 T1569 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 300272 T1569 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 300272 T1569 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51853/solr
[junit4] 2> 300273 T1569 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 300273 T1569 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 300274 T1613 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@11c72b2 name:ZooKeeperConnection Watcher:127.0.0.1:51853 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 300275 T1569 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 300276 T1569 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 300277 T1615 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1103969 name:ZooKeeperConnection Watcher:127.0.0.1:51853/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 300278 T1569 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 300282 T1569 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 301284 T1569 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39876_xvv
[junit4] 2> 301285 T1569 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:39876_xvv
[junit4] 2> 301288 T1602 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 301288 T1593 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 301288 T1615 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 301293 T1616 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 301293 T1616 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 301294 T1593 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 301294 T1616 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 301294 T1593 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 301294 T1593 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 301296 T1594 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 301296 T1594 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:39876/xvv",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:39876_xvv",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":null}
[junit4] 2> 301296 T1594 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
[junit4] 2> 301297 T1594 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 301302 T1593 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 301405 T1615 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 301405 T1602 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 301405 T1593 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 302294 T1616 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 302295 T1616 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 302295 T1616 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 302296 T1616 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 302296 T1616 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 302296 T1616 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001/tempDir-003/collection1/'
[junit4] 2> 302297 T1616 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001/tempDir-003/collection1/lib/.svn/' to classloader
[junit4] 2> 302297 T1616 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001/tempDir-003/collection1/lib/classes/' to classloader
[junit4] 2> 302298 T1616 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001/tempDir-003/collection1/lib/README' to classloader
[junit4] 2> 302329 T1616 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_10
[junit4] 2> 302358 T1616 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 302359 T1616 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 302366 T1616 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 302614 T1616 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 302617 T1616 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 302618 T1616 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 302622 T1616 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 302637 T1616 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 302637 T1616 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 302638 T1616 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001/tempDir-003/collection1/, dataDir=./solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001/tempDir-001/jetty1/
[junit4] 2> 302638 T1616 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@bcda51
[junit4] 2> 302638 T1616 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001/tempDir-001/jetty1
[junit4] 2> 302639 T1616 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001/tempDir-001/jetty1/index/
[junit4] 2> 302639 T1616 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
[junit4] 2> 302639 T1616 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001/tempDir-001/jetty1/index
[junit4] 2> 302640 T1616 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=39, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5821204580873326]
[junit4] 2> 302983 T1616 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 302984 T1616 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 302987 T1616 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 302987 T1616 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 302987 T1616 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 302987 T1616 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 302988 T1616 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 302988 T1616 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 302988 T1616 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 302988 T1616 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 302989 T1616 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 302989 T1616 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 302990 T1616 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 302990 T1616 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 302990 T1616 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 302991 T1616 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 302991 T1616 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 302992 T1616 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 302994 T1616 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 302996 T1616 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 302996 T1616 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 302997 T1616 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=15, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
[junit4] 2> 302998 T1616 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 302998 T1616 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 302998 T1616 oass.SolrIndexSearcher.<init> Opening Searcher@1679110[collection1] main
[junit4] 2> 302999 T1616 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 303000 T1616 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 303000 T1616 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 303000 T1616 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 303000 T1616 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 303001 T1616 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 303001 T1616 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 303001 T1616 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 303002 T1616 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
[junit4] 2> 303002 T1616 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 303006 T1617 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1679110[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 303008 T1616 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 303009 T1620 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:39876/xvv collection:collection1 shard:shard2
[junit4] 2> 303009 T1569 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
[junit4] 2> 303009 T1620 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 303009 T1569 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 303013 T1620 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
[junit4] 2> 303015 T1593 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 303015 T1593 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 303015 T1620 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 303015 T1620 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C176 name=collection1 org.apache.solr.core.SolrCore@10d6359 url=https://127.0.0.1:39876/xvv/collection1 node=127.0.0.1:39876_xvv C176_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:39876/xvv, core=collection1, node_name=127.0.0.1:39876_xvv}
[junit4] 2> 303015 T1620 C176 P39876 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:39876/xvv/collection1/
[junit4] 2> 303016 T1620 C176 P39876 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 303016 T1620 C176 P39876 oasc.SyncStrategy.syncToMe https://127.0.0.1:39876/xvv/collection1/ has no replicas
[junit4] 2> 303016 T1620 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:39876/xvv/collection1/ shard2
[junit4] 2> 303016 T1620 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4] 2> 303015 T1593 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 303020 T1594 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 303021 T1593 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 303024 T1615 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 303024 T1602 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 303024 T1593 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 303027 T1594 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 303029 T1593 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 303131 T1602 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 303131 T1593 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 303131 T1615 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 303175 T1620 oasc.ZkController.register We are https://127.0.0.1:39876/xvv/collection1/ and leader is https://127.0.0.1:39876/xvv/collection1/
[junit4] 2> 303175 T1620 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:39876/xvv
[junit4] 2> 303175 T1620 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 303176 T1620 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 303176 T1620 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 303177 T1593 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 303178 T1593 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 303178 T1593 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 303178 T1620 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 303179 T1594 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 303180 T1594 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"https://127.0.0.1:39876/xvv",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:39876_xvv",
[junit4] 2> "shard":"shard2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 303181 T1593 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 303288 T1593 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 303288 T1615 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 303288 T1602 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 303293 T1569 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
[junit4] 2> 303293 T1569 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 303296 T1569 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 303297 T1569 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:55674
[junit4] 2> 303300 T1569 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 303300 T1569 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 303301 T1569 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001/tempDir-004
[junit4] 2> 303301 T1569 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001/tempDir-004/'
[junit4] 2> 303322 T1569 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001/tempDir-004/solr.xml
[junit4] 2> 303367 T1569 oasc.CoreContainer.<init> New CoreContainer 25367186
[junit4] 2> 303368 T1569 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001/tempDir-004/]
[junit4] 2> 303369 T1569 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 303369 T1569 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 303370 T1569 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 303370 T1569 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 303371 T1569 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 303371 T1569 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 303372 T1569 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 303372 T1569 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 303372 T1569 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 303375 T1569 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 303376 T1569 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 303376 T1569 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 303377 T1569 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51853/solr
[junit4] 2> 303377 T1569 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 303378 T1569 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 303380 T1631 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5afdc4 name:ZooKeeperConnection Watcher:127.0.0.1:51853 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 303380 T1569 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 303382 T1569 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 303383 T1633 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@19ca35c name:ZooKeeperConnection Watcher:127.0.0.1:51853/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 303383 T1569 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 303388 T1569 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 304390 T1569 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55674_xvv
[junit4] 2> 304392 T1569 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:55674_xvv
[junit4] 2> 304394 T1602 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 304394 T1593 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 304394 T1633 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 304394 T1615 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 304401 T1634 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 304401 T1634 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 304402 T1593 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 304402 T1634 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 304402 T1593 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 304402 T1593 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 304403 T1594 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 304404 T1594 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:55674/xvv",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:55674_xvv",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":null}
[junit4] 2> 304404 T1594 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 304404 T1594 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 304406 T1593 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 304507 T1633 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 304507 T1593 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 304507 T1602 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 304507 T1615 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 305402 T1634 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 305403 T1634 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 305403 T1634 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 305404 T1634 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 305405 T1634 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 305405 T1634 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001/tempDir-004/collection1/'
[junit4] 2> 305405 T1634 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001/tempDir-004/collection1/lib/.svn/' to classloader
[junit4] 2> 305406 T1634 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001/tempDir-004/collection1/lib/classes/' to classloader
[junit4] 2> 305406 T1634 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001/tempDir-004/collection1/lib/README' to classloader
[junit4] 2> 305448 T1634 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_10
[junit4] 2> 305487 T1634 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 305488 T1634 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 305496 T1634 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 305753 T1634 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 305755 T1634 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 305756 T1634 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 305759 T1634 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 305773 T1634 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 305774 T1634 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 305774 T1634 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001/tempDir-004/collection1/, dataDir=./solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001/tempDir-001/jetty2/
[junit4] 2> 305774 T1634 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@bcda51
[junit4] 2> 305775 T1634 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001/tempDir-001/jetty2
[junit4] 2> 305775 T1634 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001/tempDir-001/jetty2/index/
[junit4] 2> 305775 T1634 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
[junit4] 2> 305776 T1634 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001/tempDir-001/jetty2/index
[junit4] 2> 305776 T1634 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=39, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5821204580873326]
[junit4] 2> 305782 T1634 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 305782 T1634 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 305785 T1634 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 305785 T1634 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 305785 T1634 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 305786 T1634 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 305786 T1634 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 305786 T1634 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 305786 T1634 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 305787 T1634 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 305787 T1634 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 305787 T1634 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 305788 T1634 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 305788 T1634 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 305788 T1634 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 305789 T1634 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 305789 T1634 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 305790 T1634 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 305792 T1634 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 305794 T1634 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 305795 T1634 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 305796 T1634 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=15, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
[junit4] 2> 305796 T1634 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 305797 T1634 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 305797 T1634 oass.SolrIndexSearcher.<init> Opening Searcher@104dd4d[collection1] main
[junit4] 2> 305797 T1634 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 305798 T1634 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 305798 T1634 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 305798 T1634 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 305799 T1634 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 305799 T1634 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 305799 T1634 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 305800 T1634 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 305800 T1634 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
[junit4] 2> 305800 T1634 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 305804 T1635 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@104dd4d[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 305806 T1634 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 305806 T1638 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:55674/xvv collection:collection1 shard:shard1
[junit4] 2> 305807 T1569 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
[junit4] 2> 305807 T1569 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 305807 T1638 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 305811 T1638 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 305813 T1593 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 305813 T1593 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 305813 T1593 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 305813 T1638 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 305814 T1638 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 305814 T1594 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> ASYNC NEW_CORE C177 name=collection1 org.apache.solr.core.SolrCore@12791c8 url=https://127.0.0.1:55674/xvv/collection1 node=127.0.0.1:55674_xvv C177_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:55674/xvv, core=collection1, node_name=127.0.0.1:55674_xvv}
[junit4] 2> 305814 T1638 C177 P55674 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:55674/xvv/collection1/
[junit4] 2> 305815 T1638 C177 P55674 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 305815 T1638 C177 P55674 oasc.SyncStrategy.syncToMe https://127.0.0.1:55674/xvv/collection1/ has no replicas
[junit4] 2> 305815 T1638 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:55674/xvv/collection1/ shard1
[junit4] 2> 305815 T1638 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 305816 T1593 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 305818 T1593 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 305823 T1593 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 305924 T1593 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 305924 T1602 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 305924 T1633 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 305924 T1615 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 305969 T1638 oasc.ZkController.register We are https://127.0.0.1:55674/xvv/collection1/ and leader is https://127.0.0.1:55674/xvv/collection1/
[junit4] 2> 305970 T1638 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:55674/xvv
[junit4] 2> 305970 T1638 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 305970 T1638 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 305970 T1638 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 305973 T1593 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 305973 T1593 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 305973 T1593 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 305974 T1638 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 305974 T1594 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 305975 T1594 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"https://127.0.0.1:55674/xvv",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:55674_xvv",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node2"}
[junit4] 2> 305977 T1593 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 306081 T1593 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 306082 T1633 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 306082 T1602 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 306082 T1615 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 306118 T1569 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
[junit4] 2> 306119 T1569 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 306121 T1569 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 306123 T1569 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:50153
[junit4] 2> 306125 T1569 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 306125 T1569 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 306125 T1569 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001/tempDir-005
[junit4] 2> 306126 T1569 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001/tempDir-005/'
[junit4] 2> 306139 T1569 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001/tempDir-005/solr.xml
[junit4] 2> 306167 T1569 oasc.CoreContainer.<init> New CoreContainer 30804075
[junit4] 2> 306168 T1569 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001/tempDir-005/]
[junit4] 2> 306169 T1569 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 306169 T1569 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 306169 T1569 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 306170 T1569 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 306170 T1569 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 306170 T1569 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 306170 T1569 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 306171 T1569 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 306171 T1569 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 306173 T1569 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 306173 T1569 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 306174 T1569 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 306174 T1569 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51853/solr
[junit4] 2> 306174 T1569 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 306175 T1569 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 306176 T1649 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e675d0 name:ZooKeeperConnection Watcher:127.0.0.1:51853 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 306177 T1569 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 306178 T1569 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 306179 T1651 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1aed934 name:ZooKeeperConnection Watcher:127.0.0.1:51853/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 306180 T1569 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 306184 T1569 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 307186 T1569 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50153_xvv
[junit4] 2> 307187 T1569 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50153_xvv
[junit4] 2> 307189 T1602 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 307190 T1651 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 307190 T1633 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 307190 T1615 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 307190 T1593 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 307197 T1652 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 307197 T1652 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 307198 T1593 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 307199 T1593 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 307198 T1652 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 307199 T1593 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 307200 T1594 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 307200 T1594 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:50153/xvv",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:50153_xvv",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":null}
[junit4] 2> 307201 T1594 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 307201 T1594 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
[junit4] 2> 307202 T1593 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 307306 T1602 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - up
[...truncated too long message...]
iter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4] 2> 372049 T2082 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4] 2> 372049 T2082 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4] 2> 372049 T2082 oasc.CachingDirectoryFactory.close Closing StandardDirectoryFactory - 2 directories currently being tracked
[junit4] 2> 372050 T2082 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001/tempDir-001/jetty11/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001/tempDir-001/jetty11/index;done=false>>]
[junit4] 2> 372050 T2082 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001/tempDir-001/jetty11/index
[junit4] 2> 372050 T2082 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001/tempDir-001/jetty11 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001/tempDir-001/jetty11;done=false>>]
[junit4] 2> 372050 T2082 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001/tempDir-001/jetty11
[junit4] 2> 372050 T2082 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> ASYNC NEW_CORE C650 name=collection1 org.apache.solr.core.SolrCore@17c5969 url=https://127.0.0.1:43423/xvv/collection1 node=127.0.0.1:43423_xvv C650_STATE=coll:collection1 core:collection1 props:{state=active, base_url=https://127.0.0.1:43423/xvv, core=collection1, node_name=127.0.0.1:43423_xvv}
[junit4] 2> 372123 T2029 C650 P43423 oasc.SyncStrategy.sync WARN Closed, skipping sync up.
[junit4] 2> 372124 T2029 oasc.ShardLeaderElectionContext.rejoinLeaderElection Not rejoining election because CoreContainer is shutdown
[junit4] 2> 372124 T2029 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@17c5969
[junit4] 2> 372125 T2029 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=2,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=72,cumulative_deletesById=52,cumulative_deletesByQuery=0,cumulative_errors=0,transaction_logs_total_size=7905,transaction_logs_total_number=2}
[junit4] 2> 372126 T2029 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4] 2> 372126 T2029 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4] 2> 372126 T2029 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4] 2> 372127 T2029 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4] 2> 372127 T2029 oasc.CachingDirectoryFactory.close Closing StandardDirectoryFactory - 2 directories currently being tracked
[junit4] 2> 372127 T2029 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001/tempDir-001/jetty12 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001/tempDir-001/jetty12;done=false>>]
[junit4] 2> 372127 T2029 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001/tempDir-001/jetty12
[junit4] 2> 372128 T2029 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001/tempDir-001/jetty12/index.20140705182714069 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001/tempDir-001/jetty12/index.20140705182714069;done=false>>]
[junit4] 2> 372128 T2029 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001/tempDir-001/jetty12/index.20140705182714069
[junit4] 2> 372128 T2029 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 372128 T2029 oasc.LeaderElector$ElectionWatcher.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:358)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
[junit4] 2>
[junit4] 2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-3C1FB6EADDDDFE71-001
[junit4] 2> NOTE: test params are: codec=Lucene41, sim=DefaultSimilarity, locale=da_DK, timezone=AST
[junit4] 2> NOTE: Linux 3.8.0-42-generic i386/Oracle Corporation 1.7.0_60 (32-bit)/cpus=8,threads=1,free=129442832,total=254279680
[junit4] 2> NOTE: All tests run in this JVM: [CSVRequestHandlerTest, TestOmitPositions, MergeStrategyTest, TestStandardQParsers, TestCollapseQParserPlugin, TestCopyFieldCollectionResource, HdfsCollectionsAPIDistributedZkTest, SynonymTokenizerTest, TestStressLucene, TestManagedResource, DistributedDebugComponentTest, TestSystemIdResolver, HdfsLockFactoryTest, SolrIndexConfigTest, AnalysisAfterCoreReloadTest, TestTolerantSearch, SimplePostToolTest, TestBadConfig, HdfsChaosMonkeySafeLeaderTest, CursorMarkTest, TestFieldTypeCollectionResource, TestSolrQueryParserResource, TestMaxScoreQueryParser, CacheHeaderTest, TriLevelCompositeIdRoutingTest, MultiTermTest, DeleteShardTest, TestCollationKeyRangeQueries, DistanceFunctionTest, TestExceedMaxTermLength, BasicZkTest, FileBasedSpellCheckerTest, DirectSolrConnectionTest, RequestHandlersTest, TestWriterPerf, TimeZoneUtilsTest, TestSimpleQParserPlugin, TestNonDefinedSimilarityFactory, PolyFieldTest, TestReRankQParserPlugin, TestSchemaVersionResource, CopyFieldTest, TestRestManager, DebugComponentTest, HdfsWriteToMultipleCollectionsTest, TestQuerySenderNoQuery, CoreMergeIndexesAdminHandlerTest, TestZkChroot, TestAnalyzeInfixSuggestions, TestIndexingPerformance, TestRTGBase, TestFastLRUCache, NumericFieldsTest, TestSuggestSpellingConverter, TestNonNRTOpen, SearchHandlerTest, DistributedQueryComponentCustomSortTest, TermVectorComponentDistributedTest, TestUpdate, PreAnalyzedFieldTest, TestFunctionQuery, TestFieldSortValues, LeaderElectionIntegrationTest, SolrPluginUtilsTest, TestFastWriter, TestNRTOpen, ChaosMonkeySafeLeaderTest]
[junit4] Completed on J1 in 74.78s, 1 test, 1 failure <<< FAILURES!
[...truncated 902 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1299: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:923: There were test failures: 405 suites, 1656 tests, 1 failure, 37 ignored (16 assumptions)
Total time: 70 minutes 53 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 32bit/jdk1.7.0_60 -client -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.7.0_60) - Build # 10617
- Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/10617/
Java: 32bit/jdk1.7.0_60 -client -XX:+UseG1GC
1 tests failed.
REGRESSION: org.apache.lucene.analysis.core.TestRandomChains.testRandomChains
Error Message:
startOffset must be non-negative, and endOffset must be >= startOffset, startOffset=9,endOffset=5
Stack Trace:
java.lang.IllegalArgumentException: startOffset must be non-negative, and endOffset must be >= startOffset, startOffset=9,endOffset=5
at __randomizedtesting.SeedInfo.seed([8C2FE2CDE4898454:B1CECBACA39B9994]:0)
at org.apache.lucene.analysis.tokenattributes.PackedTokenAttributeImpl.setOffset(PackedTokenAttributeImpl.java:107)
at org.apache.lucene.analysis.shingle.ShingleFilter.incrementToken(ShingleFilter.java:345)
at org.apache.lucene.analysis.ValidatingTokenFilter.incrementToken(ValidatingTokenFilter.java:68)
at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkAnalysisConsistency(BaseTokenStreamTestCase.java:703)
at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:614)
at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:513)
at org.apache.lucene.analysis.core.TestRandomChains.testRandomChains(TestRandomChains.java:927)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at java.lang.Thread.run(Thread.java:745)
Build Log:
[...truncated 5756 lines...]
[junit4] Suite: org.apache.lucene.analysis.core.TestRandomChains
[junit4] 2> TEST FAIL: useCharFilter=false text='&;</p \ud83c\udc3f \u039a\uda51\udef5 \ua4a0\ua4ce\ua492\ua4b3\ua494'
[junit4] 2> Exception from random analyzer:
[junit4] 2> charfilters=
[junit4] 2> tokenizer=
[junit4] 2> org.apache.lucene.analysis.path.ReversePathHierarchyTokenizer(org.apache.lucene.analysis.core.TestRandomChains$CheckThatYouDidntReadAnythingReaderWrapper@c83859)
[junit4] 2> filters=
[junit4] 2> org.apache.lucene.analysis.id.IndonesianStemFilter(ValidatingTokenFilter@18a6330 term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,keyword=false, true)
[junit4] 2> org.apache.lucene.analysis.miscellaneous.WordDelimiterFilter(LUCENE_4_10, ValidatingTokenFilter@1cffabc term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,keyword=false, 7, [regebj, y, zfeqchmqil, ujunkv])
[junit4] 2> org.apache.lucene.analysis.miscellaneous.HyphenatedWordsFilter(ValidatingTokenFilter@f013b0 term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,keyword=false)
[junit4] 2> org.apache.lucene.analysis.shingle.ShingleFilter(ValidatingTokenFilter@3eaecd term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,keyword=false, <EMAIL>)
[junit4] 2> offsetsAreCorrect=false
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestRandomChains -Dtests.method=testRandomChains -Dtests.seed=8C2FE2CDE4898454 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=mk_MK -Dtests.timezone=Europe/Kiev -Dtests.file.encoding=ISO-8859-1
[junit4] ERROR 32.3s J1 | TestRandomChains.testRandomChains <<<
[junit4] > Throwable #1: java.lang.IllegalArgumentException: startOffset must be non-negative, and endOffset must be >= startOffset, startOffset=9,endOffset=5
[junit4] > at __randomizedtesting.SeedInfo.seed([8C2FE2CDE4898454:B1CECBACA39B9994]:0)
[junit4] > at org.apache.lucene.analysis.tokenattributes.PackedTokenAttributeImpl.setOffset(PackedTokenAttributeImpl.java:107)
[junit4] > at org.apache.lucene.analysis.shingle.ShingleFilter.incrementToken(ShingleFilter.java:345)
[junit4] > at org.apache.lucene.analysis.ValidatingTokenFilter.incrementToken(ValidatingTokenFilter.java:68)
[junit4] > at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkAnalysisConsistency(BaseTokenStreamTestCase.java:703)
[junit4] > at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:614)
[junit4] > at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:513)
[junit4] > at org.apache.lucene.analysis.core.TestRandomChains.testRandomChains(TestRandomChains.java:927)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] 2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/build/analysis/common/test/J1/./lucene.analysis.core.TestRandomChains-8C2FE2CDE4898454-001
[junit4] 2> NOTE: test params are: codec=CheapBastard, sim=DefaultSimilarity, locale=mk_MK, timezone=Europe/Kiev
[junit4] 2> NOTE: Linux 3.8.0-42-generic i386/Oracle Corporation 1.7.0_60 (32-bit)/cpus=8,threads=1,free=80436608,total=310378496
[junit4] 2> NOTE: All tests run in this JVM: [TestSolrSynonymParser, TestSwedishLightStemFilterFactory, TestHindiFilters, TestSnowball, TestCharArrayIterator, TestPersianCharFilter, TestTwoSuffixes, TestWordlistLoader, TestSpanishLightStemFilterFactory, TestTurkishLowerCaseFilter, TestClassicAnalyzer, TestReverseStringFilter, TestPortugueseStemFilter, TestBulgarianStemmer, TestKeepFilterFactory, TestCharacterUtils, TestNorwegianMinimalStemFilter, TestPerFieldAnalyzerWrapper, TestPrefixAwareTokenFilter, TestCircumfix, TestFinnishLightStemFilterFactory, TestMorphData, TestCodepointCountFilter, ShingleFilterTest, TestRollingCharBuffer, TestGreekStemmer, TestPatternReplaceFilter, TestKeywordMarkerFilter, TestTurkishAnalyzer, QueryAutoStopWordAnalyzerTest, TestPortugueseLightStemFilterFactory, TestCollationKeyFilterFactory, TestFinnishAnalyzer, TestRussianAnalyzer, TestPortugueseStemFilterFactory, TestFilesystemResourceLoader, TestHunspellStemFilterFactory, TestFrenchAnalyzer, TestSynonymMap, TestWordnetSynonymParser, TestScandinavianNormalizationFilterFactory, TestStemmerOverrideFilterFactory, TestCharTokenizers, TestArmenianAnalyzer, TestLimitTokenCountFilter, DateRecognizerSinkTokenizerTest, TestCommonGramsFilterFactory, TestEnglishMinimalStemFilterFactory, TestCJKTokenizer, TestCJKBigramFilterFactory, EdgeNGramTokenizerTest, TestDanishAnalyzer, TestHindiAnalyzer, TestSpanishLightStemFilter, TestGreekStemFilterFactory, TestApostropheFilterFactory, TestSnowballPorterFilterFactory, TestCapitalizationFilter, TestIrishLowerCaseFilter, TestHungarianLightStemFilterFactory, TestPersianAnalyzer, TestCollationKeyAnalyzer, TestSoraniNormalizationFilter, TestStandardFactories, TestCzechStemmer, TestTypeTokenFilter, TestKeywordMarkerFilterFactory, TestFactories, TestRandomChains]
[junit4] Completed on J1 in 56.67s, 2 tests, 1 error <<< FAILURES!
[...truncated 421 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/build.xml:544: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:2019: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/analysis/build.xml:106: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/analysis/build.xml:38: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/module-build.xml:60: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1299: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:923: There were test failures: 271 suites, 1456 tests, 1 error, 1 ignored
Total time: 32 minutes 37 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 32bit/jdk1.7.0_60 -client -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
Re: [JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.8.0_05) - Build #
10616 - Still Failing!
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
Thanks Mark. This is the same issue as SOLR-5596. I suggest that we disable
fsync for our tests. Let's discuss on the jira issue.
https://issues.apache.org/jira/browse/SOLR-5596?focusedCommentId=14053776&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14053776
On Mon, Jul 7, 2014 at 9:15 PM, Mark Miller <ma...@gmail.com> wrote:
> On July 7, 2014 at 11:39:31 AM, Shalin Shekhar Mangar (
> shalinmangar@gmail.com) wrote:
>
> I was looking into the OverseerTest failure and the following caught my
> attention.
>
> This error can be due to a slow machine but it also happens on fast
> machines if you try to do a lot of writes very fast on ZooKeeper which is
> what the testShardLeaderChange does. Perhaps we should add a small wait
> between operations?
>
>
> It’s probably worth a try. I tried once to work around it via ZooKeeper
> configuration options, thought there was a promising path or two after some
> googling, but I don’t think I ended up getting anywhere.
>
> - Mark
>
>
>
> [junit4] 2> 472241 T2893 oazsp.FileTxnLog.commit WARN fsync-ing the write ahead log in SyncThread:0 took 11588ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
>
>
>
> On Sun, Jul 6, 2014 at 10:46 AM, Policeman Jenkins Server <
> jenkins@thetaphi.de> wrote:
>
>> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/10616/
>> Java: 32bit/jdk1.8.0_05 -client -XX:+UseParallelGC
>>
>> 5 tests failed.
>> FAILED: junit.framework.TestSuite.org.apache.solr.cloud.OverseerTest
>>
>> Error Message:
>> ERROR: SolrZkClient opens=165 closes=164
>>
>> Stack Trace:
>> java.lang.AssertionError: ERROR: SolrZkClient opens=165 closes=164
>> at __randomizedtesting.SeedInfo.seed([A819205877C572F6]:0)
>> at org.junit.Assert.fail(Assert.java:93)
>> at
>> org.apache.solr.SolrTestCaseJ4.endTrackingZkClients(SolrTestCaseJ4.java:436)
>> at
>> org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:182)
>> at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:483)
>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:790)
>> at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at
>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
>> at
>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>> at
>> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>> at
>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>> at
>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>> at
>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>> at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at
>> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
>> at
>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>> at
>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>> at
>> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>> at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>> at java.lang.Thread.run(Thread.java:745)
>>
>>
>> FAILED: junit.framework.TestSuite.org.apache.solr.cloud.OverseerTest
>>
>> Error Message:
>> 2 threads leaked from SUITE scope at org.apache.solr.cloud.OverseerTest:
>> 1) Thread[id=2969,
>> name=TEST-OverseerTest.testShardLeaderChange-seed#[A819205877C572F6]-SendThread(
>> 127.0.0.1:33880), state=TIMED_WAITING, group=TGRP-OverseerTest]
>> at java.lang.Thread.sleep(Native Method) at
>> org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003) 2)
>> Thread[id=2970,
>> name=TEST-OverseerTest.testShardLeaderChange-seed#[A819205877C572F6]-EventThread,
>> state=WAITING, group=TGRP-OverseerTest] at
>> sun.misc.Unsafe.park(Native Method) at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> at
>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
>>
>> Stack Trace:
>> com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from
>> SUITE scope at org.apache.solr.cloud.OverseerTest:
>> 1) Thread[id=2969,
>> name=TEST-OverseerTest.testShardLeaderChange-seed#[A819205877C572F6]-SendThread(
>> 127.0.0.1:33880), state=TIMED_WAITING, group=TGRP-OverseerTest]
>> at java.lang.Thread.sleep(Native Method)
>> at
>> org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
>> 2) Thread[id=2970,
>> name=TEST-OverseerTest.testShardLeaderChange-seed#[A819205877C572F6]-EventThread,
>> state=WAITING, group=TGRP-OverseerTest]
>> at sun.misc.Unsafe.park(Native Method)
>> at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> at
>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
>> at __randomizedtesting.SeedInfo.seed([A819205877C572F6]:0)
>>
>>
>> FAILED: junit.framework.TestSuite.org.apache.solr.cloud.OverseerTest
>>
>> Error Message:
>> There are still zombie threads that couldn't be terminated: 1)
>> Thread[id=2969,
>> name=TEST-OverseerTest.testShardLeaderChange-seed#[A819205877C572F6]-SendThread(
>> 127.0.0.1:33880), state=TIMED_WAITING, group=TGRP-OverseerTest]
>> at java.lang.Thread.sleep(Native Method) at
>> org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
>>
>> Stack Trace:
>> com.carrotsearch.randomizedtesting.ThreadLeakError: There are still
>> zombie threads that couldn't be terminated:
>> 1) Thread[id=2969,
>> name=TEST-OverseerTest.testShardLeaderChange-seed#[A819205877C572F6]-SendThread(
>> 127.0.0.1:33880), state=TIMED_WAITING, group=TGRP-OverseerTest]
>> at java.lang.Thread.sleep(Native Method)
>> at
>> org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
>> at __randomizedtesting.SeedInfo.seed([A819205877C572F6]:0)
>>
>>
>> REGRESSION: org.apache.solr.cloud.OverseerTest.testShardLeaderChange
>>
>> Error Message:
>> KeeperErrorCode = NodeExists for /live_nodes/node1
>>
>> Stack Trace:
>> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode
>> = NodeExists for /live_nodes/node1
>> at
>> __randomizedtesting.SeedInfo.seed([A819205877C572F6:764AA7AF6D5D8707]:0)
>> at
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
>> at
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
>> at
>> org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:432)
>> at
>> org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
>> at
>> org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:429)
>> at
>> org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:386)
>> at
>> org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:373)
>> at
>> org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:350)
>> at
>> org.apache.solr.cloud.OverseerTest$MockZKController.<init>(OverseerTest.java:85)
>> at
>> org.apache.solr.cloud.OverseerTest.testShardLeaderChange(OverseerTest.java:746)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:483)
>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
>> at
>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
>> at
>> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>> at
>> org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
>> at
>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>> at
>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>> at
>> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>> at
>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>> at
>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>> at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>> at
>> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
>> at
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
>> at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at
>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
>> at
>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>> at
>> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>> at
>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>> at
>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>> at
>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>> at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at
>> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
>> at
>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>> at
>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>> at
>> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>> at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>> at java.lang.Thread.run(Thread.java:745)
>>
>>
>> FAILED: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch
>>
>> Error Message:
>> shard2 is not consistent. Got 153 from
>> http://127.0.0.1:51995/collection1lastClient and got 132 from
>> http://127.0.0.1:36739/collection1
>>
>> Stack Trace:
>> java.lang.AssertionError: shard2 is not consistent. Got 153 from
>> http://127.0.0.1:51995/collection1lastClient and got 132 from
>> http://127.0.0.1:36739/collection1
>> at
>> __randomizedtesting.SeedInfo.seed([A819205877C572F6:29FFAE40009A12CA]:0)
>> at org.junit.Assert.fail(Assert.java:93)
>> at
>> org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1139)
>> at
>> org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1118)
>> at
>> org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:150)
>> at
>> org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:483)
>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
>> at
>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
>> at
>> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>> at
>> org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
>> at
>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>> at
>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>> at
>> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>> at
>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>> at
>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>> at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>> at
>> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
>> at
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
>> at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at
>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
>> at
>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>> at
>> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>> at
>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>> at
>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>> at
>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>> at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at
>> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
>> at
>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>> at
>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>> at
>> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>> at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>> at java.lang.Thread.run(Thread.java:745)
>>
>>
>>
>>
>> Build Log:
>> [...truncated 11133 lines...]
>> [junit4] Suite: org.apache.solr.cloud.OverseerTest
>> [junit4] 2> Creating dataDir:
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.OverseerTest-A819205877C572F6-001/init-core-data-001
>> [junit4] 2> 443102 T2522 oas.SolrTestCaseJ4.buildSSLConfig
>> Randomized ssl (false) and clientAuth (false)
>> [junit4] 2> 443103 T2522 oas.SolrTestCaseJ4.initCore ####initCore
>> [junit4] 2> 443103 T2522 oas.SolrTestCaseJ4.initCore ####initCore end
>> [junit4] 2> 443107 T2522 oas.SolrTestCaseJ4.setUp ###Starting
>> testStateChange
>> [junit4] 2> 443107 T2522 oasc.ZkTestServer.run STARTING ZK TEST
>> SERVER
>> [junit4] 1> client port:0.0.0.0/0.0.0.0:0
>> [junit4] 2> 443108 T2523
>> oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
>> [junit4] 2> 443208 T2522 oasc.ZkTestServer.run start zk server on
>> port:45124
>> [junit4] 2> 443209 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 443282 T2529 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@ed72a5
>> name:ZooKeeperConnection Watcher:127.0.0.1:45124/solr got event
>> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 443283 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 443284 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 443287 T2531 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@6366c4
>> name:ZooKeeperConnection Watcher:127.0.0.1:45124 got event WatchedEvent
>> state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 443287 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 443290 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 443292 T2533 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@60240b
>> name:ZooKeeperConnection Watcher:127.0.0.1:45124 got event WatchedEvent
>> state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 443292 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 443293 T2522 oascc.SolrZkClient.makePath makePath: /solr
>> [junit4] 2> 443298 T2522 oascc.SolrZkClient.makePath makePath:
>> /live_nodes
>> [junit4] 2> 443301 T2522 oascc.SolrZkClient.makePath makePath:
>> /clusterstate.json
>> [junit4] 2> 443304 T2522 oascc.SolrZkClient.makePath makePath:
>> /aliases.json
>> [junit4] 2> 443306 T2522
>> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
>> state from ZooKeeper...
>> [junit4] 2> 443311 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 443312 T2535 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@68e786
>> name:ZooKeeperConnection Watcher:127.0.0.1:45124/solr got event
>> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 443313 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 443314 T2522 oascc.SolrZkClient.makePath makePath:
>> /overseer_elect
>> [junit4] 2> 443318 T2522 oascc.SolrZkClient.makePath makePath:
>> /overseer_elect/election
>> [junit4] 2> 443324 T2522
>> oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader
>> 127.0.0.1:45124_solr
>> [junit4] 2> 443324 T2522 oascc.SolrZkClient.makePath makePath:
>> /overseer_elect/leader
>> [junit4] 2> 443328 T2522 oasc.Overseer.start Overseer
>> (id=92053395863699459-127.0.0.1:45124_solr-n_0000000000) starting
>> [junit4] 2> 443331 T2522 oascc.SolrZkClient.makePath makePath:
>> /overseer/queue
>> [junit4] 2> 443336 T2522 oascc.SolrZkClient.makePath makePath:
>> /overseer/queue-work
>> [junit4] 2> 443340 T2522 oascc.SolrZkClient.makePath makePath:
>> /overseer/collection-map-failure
>> [junit4] 2> 443345 T2522 oascc.SolrZkClient.makePath makePath:
>> /overseer/collection-map-running
>> [junit4] 2> 443350 T2522 oascc.SolrZkClient.makePath makePath:
>> /overseer/collection-map-completed
>> [junit4] 2> 443357 T2522 oascc.SolrZkClient.makePath makePath:
>> /overseer/collection-queue-work
>> [junit4] 2> 443365 T2537 oasc.OverseerCollectionProcessor.run
>> Process current queue of collection creations
>> [junit4] 2> 443366 T2536 oasc.Overseer$ClusterStateUpdater.run
>> Starting to work on the main queue
>> [junit4] 2> 443368 T2522 oascc.ZkStateReader.updateClusterState
>> Updating cloud state from ZooKeeper...
>> [junit4] 2> 443369 T2536 oascc.ZkStateReader.updateClusterState
>> Updating cloud state from ZooKeeper...
>> [junit4] 2> 443370 T2536
>> oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null
>> message={
>> [junit4] 2> "operation":"state",
>> [junit4] 2> "base_url":"http://127.0.0.1/solr",
>> [junit4] 2> "node_name":"node1",
>> [junit4] 2> "collection":"collection1",
>> [junit4] 2> "core":"core1",
>> [junit4] 2> "roles":"",
>> [junit4] 2> "state":"recovering"}
>> [junit4] 2> 443370 T2536
>> oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard
>> shard=shard1
>> [junit4] 2> 443374 T2535
>> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
>> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>> [junit4] 2> 443375 T2529 oascc.ZkStateReader$2.process A cluster
>> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
>> [junit4] 2> 443420 T2522 oascc.ZkStateReader.updateClusterState
>> Updating cloud state from ZooKeeper...
>> [junit4] 2> 443422 T2535
>> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
>> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>> [junit4] 2> 443424 T2536 oascc.ZkStateReader.updateClusterState
>> Updating cloud state from ZooKeeper...
>> [junit4] 2> 443425 T2536
>> oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null
>> message={
>> [junit4] 2> "operation":"state",
>> [junit4] 2> "base_url":"http://127.0.0.1/solr",
>> [junit4] 2> "node_name":"node1",
>> [junit4] 2> "collection":"collection1",
>> [junit4] 2> "core":"core1",
>> [junit4] 2> "roles":"",
>> [junit4] 2> "state":"active"}
>> [junit4] 2> 443426 T2536
>> oasc.Overseer$ClusterStateUpdater.updateState node=core_node1 is already
>> registered
>> [junit4] 2> 443426 T2536
>> oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already
>> registered
>> [junit4] 2> 443431 T2535
>> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
>> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>> [junit4] 2> 443534 T2529 oascc.ZkStateReader$2.process A cluster
>> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
>> [junit4] 2> 443575 T2522 oasc.ZkTestServer.send4LetterWord
>> connecting to 127.0.0.1:45124 45124
>> [junit4] 2> 443988 T2523 oasc.ZkTestServer.send4LetterWord
>> connecting to 127.0.0.1:45124 45124
>> [junit4] 2> 443991 T2522 oas.SolrTestCaseJ4.tearDown ###Ending
>> testStateChange
>> [junit4] 2> 443992 T2536 oasc.Overseer$ClusterStateUpdater.run
>> Overseer Loop exiting : 127.0.0.1:45124_solr
>> [junit4] 2> 443994 T2538
>> oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not
>> read the data org.apache.zookeeper.KeeperException$SessionExpiredException:
>> KeeperErrorCode = Session expired for /overseer_elect/leader
>> [junit4] 2> at
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
>> [junit4] 2> at
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>> [junit4] 2> at
>> org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
>> [junit4] 2> at
>> org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
>> [junit4] 2> at
>> org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
>> [junit4] 2> at
>> org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
>> [junit4] 2> at
>> org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
>> [junit4] 2> at
>> org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:316)
>> [junit4] 2> at
>> org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
>> [junit4] 2> at
>> org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:288)
>> [junit4] 2>
>> [junit4] 2> 444000 T2522 oas.SolrTestCaseJ4.setUp ###Starting
>> testShardAssignment
>> [junit4] 2> 444001 T2522 oasc.ZkTestServer.run STARTING ZK TEST
>> SERVER
>> [junit4] 1> client port:0.0.0.0/0.0.0.0:0
>> [junit4] 2> 444002 T2539
>> oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
>> [junit4] 2> 444101 T2522 oasc.ZkTestServer.run start zk server on
>> port:57742
>> [junit4] 2> 444103 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 444107 T2545 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@9fc604
>> name:ZooKeeperConnection Watcher:127.0.0.1:57742 got event WatchedEvent
>> state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 444108 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 444111 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 444112 T2547 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@104f2e0
>> name:ZooKeeperConnection Watcher:127.0.0.1:57742 got event WatchedEvent
>> state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 444113 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 444114 T2522 oascc.SolrZkClient.makePath makePath: /solr
>> [junit4] 2> 444119 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 444121 T2549 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@896251
>> name:ZooKeeperConnection Watcher:127.0.0.1:57742/solr got event
>> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 444121 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 444122 T2522 oascc.SolrZkClient.makePath makePath:
>> /live_nodes
>> [junit4] 2> 444125 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 444127 T2551 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@10c3e42
>> name:ZooKeeperConnection Watcher:127.0.0.1:57742/solr got event
>> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 444127 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 444128 T2522 oascc.SolrZkClient.makePath makePath:
>> /overseer_elect
>> [junit4] 2> 444132 T2522 oascc.SolrZkClient.makePath makePath:
>> /overseer_elect/election
>> [junit4] 2> 444137 T2522
>> oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader
>> 127.0.0.1:57742_solr
>> [junit4] 2> 444138 T2522 oascc.SolrZkClient.makePath makePath:
>> /overseer_elect/leader
>> [junit4] 2> 444141 T2522 oasc.Overseer.start Overseer
>> (id=92053395922223107-127.0.0.1:57742_solr-n_0000000000) starting
>> [junit4] 2> 444144 T2522 oascc.SolrZkClient.makePath makePath:
>> /overseer/queue
>> [junit4] 2> 444149 T2522 oascc.SolrZkClient.makePath makePath:
>> /overseer/queue-work
>> [junit4] 2> 444153 T2522 oascc.SolrZkClient.makePath makePath:
>> /overseer/collection-map-failure
>> [junit4] 2> 444157 T2522 oascc.SolrZkClient.makePath makePath:
>> /overseer/collection-map-running
>> [junit4] 2> 444162 T2522 oascc.SolrZkClient.makePath makePath:
>> /overseer/collection-map-completed
>> [junit4] 2> 444166 T2522 oascc.SolrZkClient.makePath makePath:
>> /overseer/collection-queue-work
>> [junit4] 2> 444173 T2553 oasc.OverseerCollectionProcessor.run
>> Process current queue of collection creations
>> [junit4] 2> 444175 T2522 oascc.SolrZkClient.makePath makePath:
>> /clusterstate.json
>> [junit4] 2> 444175 T2552 oasc.Overseer$ClusterStateUpdater.run
>> Starting to work on the main queue
>> [junit4] 2> 444177 T2522 oascc.SolrZkClient.makePath makePath:
>> /aliases.json
>> [junit4] 2> 444179 T2522
>> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
>> state from ZooKeeper...
>> [junit4] 2> 444183 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 444185 T2555 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@1196b81
>> name:ZooKeeperConnection Watcher:127.0.0.1:57742/solr got event
>> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 444185 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 444187 T2522
>> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
>> state from ZooKeeper...
>> [junit4] 2> 444190 T2522 oascc.SolrZkClient.makePath makePath:
>> /live_nodes/127.0.0.1
>> [junit4] 2> 444194 T2549 oascc.ZkStateReader$3.process Updating live
>> nodes... (1)
>> [junit4] 2> 444194 T2555 oascc.ZkStateReader$3.process Updating live
>> nodes... (1)
>> [junit4] 2> 444197 T2551
>> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
>> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>> [junit4] 2> 444199 T2552 oascc.ZkStateReader.updateClusterState
>> Updating cloud state from ZooKeeper...
>> [junit4] 2> 444199 T2552
>> oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3
>> message={
>> [junit4] 2> "operation":"state",
>> [junit4] 2> "state":"active",
>> [junit4] 2> "node_name":"127.0.0.1",
>> [junit4] 2> "core":"core1",
>> [junit4] 2> "core_node_name":"node1",
>> [junit4] 2> "collection":"collection1",
>> [junit4] 2> "numShards":"3",
>> [junit4] 2> "base_url":"http://127.0.0.1/solr/"}
>> [junit4] 2> 444200 T2552
>> oasc.Overseer$ClusterStateUpdater.createCollection Create collection
>> collection1 with shards [shard1, shard2, shard3]
>> [junit4] 2> 444200 T2552
>> oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard
>> shard=shard2
>> [junit4] 2> 444204 T2551
>> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
>> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>> [junit4] 2> 444205 T2555 oascc.ZkStateReader$2.process A cluster
>> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>> [junit4] 2> 444205 T2549 oascc.ZkStateReader$2.process A cluster
>> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>> [junit4] 2> 444697 T2522 oascc.SolrZkClient.makePath makePath:
>> /collections/collection1/leader_elect/shard2/election
>> [junit4] 2> 444712 T2522 oascc.SolrZkClient.makePath makePath:
>> /collections/collection1/leaders/shard2
>> [junit4] 2> 444719 T2551
>> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
>> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>> [junit4] 2> 444721 T2551
>> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
>> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>> [junit4] 2> 444721 T2552 oascc.ZkStateReader.updateClusterState
>> Updating cloud state from ZooKeeper...
>> [junit4] 2> 444727 T2552
>> oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3
>> message={
>> [junit4] 2> "operation":"state",
>> [junit4] 2> "state":"active",
>> [junit4] 2> "node_name":"127.0.0.1",
>> [junit4] 2> "core":"core2",
>> [junit4] 2> "core_node_name":"node2",
>> [junit4] 2> "collection":"collection1",
>> [junit4] 2> "numShards":"3",
>> [junit4] 2> "base_url":"http://127.0.0.1/solr/"}
>> [junit4] 2> 444728 T2552
>> oasc.Overseer$ClusterStateUpdater.updateState Collection already exists
>> with numShards=3
>> [junit4] 2> 444728 T2552
>> oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard
>> shard=shard3
>> [junit4] 2> 444731 T2551
>> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
>> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>> [junit4] 2> 444834 T2555 oascc.ZkStateReader$2.process A cluster
>> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>> [junit4] 2> 444834 T2549 oascc.ZkStateReader$2.process A cluster
>> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>> [junit4] 2> 445222 T2522 oascc.SolrZkClient.makePath makePath:
>> /collections/collection1/leader_elect/shard3/election
>> [junit4] 2> 445233 T2522 oascc.SolrZkClient.makePath makePath:
>> /collections/collection1/leaders/shard3
>> [junit4] 2> 445240 T2551
>> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
>> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>> [junit4] 2> 445240 T2551
>> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
>> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>> [junit4] 2> 445241 T2551
>> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
>> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>> [junit4] 2> 445242 T2551
>> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
>> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>> [junit4] 2> 445242 T2552 oascc.ZkStateReader.updateClusterState
>> Updating cloud state from ZooKeeper...
>> [junit4] 2> 445248 T2552
>> oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3
>> message={
>> [junit4] 2> "operation":"state",
>> [junit4] 2> "state":"active",
>> [junit4] 2> "node_name":"127.0.0.1",
>> [junit4] 2> "core":"core3",
>> [junit4] 2> "core_node_name":"node3",
>> [junit4] 2> "collection":"collection1",
>> [junit4] 2> "numShards":"3",
>> [junit4] 2> "base_url":"http://127.0.0.1/solr/"}
>> [junit4] 2> 445248 T2552
>> oasc.Overseer$ClusterStateUpdater.updateState Collection already exists
>> with numShards=3
>> [junit4] 2> 445248 T2552
>> oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard
>> shard=shard1
>> [junit4] 2> 445251 T2551
>> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
>> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>> [junit4] 2> 445354 T2549 oascc.ZkStateReader$2.process A cluster
>> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>> [junit4] 2> 445354 T2555 oascc.ZkStateReader$2.process A cluster
>> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>> [junit4] 2> 445742 T2522 oascc.SolrZkClient.makePath makePath:
>> /collections/collection1/leader_elect/shard1/election
>> [junit4] 2> 445753 T2522 oascc.SolrZkClient.makePath makePath:
>> /collections/collection1/leaders/shard1
>> [junit4] 2> 445764 T2551
>> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
>> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>> [junit4] 2> 445765 T2551
>> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
>> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>> [junit4] 2> 445765 T2551
>> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
>> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>> [junit4] 2> 445766 T2551
>> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
>> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>> [junit4] 2> 445767 T2552 oascc.ZkStateReader.updateClusterState
>> Updating cloud state from ZooKeeper...
>> [junit4] 2> 445771 T2552
>> oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3
>> message={
>> [junit4] 2> "operation":"state",
>> [junit4] 2> "state":"active",
>> [junit4] 2> "node_name":"127.0.0.1",
>> [junit4] 2> "core":"core4",
>> [junit4] 2> "core_node_name":"node4",
>> [junit4] 2> "collection":"collection1",
>> [junit4] 2> "numShards":"3",
>> [junit4] 2> "base_url":"http://127.0.0.1/solr/"}
>> [junit4] 2> 445771 T2552
>> oasc.Overseer$ClusterStateUpdater.updateState Collection already exists
>> with numShards=3
>> [junit4] 2> 445771 T2552
>> oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard
>> shard=shard2
>> [junit4] 2> 445773 T2551
>> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
>> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>> [junit4] 2> 445876 T2555 oascc.ZkStateReader$2.process A cluster
>> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>> [junit4] 2> 445876 T2549 oascc.ZkStateReader$2.process A cluster
>> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>> [junit4] 2> 446267 T2522 oascc.SolrZkClient.makePath makePath:
>> /collections/collection1/leader_elect/shard2/election
>> [junit4] 2> 446276 T2551
>> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
>> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>> [junit4] 2> 446276 T2551
>> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
>> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>> [junit4] 2> 446276 T2551
>> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
>> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>> [junit4] 2> 446278 T2552 oascc.ZkStateReader.updateClusterState
>> Updating cloud state from ZooKeeper...
>> [junit4] 2> 446279 T2552
>> oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3
>> message={
>> [junit4] 2> "operation":"state",
>> [junit4] 2> "state":"active",
>> [junit4] 2> "node_name":"127.0.0.1",
>> [junit4] 2> "core":"core5",
>> [junit4] 2> "core_node_name":"node5",
>> [junit4] 2> "collection":"collection1",
>> [junit4] 2> "numShards":"3",
>> [junit4] 2> "base_url":"http://127.0.0.1/solr/"}
>> [junit4] 2> 446280 T2552
>> oasc.Overseer$ClusterStateUpdater.updateState Collection already exists
>> with numShards=3
>> [junit4] 2> 446280 T2552
>> oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard
>> shard=shard3
>> [junit4] 2> 446283 T2551
>> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
>> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>> [junit4] 2> 446386 T2555 oascc.ZkStateReader$2.process A cluster
>> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>> [junit4] 2> 446386 T2549 oascc.ZkStateReader$2.process A cluster
>> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>> [junit4] 2> 446776 T2522 oascc.SolrZkClient.makePath makePath:
>> /collections/collection1/leader_elect/shard3/election
>> [junit4] 2> 446787 T2551
>> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
>> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>> [junit4] 2> 446788 T2551
>> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
>> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>> [junit4] 2> 446788 T2551
>> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
>> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>> [junit4] 2> 446790 T2552 oascc.ZkStateReader.updateClusterState
>> Updating cloud state from ZooKeeper...
>> [junit4] 2> 446791 T2552
>> oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3
>> message={
>> [junit4] 2> "operation":"state",
>> [junit4] 2> "state":"active",
>> [junit4] 2> "node_name":"127.0.0.1",
>> [junit4] 2> "core":"core6",
>> [junit4] 2> "core_node_name":"node6",
>> [junit4] 2> "collection":"collection1",
>> [junit4] 2> "numShards":"3",
>> [junit4] 2> "base_url":"http://127.0.0.1/solr/"}
>> [junit4] 2> 446791 T2552
>> oasc.Overseer$ClusterStateUpdater.updateState Collection already exists
>> with numShards=3
>> [junit4] 2> 446791 T2552
>> oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard
>> shard=shard1
>> [junit4] 2> 446795 T2551
>> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
>> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>> [junit4] 2> 446899 T2549 oascc.ZkStateReader$2.process A cluster
>> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>> [junit4] 2> 446899 T2555 oascc.ZkStateReader$2.process A cluster
>> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>> [junit4] 2> 447288 T2522 oascc.SolrZkClient.makePath makePath:
>> /collections/collection1/leader_elect/shard1/election
>> [junit4] 2> 447298 T2555 oascc.ZkStateReader$3.process Updating live
>> nodes... (0)
>> [junit4] 2> 447301 T2522 oasc.ZkTestServer.send4LetterWord
>> connecting to 127.0.0.1:57742 57742
>> [junit4] 2> 447869 T2539 oasc.ZkTestServer.send4LetterWord
>> connecting to 127.0.0.1:57742 57742
>> [junit4] 2> 447871 T2522 oas.SolrTestCaseJ4.tearDown ###Ending
>> testShardAssignment
>> [junit4] 2> 447872 T2552 oasc.Overseer$ClusterStateUpdater.run
>> Overseer Loop exiting : 127.0.0.1:57742_solr
>> [junit4] 2> 447875 T2556
>> oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not
>> read the data org.apache.zookeeper.KeeperException$SessionExpiredException:
>> KeeperErrorCode = Session expired for /overseer_elect/leader
>> [junit4] 2> at
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
>> [junit4] 2> at
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>> [junit4] 2> at
>> org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
>> [junit4] 2> at
>> org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
>> [junit4] 2> at
>> org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
>> [junit4] 2> at
>> org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
>> [junit4] 2> at
>> org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
>> [junit4] 2> at
>> org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:316)
>> [junit4] 2> at
>> org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
>> [junit4] 2> at
>> org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:288)
>> [junit4] 2>
>> [junit4] 2> 447881 T2522 oas.SolrTestCaseJ4.setUp ###Starting
>> testShardAssignmentBigger
>> [junit4] 2> 447882 T2522 oasc.ZkTestServer.run STARTING ZK TEST
>> SERVER
>> [junit4] 1> client port:0.0.0.0/0.0.0.0:0
>> [junit4] 2> 447883 T2557
>> oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
>> [junit4] 2> 447983 T2522 oasc.ZkTestServer.run start zk server on
>> port:50146
>> [junit4] 2> 447984 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 447988 T2563 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@daa195
>> name:ZooKeeperConnection Watcher:127.0.0.1:50146 got event WatchedEvent
>> state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 447989 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 447993 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 447994 T2565 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@f8aac
>> name:ZooKeeperConnection Watcher:127.0.0.1:50146 got event WatchedEvent
>> state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 447995 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 447995 T2522 oascc.SolrZkClient.makePath makePath: /solr
>> [junit4] 2> 448000 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 448002 T2567 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@141291c
>> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
>> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 448002 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 448003 T2522 oascc.SolrZkClient.makePath makePath:
>> /live_nodes
>> [junit4] 2> 448006 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 448008 T2569 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@49f99d
>> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
>> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 448008 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 448009 T2522 oascc.SolrZkClient.makePath makePath:
>> /overseer_elect
>> [junit4] 2> 448012 T2522 oascc.SolrZkClient.makePath makePath:
>> /overseer_elect/election
>> [junit4] 2> 448018 T2522
>> oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader
>> 127.0.0.1:50146_solr
>> [junit4] 2> 448018 T2522 oascc.SolrZkClient.makePath makePath:
>> /overseer_elect/leader
>> [junit4] 2> 448021 T2522 oasc.Overseer.start Overseer
>> (id=92053396176633859-127.0.0.1:50146_solr-n_0000000000) starting
>> [junit4] 2> 448024 T2522 oascc.SolrZkClient.makePath makePath:
>> /overseer/queue
>> [junit4] 2> 448029 T2522 oascc.SolrZkClient.makePath makePath:
>> /overseer/queue-work
>> [junit4] 2> 448034 T2522 oascc.SolrZkClient.makePath makePath:
>> /overseer/collection-map-failure
>> [junit4] 2> 448038 T2522 oascc.SolrZkClient.makePath makePath:
>> /overseer/collection-map-running
>> [junit4] 2> 448043 T2522 oascc.SolrZkClient.makePath makePath:
>> /overseer/collection-map-completed
>> [junit4] 2> 448048 T2522 oascc.SolrZkClient.makePath makePath:
>> /overseer/collection-queue-work
>> [junit4] 2> 448055 T2571 oasc.OverseerCollectionProcessor.run
>> Process current queue of collection creations
>> [junit4] 2> 448056 T2522 oascc.SolrZkClient.makePath makePath:
>> /clusterstate.json
>> [junit4] 2> 448056 T2570 oasc.Overseer$ClusterStateUpdater.run
>> Starting to work on the main queue
>> [junit4] 2> 448058 T2522 oascc.SolrZkClient.makePath makePath:
>> /aliases.json
>> [junit4] 2> 448061 T2522
>> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
>> state from ZooKeeper...
>> [junit4] 2> 448065 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 448066 T2573 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@1242e08
>> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
>> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 448067 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 448069 T2522
>> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
>> state from ZooKeeper...
>> [junit4] 2> 448072 T2522 oascc.SolrZkClient.makePath makePath:
>> /live_nodes/node0
>> [junit4] 2> 448076 T2573 oascc.ZkStateReader$3.process Updating live
>> nodes... (1)
>> [junit4] 2> 448076 T2567 oascc.ZkStateReader$3.process Updating live
>> nodes... (1)
>> [junit4] 2> 448077 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 448078 T2575 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@1fc812a
>> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
>> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 448078 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 448080 T2522
>> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
>> state from ZooKeeper...
>> [junit4] 2> 448083 T2522 oascc.SolrZkClient.makePath makePath:
>> /live_nodes/node1
>> [junit4] 2> 448087 T2573 oascc.ZkStateReader$3.process Updating live
>> nodes... (2)
>> [junit4] 2> 448087 T2567 oascc.ZkStateReader$3.process Updating live
>> nodes... (2)
>> [junit4] 2> 448087 T2575 oascc.ZkStateReader$3.process Updating live
>> nodes... (2)
>> [junit4] 2> 448088 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 448089 T2577 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@1714ec0
>> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
>> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 448090 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 448091 T2522
>> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
>> state from ZooKeeper...
>> [junit4] 2> 448095 T2522 oascc.SolrZkClient.makePath makePath:
>> /live_nodes/node2
>> [junit4] 2> 448099 T2573 oascc.ZkStateReader$3.process Updating live
>> nodes... (3)
>> [junit4] 2> 448099 T2577 oascc.ZkStateReader$3.process Updating live
>> nodes... (3)
>> [junit4] 2> 448099 T2575 oascc.ZkStateReader$3.process Updating live
>> nodes... (3)
>> [junit4] 2> 448100 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 448099 T2567 oascc.ZkStateReader$3.process Updating live
>> nodes... (3)
>> [junit4] 2> 448102 T2579 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@fbc5e2
>> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
>> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 448102 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 448104 T2522
>> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
>> state from ZooKeeper...
>> [junit4] 2> 448107 T2522 oascc.SolrZkClient.makePath makePath:
>> /live_nodes/node3
>> [junit4] 2> 448112 T2577 oascc.ZkStateReader$3.process Updating live
>> nodes... (4)
>> [junit4] 2> 448112 T2575 oascc.ZkStateReader$3.process Updating live
>> nodes... (4)
>> [junit4] 2> 448112 T2567 oascc.ZkStateReader$3.process Updating live
>> nodes... (4)
>> [junit4] 2> 448112 T2573 oascc.ZkStateReader$3.process Updating live
>> nodes... (4)
>> [junit4] 2> 448113 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 448113 T2579 oascc.ZkStateReader$3.process Updating live
>> nodes... (4)
>> [junit4] 2> 448115 T2581 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@75e1db
>> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
>> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 448115 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 448117 T2522
>> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
>> state from ZooKeeper...
>> [junit4] 2> 448121 T2522 oascc.SolrZkClient.makePath makePath:
>> /live_nodes/node4
>> [junit4] 2> 448126 T2577 oascc.ZkStateReader$3.process Updating live
>> nodes... (5)
>> [junit4] 2> 448126 T2575 oascc.ZkStateReader$3.process Updating live
>> nodes... (5)
>> [junit4] 2> 448126 T2573 oascc.ZkStateReader$3.process Updating live
>> nodes... (5)
>> [junit4] 2> 448127 T2581 oascc.ZkStateReader$3.process Updating live
>> nodes... (5)
>> [junit4] 2> 448126 T2567 oascc.ZkStateReader$3.process Updating live
>> nodes... (5)
>> [junit4] 2> 448126 T2579 oascc.ZkStateReader$3.process Updating live
>> nodes... (5)
>> [junit4] 2> 448127 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 448129 T2583 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@11dc02e
>> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
>> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 448130 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 448132 T2522
>> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
>> state from ZooKeeper...
>> [junit4] 2> 448136 T2522 oascc.SolrZkClient.makePath makePath:
>> /live_nodes/node5
>> [junit4] 2> 448139 T2575 oascc.ZkStateReader$3.process Updating live
>> nodes... (6)
>> [junit4] 2> 448140 T2577 oascc.ZkStateReader$3.process Updating live
>> nodes... (6)
>> [junit4] 2> 448140 T2581 oascc.ZkStateReader$3.process Updating live
>> nodes... (6)
>> [junit4] 2> 448140 T2573 oascc.ZkStateReader$3.process Updating live
>> nodes... (6)
>> [junit4] 2> 448140 T2579 oascc.ZkStateReader$3.process Updating live
>> nodes... (6)
>> [junit4] 2> 448141 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 448141 T2583 oascc.ZkStateReader$3.process Updating live
>> nodes... (6)
>> [junit4] 2> 448140 T2567 oascc.ZkStateReader$3.process Updating live
>> nodes... (6)
>> [junit4] 2> 448143 T2585 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@6d9f03
>> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
>> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 448143 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 448145 T2522
>> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
>> state from ZooKeeper...
>> [junit4] 2> 448148 T2522 oascc.SolrZkClient.makePath makePath:
>> /live_nodes/node6
>> [junit4] 2> 448153 T2575 oascc.ZkStateReader$3.process Updating live
>> nodes... (7)
>> [junit4] 2> 448153 T2577 oascc.ZkStateReader$3.process Updating live
>> nodes... (7)
>> [junit4] 2> 448153 T2573 oascc.ZkStateReader$3.process Updating live
>> nodes... (7)
>> [junit4] 2> 448153 T2585 oascc.ZkStateReader$3.process Updating live
>> nodes... (7)
>> [junit4] 2> 448153 T2583 oascc.ZkStateReader$3.process Updating live
>> nodes... (7)
>> [junit4] 2> 448153 T2579 oascc.ZkStateReader$3.process Updating live
>> nodes... (7)
>> [junit4] 2> 448153 T2581 oascc.ZkStateReader$3.process Updating live
>> nodes... (7)
>> [junit4] 2> 448153 T2567 oascc.ZkStateReader$3.process Updating live
>> nodes... (7)
>> [junit4] 2> 448154 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 448156 T2587 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@958679
>> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
>> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 448157 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 448159 T2522
>> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
>> state from ZooKeeper...
>> [junit4] 2> 448162 T2522 oascc.SolrZkClient.makePath makePath:
>> /live_nodes/node7
>> [junit4] 2> 448167 T2567 oascc.ZkStateReader$3.process Updating live
>> nodes... (8)
>> [junit4] 2> 448167 T2577 oascc.ZkStateReader$3.process Updating live
>> nodes... (8)
>> [junit4] 2> 448168 T2585 oascc.ZkStateReader$3.process Updating live
>> nodes... (8)
>> [junit4] 2> 448168 T2583 oascc.ZkStateReader$3.process Updating live
>> nodes... (8)
>> [junit4] 2> 448168 T2581 oascc.ZkStateReader$3.process Updating live
>> nodes... (8)
>> [junit4] 2> 448168 T2579 oascc.ZkStateReader$3.process Updating live
>> nodes... (8)
>> [junit4] 2> 448168 T2573 oascc.ZkStateReader$3.process Updating live
>> nodes... (8)
>> [junit4] 2> 448167 T2587 oascc.ZkStateReader$3.process Updating live
>> nodes... (8)
>> [junit4] 2> 448167 T2575 oascc.ZkStateReader$3.process Updating live
>> nodes... (8)
>> [junit4] 2> 448169 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 448171 T2589 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@169d949
>> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
>> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 448171 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 448174 T2522
>> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
>> state from ZooKeeper...
>> [junit4] 2> 448177 T2522 oascc.SolrZkClient.makePath makePath:
>> /live_nodes/node8
>> [junit4] 2> 448182 T2589 oascc.ZkStateReader$3.process Updating live
>> nodes... (9)
>> [junit4] 2> 448183 T2585 oascc.ZkStateReader$3.process Updating live
>> nodes... (9)
>> [junit4] 2> 448183 T2583 oascc.ZkStateReader$3.process Updating live
>> nodes... (9)
>> [junit4] 2> 448183 T2587 oascc.ZkStateReader$3.process Updating live
>> nodes... (9)
>> [junit4] 2> 448183 T2573 oascc.ZkStateReader$3.process Updating live
>> nodes... (9)
>> [junit4] 2> 448183 T2579 oascc.ZkStateReader$3.process Updating live
>> nodes... (9)
>> [junit4] 2> 448183 T2581 oascc.ZkStateReader$3.process Updating live
>> nodes... (9)
>> [junit4] 2> 448183 T2575 oascc.ZkStateReader$3.process Updating live
>> nodes... (9)
>> [junit4] 2> 448183 T2577 oascc.ZkStateReader$3.process Updating live
>> nodes... (9)
>> [junit4] 2> 448182 T2567 oascc.ZkStateReader$3.process Updating live
>> nodes... (9)
>> [junit4] 2> 448185 T2591 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@2e78fa
>> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
>> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 448184 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 448188 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 448190 T2522
>> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
>> state from ZooKeeper...
>> [junit4] 2> 448192 T2522 oascc.SolrZkClient.makePath makePath:
>> /live_nodes/node9
>> [junit4] 2> 448195 T2589 oascc.ZkStateReader$3.process Updating live
>> nodes... (10)
>> [junit4] 2> 448195 T2577 oascc.ZkStateReader$3.process Updating live
>> nodes... (10)
>> [junit4] 2> 448195 T2567 oascc.ZkStateReader$3.process Updating live
>> nodes... (10)
>> [junit4] 2> 448196 T2573 oascc.ZkStateReader$3.process Updating live
>> nodes... (10)
>> [junit4] 2> 448195 T2575 oascc.ZkStateReader$3.process Updating live
>> nodes... (10)
>> [junit4] 2> 448195 T2587 oascc.ZkStateReader$3.process Updating live
>> nodes... (10)
>> [junit4] 2> 448195 T2579 oascc.ZkStateReader$3.process Updating live
>> nodes... (10)
>> [junit4] 2> 448195 T2583 oascc.ZkStateReader$3.process Updating live
>> nodes... (10)
>> [junit4] 2> 448196 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 448198 T2593 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@156c3b
>> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
>> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 448196 T2591 oascc.ZkStateReader$3.process Updating live
>> nodes... (10)
>> [junit4] 2> 448196 T2585 oascc.ZkStateReader$3.process Updating live
>> nodes... (10)
>> [junit4] 2> 448196 T2581 oascc.ZkStateReader$3.process Updating live
>> nodes... (10)
>> [junit4] 2> 448198 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 448200 T2522
>> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
>> state from ZooKeeper...
>> [junit4] 2> 448203 T2522 oascc.SolrZkClient.makePath makePath:
>> /live_nodes/node10
>> [junit4] 2> 448207 T2575 oascc.ZkStateReader$3.process Updating live
>> nodes... (11)
>> [junit4] 2> 448207 T2583 oascc.ZkStateReader$3.process Updating live
>> nodes... (11)
>> [junit4] 2> 448208 T2579 oascc.ZkStateReader$3.process Updating live
>> nodes... (11)
>> [junit4] 2> 448207 T2577 oascc.ZkStateReader$3.process Updating live
>> nodes... (11)
>> [junit4] 2> 448207 T2573 oascc.ZkStateReader$3.process Updating live
>> nodes... (11)
>> [junit4] 2> 448207 T2589 oascc.ZkStateReader$3.process Updating live
>> nodes... (11)
>> [junit4] 2> 448207 T2587 oascc.ZkStateReader$3.process Updating live
>> nodes... (11)
>> [junit4] 2> 448209 T2585 oascc.ZkStateReader$3.process Updating live
>> nodes... (11)
>> [junit4] 2> 448208 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 448208 T2581 oascc.ZkStateReader$3.process Updating live
>> nodes... (11)
>> [junit4] 2> 448208 T2593 oascc.ZkStateReader$3.process Updating live
>> nodes... (11)
>> [junit4] 2> 448208 T2591 oascc.ZkStateReader$3.process Updating live
>> nodes... (11)
>> [junit4] 2> 448208 T2567 oascc.ZkStateReader$3.process Updating live
>> nodes... (11)
>> [junit4] 2> 448211 T2595 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@f1978e
>> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
>> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 448212 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 448213 T2522
>> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
>> state from ZooKeeper...
>> [junit4] 2> 448216 T2522 oascc.SolrZkClient.makePath makePath:
>> /live_nodes/node11
>> [junit4] 2> 448220 T2587 oascc.ZkStateReader$3.process Updating live
>> nodes... (12)
>> [junit4] 2> 448220 T2567 oascc.ZkStateReader$3.process Updating live
>> nodes... (12)
>> [junit4] 2> 448221 T2583 oascc.ZkStateReader$3.process Updating live
>> nodes... (12)
>> [junit4] 2> 448220 T2589 oascc.ZkStateReader$3.process Updating live
>> nodes... (12)
>> [junit4] 2> 448222 T2595 oascc.ZkStateReader$3.process Updating live
>> nodes... (12)
>> [junit4] 2> 448220 T2575 oascc.ZkStateReader$3.process Updating live
>> nodes... (12)
>> [junit4] 2> 448222 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 448222 T2593 oascc.ZkStateReader$3.process Updating live
>> nodes... (12)
>> [junit4] 2> 448222 T2581 oascc.ZkStateReader$3.process Updating live
>> nodes... (12)
>> [junit4] 2> 448224 T2597 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@175be88
>> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
>> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 448221 T2591 oascc.ZkStateReader$3.process Updating live
>> nodes... (12)
>> [junit4] 2> 448221 T2573 oascc.ZkStateReader$3.process Updating live
>> nodes... (12)
>> [junit4] 2> 448221 T2577 oascc.ZkStateReader$3.process Updating live
>> nodes... (12)
>> [junit4] 2> 448221 T2585 oascc.ZkStateReader$3.process Updating live
>> nodes... (12)
>> [junit4] 2> 448221 T2579 oascc.ZkStateReader$3.process Updating live
>> nodes... (12)
>> [junit4] 2> 448224 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 448228 T2522
>> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
>> state from ZooKeeper...
>> [junit4] 2> 448232 T2522 oascc.SolrZkClient.makePath makePath:
>> /live_nodes/node12
>> [junit4] 2> 448236 T2573 oascc.ZkStateReader$3.process Updating live
>> nodes... (13)
>> [junit4] 2> 448236 T2577 oascc.ZkStateReader$3.process Updating live
>> nodes... (13)
>> [junit4] 2> 448237 T2595 oascc.ZkStateReader$3.process Updating live
>> nodes... (13)
>> [junit4] 2> 448237 T2593 oascc.ZkStateReader$3.process Updating live
>> nodes... (13)
>> [junit4] 2> 448237 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 448238 T2581 oascc.ZkStateReader$3.process Updating live
>> nodes... (13)
>> [junit4] 2> 448238 T2597 oascc.ZkStateReader$3.process Updating live
>> nodes... (13)
>> [junit4] 2> 448237 T2583 oascc.ZkStateReader$3.process Updating live
>> nodes... (13)
>> [junit4] 2> 448239 T2599 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@dde0f4
>> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
>> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 448237 T2589 oascc.ZkStateReader$3.process Updating live
>> nodes... (13)
>> [junit4] 2> 448237 T2575 oascc.ZkStateReader$3.process Updating live
>> nodes... (13)
>> [junit4] 2> 448237 T2587 oascc.ZkStateReader$3.process Updating live
>> nodes... (13)
>> [junit4] 2> 448237 T2585 oascc.ZkStateReader$3.process Updating live
>> nodes... (13)
>> [junit4] 2> 448236 T2567 oascc.ZkStateReader$3.process Updating live
>> nodes... (13)
>> [junit4] 2> 448236 T2579 oascc.ZkStateReader$3.process Updating live
>> nodes... (13)
>> [junit4] 2> 448236 T2591 oascc.ZkStateReader$3.process Updating live
>> nodes... (13)
>> [junit4] 2> 448240 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 448244 T2522
>> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
>> state from ZooKeeper...
>> [junit4] 2> 448248 T2522 oascc.SolrZkClient.makePath makePath:
>> /live_nodes/node13
>> [junit4] 2> 448254 T2587 oascc.ZkStateReader$3.process Updating live
>> nodes... (14)
>> [junit4] 2> 448255 T2577 oascc.ZkStateReader$3.process Updating live
>> nodes... (14)
>> [junit4] 2> 448255 T2579 oascc.ZkStateReader$3.process Updating live
>> nodes... (14)
>> [junit4] 2> 448255 T2595 oascc.ZkStateReader$3.process Updating live
>> nodes... (14)
>> [junit4] 2> 448255 T2573 oascc.ZkStateReader$3.process Updating live
>> nodes... (14)
>> [junit4] 2> 448256 T2597 oascc.ZkStateReader$3.process Updating live
>> nodes... (14)
>> [junit4] 2> 448256 T2599 oascc.ZkStateReader$3.process Updating live
>> nodes... (14)
>> [junit4] 2> 448256 T2593 oascc.ZkStateReader$3.process Updating live
>> nodes... (14)
>> [junit4] 2> 448256 T2585 oascc.ZkStateReader$3.process Updating live
>> nodes... (14)
>> [junit4] 2> 448256 T2581 oascc.ZkStateReader$3.process Updating live
>> nodes... (14)
>> [junit4] 2> 448256 T2583 oascc.ZkStateReader$3.process Updating live
>> nodes... (14)
>> [junit4] 2> 448256 T2567 oascc.ZkStateReader$3.process Updating live
>> nodes... (14)
>> [junit4] 2> 448256 T2575 oascc.ZkStateReader$3.process Updating live
>> nodes... (14)
>> [junit4] 2> 448256 T2589 oascc.ZkStateReader$3.process Updating live
>> nodes... (14)
>> [junit4] 2> 448256 T2591 oascc.ZkStateReader$3.process Updating live
>> nodes... (14)
>> [junit4] 2> 448259 T2601 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@19e3760
>> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
>> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 448257 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 448262 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 448264 T2522
>> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
>> state from ZooKeeper...
>> [junit4] 2> 448268 T2522 oascc.SolrZkClient.makePath makePath:
>> /live_nodes/node14
>> [junit4] 2> 448273 T2573 oascc.ZkStateReader$3.process Updating live
>> nodes... (15)
>> [junit4] 2> 448273 T2587 oascc.ZkStateReader$3.process Updating live
>> nodes... (15)
>> [junit4] 2> 448273 T2579 oascc.ZkStateReader$3.process Updating live
>> nodes... (15)
>> [junit4] 2> 448273 T2591 oascc.ZkStateReader$3.process Updating live
>> nodes... (15)
>> [junit4] 2> 448273 T2577 oascc.ZkStateReader$3.process Updating live
>> nodes... (15)
>> [junit4] 2> 448273 T2595 oascc.ZkStateReader$3.process Updating live
>> nodes... (15)
>> [junit4] 2> 448275 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 448275 T2599 oascc.ZkStateReader$3.process Updating live
>> nodes... (15)
>> [junit4] 2> 448275 T2583 oascc.ZkStateReader$3.process Updating live
>> nodes... (15)
>> [junit4] 2> 448275 T2585 oascc.ZkStateReader$3.process Updating live
>> nodes... (15)
>> [junit4] 2> 448275 T2597 oascc.ZkStateReader$3.process Updating live
>> nodes... (15)
>> [junit4] 2> 448275 T2601 oascc.ZkStateReader$3.process Updating live
>> nodes... (15)
>> [junit4] 2> 448275 T2567 oascc.ZkStateReader$3.process Updating live
>> nodes... (15)
>> [junit4] 2> 448275 T2575 oascc.ZkStateReader$3.process Updating live
>> nodes... (15)
>> [junit4] 2> 448275 T2581 oascc.ZkStateReader$3.process Updating live
>> nodes... (15)
>> [junit4] 2> 448274 T2593 oascc.ZkStateReader$3.process Updating live
>> nodes... (15)
>> [junit4] 2> 448273 T2589 oascc.ZkStateReader$3.process Updating live
>> nodes... (15)
>> [junit4] 2> 448277 T2603 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@1818cfa
>> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
>> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 448281 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 448283 T2522
>> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
>> state from ZooKeeper...
>> [junit4] 2> 448286 T2522 oascc.SolrZkClient.makePath makePath:
>> /live_nodes/node15
>> [junit4] 2> 448291 T2577 oascc.ZkStateReader$3.process Updating live
>> nodes... (16)
>> [junit4] 2> 448292 T2573 oascc.ZkStateReader$3.process Updating live
>> nodes... (16)
>> [junit4] 2> 448291 T2587 oascc.ZkStateReader$3.process Updating live
>> nodes... (16)
>> [junit4] 2> 448293 T2579 oascc.ZkStateReader$3.process Updating live
>> nodes... (16)
>> [junit4] 2> 448293 T2575 oascc.ZkStateReader$3.process Updating live
>> nodes... (16)
>> [junit4] 2> 448293 T2601 oascc.ZkStateReader$3.process Updating live
>> nodes... (16)
>> [junit4] 2> 448294 T2599 oascc.ZkStateReader$3.process Updating live
>> nodes... (16)
>> [junit4] 2> 448293 T2591 oascc.ZkStateReader$3.process Updating live
>> nodes... (16)
>> [junit4] 2> 448293 T2597 oascc.ZkStateReader$3.process Updating live
>> nodes... (16)
>> [junit4] 2> 448293 T2583 oascc.ZkStateReader$3.process Updating live
>> nodes... (16)
>> [junit4] 2> 448293 T2567 oascc.ZkStateReader$3.process Updating live
>> nodes... (16)
>> [junit4] 2> 448293 T2589 oascc.ZkStateReader$3.process Updating live
>> nodes... (16)
>> [junit4] 2> 448295 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 448294 T2603 oascc.ZkStateReader$3.process Updating live
>> nodes... (16)
>> [junit4] 2> 448294 T2581 oascc.ZkStateReader$3.process Updating live
>> nodes... (16)
>> [junit4] 2> 448294 T2585 oascc.ZkStateReader$3.process Updating live
>> nodes... (16)
>> [junit4] 2> 448294 T2595 oascc.ZkStateReader$3.process Updating live
>> nodes... (16)
>> [junit4] 2> 448294 T2593 oascc.ZkStateReader$3.process Updating live
>> nodes... (16)
>> [junit4] 2> 448297 T2605 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@154e83c
>> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
>> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 448299 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 448300 T2522
>> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
>> state from ZooKeeper...
>> [junit4] 2> 448304 T2522 oascc.SolrZkClient.makePath makePath:
>> /live_nodes/node16
>> [junit4] 2> 448313 T2587 oascc.ZkStateReader$3.process Updating live
>> nodes... (17)
>> [junit4] 2> 448313 T2589 oascc.ZkStateReader$3.process Updating live
>> nodes... (17)
>> [junit4] 2> 448314 T2595 oascc.ZkStateReader$3.process Updating live
>> nodes... (17)
>> [junit4] 2> 448315 T2575 oascc.ZkStateReader$3.process Updating live
>> nodes... (17)
>> [junit4] 2> 448314 T2585 oascc.ZkStateReader$3.process Updating live
>> nodes... (17)
>> [junit4] 2> 448316 T2591 oascc.ZkStateReader$3.process Updating live
>> nodes... (17)
>> [junit4] 2> 448313 T2573 oascc.ZkStateReader$3.process Updating live
>> nodes... (17)
>> [junit4] 2> 448313 T2577 oascc.ZkStateReader$3.process Updating live
>> nodes... (17)
>> [junit4] 2> 448313 T2593 oascc.ZkStateReader$3.process Updating live
>> nodes... (17)
>> [junit4] 2> 448317 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 448316 T2599 oascc.ZkStateReader$3.process Updating live
>> nodes... (17)
>> [junit4] 2> 448316 T2581 oascc.ZkStateReader$3.process Updating live
>> nodes... (17)
>> [junit4] 2> 448316 T2605 oascc.ZkStateReader$3.process Updating live
>> nodes... (17)
>> [junit4] 2> 448316 T2603 oascc.ZkStateReader$3.process Updating live
>> nodes... (17)
>> [junit4] 2> 448316 T2583 oascc.ZkStateReader$3.process Updating live
>> nodes... (17)
>> [junit4] 2> 448316 T2567 oascc.ZkStateReader$3.process Updating live
>> nodes... (17)
>> [junit4] 2> 448315 T2601 oascc.ZkStateReader$3.process Updating live
>> nodes... (17)
>> [junit4] 2> 448315 T2597 oascc.ZkStateReader$3.process Updating live
>> nodes... (17)
>> [junit4] 2> 448315 T2579 oascc.ZkStateReader$3.process Updating live
>> nodes... (17)
>> [junit4] 2> 448319 T2607 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@d963a2
>> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
>> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 448323 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 448324 T2522
>> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
>> state from ZooKeeper...
>> [junit4] 2> 448328 T2522 oascc.SolrZkClient.makePath makePath:
>> /live_nodes/node17
>> [junit4] 2> 448333 T2573 oascc.ZkStateReader$3.process Updating live
>> nodes... (18)
>> [junit4] 2> 448333 T2577 oascc.ZkStateReader$3.process Updating live
>> nodes... (18)
>> [junit4] 2> 448334 T2601 oascc.ZkStateReader$3.process Updating live
>> nodes... (18)
>> [junit4] 2> 448334 T2607 oascc.ZkStateReader$3.process Updating live
>> nodes... (18)
>> [junit4] 2> 448335 T2567 oascc.ZkStateReader$3.process Updating live
>> nodes... (18)
>> [junit4] 2> 448335 T2583 oascc.ZkStateReader$3.process Updating live
>> nodes... (18)
>> [junit4] 2> 448335 T2603 oascc.ZkStateReader$3.process Updating live
>> nodes... (18)
>> [junit4] 2> 448334 T2589 oascc.ZkStateReader$3.process Updating live
>> nodes... (18)
>> [junit4] 2> 448336 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 448334 T2595 oascc.ZkStateReader$3.process Updating live
>> nodes... (18)
>> [junit4] 2> 448334 T2579 oascc.ZkStateReader$3.process Updating live
>> nodes... (18)
>> [junit4] 2> 448334 T2575 oascc.ZkStateReader$3.process Updating live
>> nodes... (18)
>> [junit4] 2> 448338 T2609 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@826a2e
>> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
>> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 448333 T2587 oascc.ZkStateReader$3.process Updating live
>> nodes... (18)
>> [junit4] 2> 448333 T2591 oascc.ZkStateReader$3.process Updating live
>> nodes... (18)
>> [junit4] 2> 448338 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 448336 T2585 oascc.ZkStateReader$3.process Updating live
>> nodes... (18)
>> [junit4] 2> 448335 T2593 oascc.ZkStateReader$3.process Updating live
>> nodes... (18)
>> [junit4] 2> 448335 T2599 oascc.ZkStateReader$3.process Updating live
>> nodes... (18)
>> [junit4] 2> 448335 T2581 oascc.ZkStateReader$3.process Updating live
>> nodes... (18)
>> [junit4] 2> 448335 T2605 oascc.ZkStateReader$3.process Updating live
>> nodes... (18)
>> [junit4] 2> 448335 T2597 oascc.ZkStateReader$3.process Updating live
>> nodes... (18)
>> [junit4] 2> 448341 T2522
>> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
>> state from ZooKeeper...
>> [junit4] 2> 448346 T2522 oascc.SolrZkClient.makePath makePath:
>> /live_nodes/node18
>> [junit4] 2> 448350 T2573 oascc.ZkStateReader$3.process Updating live
>> nodes... (19)
>> [junit4] 2> 448350 T2577 oascc.ZkStateReader$3.process Updating live
>> nodes... (19)
>> [junit4] 2> 448351 T2607 oascc.ZkStateReader$3.process Updating live
>> nodes... (19)
>> [junit4] 2> 448352 T2587 oascc.ZkStateReader$3.process Updating live
>> nodes... (19)
>> [junit4] 2> 448352 T2593 oascc.ZkStateReader$3.process Updating live
>> nodes... (19)
>> [junit4] 2> 448351 T2601 oascc.ZkStateReader$3.process Updating live
>> nodes... (19)
>> [junit4] 2> 448353 T2579 oascc.ZkStateReader$3.process Updating live
>> nodes... (19)
>> [junit4] 2> 448353 T2581 oascc.ZkStateReader$3.process Updating live
>> nodes... (19)
>> [junit4] 2> 448351 T2595 oascc.ZkStateReader$3.process Updating live
>> nodes... (19)
>> [junit4] 2> 448353 T2605 oascc.ZkStateReader$3.process Updating live
>> nodes... (19)
>> [junit4] 2> 448354 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 448353 T2609 oascc.ZkStateReader$3.process Updating live
>> nodes... (19)
>> [junit4] 2> 448353 T2603 oascc.ZkStateReader$3.process Updating live
>> nodes... (19)
>> [junit4] 2> 448353 T2597 oascc.ZkStateReader$3.process Updating live
>> nodes... (19)
>> [junit4] 2> 448353 T2589 oascc.ZkStateReader$3.process Updating live
>> nodes... (19)
>> [junit4] 2> 448353 T2567 oascc.ZkStateReader$3.process Updating live
>> nodes... (19)
>> [junit4] 2> 448353 T2591 oascc.ZkStateReader$3.process Updating live
>> nodes... (19)
>> [junit4] 2> 448353 T2575 oascc.ZkStateReader$3.process Updating live
>> nodes... (19)
>> [junit4] 2> 448352 T2599 oascc.ZkStateReader$3.process Updating live
>> nodes... (19)
>> [junit4] 2> 448352 T2583 oascc.ZkStateReader$3.process Updating live
>> nodes... (19)
>> [junit4] 2> 448352 T2585 oascc.ZkStateReader$3.process Updating live
>> nodes... (19)
>> [junit4] 2> 448356 T2611 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@1b5c498
>> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
>> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 448359 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 448362 T2522
>> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
>> state from ZooKeeper...
>> [junit4] 2> 448366 T2522 oascc.SolrZkClient.makePath makePath:
>> /live_nodes/node19
>> [junit4] 2> 448371 T2587 oascc.ZkStateReader$3.process Updating live
>> nodes... (20)
>> [junit4] 2> 448371 T2591 oascc.ZkStateReader$3.process Updating live
>> nodes... (20)
>> [junit4] 2> 448372 T2579 oascc.ZkStateReader$3.process Updating live
>> nodes... (20)
>> [junit4] 2> 448372 T2575 oascc.ZkStateReader$3.process Updating live
>> nodes... (20)
>> [junit4] 2> 448372 T2577 oascc.ZkStateReader$3.process Updating live
>> nodes... (20)
>> [junit4] 2> 448373 T2581 oascc.ZkStateReader$3.process Updating live
>> nodes... (20)
>> [junit4] 2> 448372 T2589 oascc.ZkStateReader$3.process Updating live
>> nodes... (20)
>> [junit4] 2> 448372 T2573 oascc.ZkStateReader$3.process Updating live
>> nodes... (20)
>> [junit4] 2> 448372 T2595 oascc.ZkStateReader$3.process Updating live
>> nodes... (20)
>> [junit4] 2> 448374 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 448373 T2609 oascc.ZkStateReader$3.process Updating live
>> nodes... (20)
>> [junit4] 2> 448373 T2611 oascc.ZkStateReader$3.process Updating live
>> nodes... (20)
>> [junit4] 2> 448373 T2603 oascc.ZkStateReader$3.process Updating live
>> nodes... (20)
>> [junit4] 2> 448373 T2585 oascc.ZkStateReader$3.process Updating live
>> nodes... (20)
>> [junit4] 2> 448373 T2593 oascc.ZkStateReader$3.process Updating live
>> nodes... (20)
>> [junit4] 2> 448373 T2567 oascc.ZkStateReader$3.process Updating live
>> nodes... (20)
>> [junit4] 2> 448373 T2607 oascc.ZkStateReader$3.process Updating live
>> nodes... (20)
>> [junit4] 2> 448372 T2599 oascc.ZkStateReader$3.process Updating live
>> nodes... (20)
>> [junit4] 2> 448372 T2597 oascc.ZkStateReader$3.process Updating live
>> nodes... (20)
>> [junit4] 2> 448372 T2601 oascc.ZkStateReader$3.process Updating live
>> nodes... (20)
>> [junit4] 2> 448372 T2583 oascc.ZkStateReader$3.process Updating live
>> nodes... (20)
>> [junit4] 2> 448372 T2605 oascc.ZkStateReader$3.process Updating live
>> nodes... (20)
>> [junit4] 2> 448376 T2613 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@18ed79a
>> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
>> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 448380 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 448382 T2522
>> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
>> state from ZooKeeper...
>> [junit4] 2> 448385 T2522 oascc.SolrZkClient.makePath makePath:
>> /live_nodes/node20
>> [junit4] 2> 448390 T2577 oascc.ZkStateReader$3.process Updating live
>> nodes... (21)
>> [junit4] 2> 448390 T2579 oascc.ZkStateReader$3.process Updating live
>> nodes... (21)
>> [junit4] 2> 448391 T2589 oascc.ZkStateReader$3.process Updating live
>> nodes... (21)
>> [junit4] 2> 448391 T2601 oascc.ZkStateReader$3.process Updating live
>> nodes... (21)
>> [junit4] 2> 448391 T2599 oascc.ZkStateReader$3.process Updating live
>> nodes... (21)
>> [junit4] 2> 448391 T2573 oascc.ZkStateReader$3.process Updating live
>> nodes... (21)
>> [junit4] 2> 448392 T2581 oascc.ZkStateReader$3.process Updating live
>> nodes... (21)
>> [junit4] 2> 448391 T2605 oascc.ZkStateReader$3.process Updating live
>> nodes... (21)
>> [junit4] 2> 448390 T2591 oascc.ZkStateReader$3.process Updating live
>> nodes... (21)
>> [junit4] 2> 448390 T2595 oascc.ZkStateReader$3.process Updating live
>> nodes... (21)
>> [junit4] 2> 448393 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 448392 T2603 oascc.ZkStateReader$3.process Updating live
>> nodes... (21)
>> [junit4] 2> 448392 T2609 oascc.ZkStateReader$3.process Updating live
>> nodes... (21)
>> [junit4] 2> 448392 T2611 oascc.ZkStateReader$3.process Updating live
>> nodes... (21)
>> [junit4] 2> 448392 T2593 oascc.ZkStateReader$3.process Updating live
>> nodes... (21)
>> [junit4] 2> 448392 T2585 oascc.ZkStateReader$3.process Updating live
>> nodes... (21)
>> [junit4] 2> 448392 T2567 oascc.ZkStateReader$3.process Updating live
>> nodes... (21)
>> [junit4] 2> 448392 T2575 oascc.ZkStateReader$3.process Updating live
>> nodes... (21)
>> [junit4] 2> 448391 T2613 oascc.ZkStateReader$3.process Updating live
>> nodes... (21)
>> [junit4] 2> 448391 T2583 oascc.ZkStateReader$3.process Updating live
>> nodes... (21)
>> [junit4] 2> 448391 T2607 oascc.ZkStateReader$3.process Updating live
>> nodes... (21)
>> [junit4] 2> 448391 T2597 oascc.ZkStateReader$3.process Updating live
>> nodes... (21)
>> [junit4] 2> 448391 T2587 oascc.ZkStateReader$3.process Updating live
>> nodes... (21)
>> [junit4] 2> 448395 T2615 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@5f6bba
>> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
>> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 448399 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 448401 T2522
>> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
>> state from ZooKeeper...
>> [junit4] 2> 448404 T2522 oascc.SolrZkClient.makePath makePath:
>> /live_nodes/node21
>> [junit4] 2> 448409 T2587 oascc.ZkStateReader$3.process Updating live
>> nodes... (22)
>> [junit4] 2> 448410 T2573 oascc.ZkStateReader$3.process Updating live
>> nodes... (22)
>> [junit4] 2> 448410 T2595 oascc.ZkStateReader$3.process Updating live
>> nodes... (22)
>> [junit4] 2> 448410 T2577 oascc.ZkStateReader$3.process Updating live
>> nodes... (22)
>> [junit4] 2> 448409 T2591 oascc.ZkStateReader$3.process Updating live
>> nodes... (22)
>> [junit4] 2> 448412 T2607 oascc.ZkStateReader$3.process Updating live
>> nodes... (22)
>> [junit4] 2> 448412 T2593 oascc.ZkStateReader$3.process Updating live
>> nodes... (22)
>> [junit4] 2> 448412 T2599 oascc.ZkStateReader$3.process Updating live
>> nodes... (22)
>> [junit4] 2> 448413 T2603 oascc.ZkStateReader$3.process Updating live
>> nodes... (22)
>> [junit4] 2> 448413 T2609 oascc.ZkStateReader$3.process Updating live
>> nodes... (22)
>> [junit4] 2> 448412 T2567 oascc.ZkStateReader$3.process Updating live
>> nodes... (22)
>> [junit4] 2> 448412 T2613 oascc.ZkStateReader$3.process Updating live
>> nodes... (22)
>> [junit4] 2> 448412 T2575 oascc.ZkStateReader$3.process Updating live
>> nodes... (22)
>> [junit4] 2> 448411 T2583 oascc.ZkStateReader$3.process Updating live
>> nodes... (22)
>> [junit4] 2> 448411 T2601 oascc.ZkStateReader$3.process Updating live
>> nodes... (22)
>> [junit4] 2> 448411 T2597 oascc.ZkStateReader$3.process Updating live
>> nodes... (22)
>> [junit4] 2> 448411 T2579 oascc.ZkStateReader$3.process Updating live
>> nodes... (22)
>> [junit4] 2> 448411 T2605 oascc.ZkStateReader$3.process Updating live
>> nodes... (22)
>> [junit4] 2> 448411 T2589 oascc.ZkStateReader$3.process Updating live
>> nodes... (22)
>> [junit4] 2> 448416 T2617 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@13fabc5
>> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
>> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 448414 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 448413 T2615 oascc.ZkStateReader$3.process Updating live
>> nodes... (22)
>> [junit4] 2> 448413 T2611 oascc.ZkStateReader$3.process Updating live
>> nodes... (22)
>> [junit4] 2> 448412 T2581 oascc.ZkStateReader$3.process Updating live
>> nodes... (22)
>> [junit4] 2> 448412 T2585 oascc.ZkStateReader$3.process Updating live
>> nodes... (22)
>> [junit4] 2> 448418 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 448421 T2522
>> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
>> state from ZooKeeper...
>> [junit4] 2> 448426 T2522 oascc.SolrZkClient.makePath makePath:
>> /live_nodes/node22
>> [junit4] 2> 448431 T2587 oascc.ZkStateReader$3.process Updating live
>> nodes... (23)
>> [junit4] 2> 448432 T2577 oascc.ZkStateReader$3.process Updating live
>> nodes... (23)
>> [junit4] 2> 448432 T2579 oascc.ZkStateReader$3.process Updating live
>> nodes... (23)
>> [junit4] 2> 448432 T2605 oascc.ZkStateReader$3.process Updating live
>> nodes... (23)
>> [junit4] 2> 448433 T2611 oascc.ZkStateReader$3.process Updating live
>> nodes... (23)
>> [junit4] 2> 448432 T2595 oascc.ZkStateReader$3.process Updating live
>> nodes... (23)
>> [junit4] 2> 448432 T2591 oascc.ZkStateReader$3.process Updating live
>> nodes... (23)
>> [junit4] 2> 448434 T2609 oascc.ZkStateReader$3.process Updating live
>> nodes... (23)
>> [junit4] 2> 448432 T2573 oascc.ZkStateReader$3.process Updating live
>> nodes... (23)
>> [junit4] 2> 448435 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 448434 T2603 oascc.ZkStateReader$3.process Updating live
>> nodes... (23)
>> [junit4] 2> 448434 T2617 oascc.ZkStateReader$3.process Updating live
>> nodes... (23)
>> [junit4] 2> 448434 T2581 oascc.ZkStateReader$3.process Updating live
>> nodes... (23)
>> [junit4] 2> 448437 T2619 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@71217e
>> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
>> WatchedEvent state:SyncConnected type:None path:null
>>
>> [...truncated too long message...]
>>
>> queue/qn-
>> [junit4] 2> at
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
>> [junit4] 2> at
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>> [junit4] 2> at
>> org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
>> [junit4] 2> at
>> org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:243)
>> [junit4] 2> at
>> org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:240)
>> [junit4] 2> at
>> org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
>> [junit4] 2> at
>> org.apache.solr.common.cloud.SolrZkClient.create(SolrZkClient.java:240)
>> [junit4] 2> at
>> org.apache.solr.cloud.DistributedQueue.createData(DistributedQueue.java:346)
>> [junit4] 2> at
>> org.apache.solr.cloud.DistributedQueue.offer(DistributedQueue.java:330)
>> [junit4] 2> at
>> org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:198)
>> [junit4] 2> at
>> org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:163)
>> [junit4] 2> at
>> org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:125)
>> [junit4] 2> at
>> org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:314)
>> [junit4] 2> at
>> org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:221)
>> [junit4] 2> at
>> org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:452)
>> [junit4] 2> at
>> org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:216)
>> [junit4] 2> at
>> org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:163)
>> [junit4] 2> at
>> org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:125)
>> [junit4] 2> at
>> org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
>> [junit4] 2> at
>> org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:358)
>> [junit4] 2> at
>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
>> [junit4] 2> at
>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
>> [junit4] 2>
>> [junit4] 2> 750036 T3044 oascc.ZkStateReader$3.process WARN
>> ZooKeeper watch triggered, but Solr cannot talk to ZK
>> [junit4] 2> 750037 T3044 oasc.LeaderElector$ElectionWatcher.process
>> WARN org.apache.zookeeper.KeeperException$SessionExpiredException:
>> KeeperErrorCode = Session expired for /overseer_elect/election
>> [junit4] 2> at
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
>> [junit4] 2> at
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>> [junit4] 2> at
>> org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
>> [junit4] 2> at
>> org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
>> [junit4] 2> at
>> org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
>> [junit4] 2> at
>> org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
>> [junit4] 2> at
>> org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
>> [junit4] 2> at
>> org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
>> [junit4] 2> at
>> org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
>> [junit4] 2> at
>> org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:358)
>> [junit4] 2> at
>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
>> [junit4] 2> at
>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
>> [junit4] 2>
>> [junit4] 2> 750038 T3044 oascc.ZkStateReader$2.process A cluster
>> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
>> [junit4] 2> 750038 T3044 oascc.ZkStateReader$2.process WARN
>> ZooKeeper watch triggered, but Solr cannot talk to ZK
>> [junit4] 2> 750121 T3116 oascc.ZkStateReader$3.process WARN
>> ZooKeeper watch triggered, but Solr cannot talk to ZK
>> [junit4] 2> 750206 T2787 oascc.ZkStateReader$3.process WARN
>> ZooKeeper watch triggered, but Solr cannot talk to ZK
>> [junit4] 2> 750206 T2787
>> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
>> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>> [junit4] 2> 750207 T2787
>> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
>> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>> [junit4] 2> 750207 T2787
>> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
>> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>> [junit4] 2> 750207 T2787
>> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
>> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>> [junit4] 2> 750206 T3156
>> oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not
>> read the data org.apache.zookeeper.KeeperException$SessionExpiredException:
>> KeeperErrorCode = Session expired for /overseer_elect/leader
>> [junit4] 2> at
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
>> [junit4] 2> at
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>> [junit4] 2> at
>> org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
>> [junit4] 2> at
>> org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
>> [junit4] 2> at
>> org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
>> [junit4] 2> at
>> org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
>> [junit4] 2> at
>> org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
>> [junit4] 2> at
>> org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:316)
>> [junit4] 2> at
>> org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
>> [junit4] 2> at
>> org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:288)
>> [junit4] 2>
>> [junit4] 2> NOTE: leaving temporary files on disk at:
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-A819205877C572F6-001
>> [junit4] 2> NOTE: test params are: codec=Lucene41,
>> sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=ar_JO,
>> timezone=Africa/Ndjamena
>> [junit4] 2> NOTE: Linux 3.8.0-42-generic i386/Oracle Corporation
>> 1.8.0_05 (32-bit)/cpus=8,threads=1,free=118175464,total=317980672
>> [junit4] 2> NOTE: All tests run in this JVM: [TestRecovery,
>> OpenCloseCoreStressTest, HardAutoCommitTest, DeleteReplicaTest,
>> IndexSchemaRuntimeFieldTest, TestXIncludeConfig, TestFreeTextSuggestions,
>> CollectionsAPIDistributedZkTest, TestFieldTypeResource, BadCopyFieldTest,
>> CopyFieldTest, TestCSVLoader, UniqFieldsUpdateProcessorFactoryTest,
>> TestIndexingPerformance, TestDocumentBuilder, AutoCommitTest,
>> AtomicUpdatesTest, TestQueryTypes, TestCollapseQParserPlugin,
>> BasicDistributedZk2Test, TestWriterPerf, TestCoreContainer,
>> TestComplexPhraseQParserPlugin, TestDistribDocBasedVersion,
>> TestFastLRUCache, TestDynamicFieldResource, DOMUtilTest,
>> QueryResultKeyTest, DistribDocExpirationUpdateProcessorTest,
>> UpdateRequestProcessorFactoryTest, TimeZoneUtilsTest,
>> SchemaVersionSpecificBehaviorTest, PreAnalyzedUpdateProcessorTest,
>> SOLR749Test, TestNonDefinedSimilarityFactory, TestFileDictionaryLookup,
>> MultiThreadedOCPTest, TestRTGBase, QueryParsingTest, OutputWriterTest,
>> LoggingHandlerTest, TestInfoStreamLogging, TestPseudoReturnFields,
>> TestFastWriter, TestReplicationHandlerBackup, AssignTest,
>> TermVectorComponentTest, SolrRequestParserTest, TestPerFieldSimilarity,
>> TestAtomicUpdateErrorCases, TestManagedResource,
>> TermVectorComponentDistributedTest, TestReRankQParserPlugin,
>> TestSerializedLuceneMatchVersion, TestChildDocTransformer,
>> TestNoOpRegenerator, TestGroupingSearch, TestFieldTypeCollectionResource,
>> TestBinaryField, StatsComponentTest, CursorMarkTest, BasicZkTest,
>> SuggesterFSTTest, BJQParserTest, CurrencyFieldXmlFileTest,
>> DistributedQueryElevationComponentTest, TestDistributedMissingSort,
>> SearchHandlerTest, TestLFUCache, TestSolrJ, FileBasedSpellCheckerTest,
>> TestRealTimeGet, RequestHandlersTest, AnalysisAfterCoreReloadTest,
>> SolrInfoMBeanTest, TestDistributedGrouping, TestExtendedDismaxParser,
>> URLClassifyProcessorTest, SuggestComponentTest, ChaosMonkeySafeLeaderTest]
>> [junit4] Completed on J1 in 74.71s, 1 test, 1 failure <<< FAILURES!
>>
>> [...truncated 59 lines...]
>> BUILD FAILED
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:467: The
>> following error occurred while executing this line:
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:447: The
>> following error occurred while executing this line:
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:45: The
>> following error occurred while executing this line:
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37:
>> The following error occurred while executing this line:
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The
>> following error occurred while executing this line:
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:496:
>> The following error occurred while executing this line:
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1299:
>> The following error occurred while executing this line:
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:923:
>> There were test failures: 405 suites, 1656 tests, 3 suite-level errors, 1
>> error, 1 failure, 954 ignored (13 assumptions)
>>
>> Total time: 65 minutes 33 seconds
>> Build step 'Invoke Ant' marked build as failure
>> [description-setter] Description set: Java: 32bit/jdk1.8.0_05 -client
>> -XX:+UseParallelGC
>> Archiving artifacts
>> Recording test results
>> Email was triggered for: Failure - Any
>> Sending email for trigger: Failure - Any
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
>
>
--
Regards,
Shalin Shekhar Mangar.
Re: [JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.8.0_05) - Build
# 10616 - Still Failing!
Posted by Mark Miller <ma...@gmail.com>.
On July 7, 2014 at 11:39:31 AM, Shalin Shekhar Mangar (shalinmangar@gmail.com) wrote:
I was looking into the OverseerTest failure and the following caught my attention.
This error can be due to a slow machine but it also happens on fast machines if you try to do a lot of writes very fast on ZooKeeper which is what the testShardLeaderChange does. Perhaps we should add a small wait between operations?
It’s probably worth a try. I tried once to work around it via ZooKeeper configuration options, thought there was a promising path or two after some googling, but I don’t think I ended up getting anywhere.
- Mark
[junit4] 2> 472241 T2893 oazsp.FileTxnLog.commit WARN fsync-ing the write ahead log in SyncThread:0 took 11588ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
On Sun, Jul 6, 2014 at 10:46 AM, Policeman Jenkins Server <je...@thetaphi.de> wrote:
Build:Â http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/10616/
Java: 32bit/jdk1.8.0_05 -client -XX:+UseParallelGC
5 tests failed.
FAILED: Â junit.framework.TestSuite.org.apache.solr.cloud.OverseerTest
Error Message:
ERROR: SolrZkClient opens=165 closes=164
Stack Trace:
java.lang.AssertionError: ERROR: SolrZkClient opens=165 closes=164
    at __randomizedtesting.SeedInfo.seed([A819205877C572F6]:0)
    at org.junit.Assert.fail(Assert.java:93)
    at org.apache.solr.SolrTestCaseJ4.endTrackingZkClients(SolrTestCaseJ4.java:436)
    at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:182)
    at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:483)
    at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:790)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
    at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
    at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
    at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
    at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
    at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
    at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
    at java.lang.Thread.run(Thread.java:745)
FAILED: Â junit.framework.TestSuite.org.apache.solr.cloud.OverseerTest
Error Message:
2 threads leaked from SUITE scope at org.apache.solr.cloud.OverseerTest: Â Â 1) Thread[id=2969, name=TEST-OverseerTest.testShardLeaderChange-seed#[A819205877C572F6]-SendThread(127.0.0.1:33880), state=TIMED_WAITING, group=TGRP-OverseerTest] Â Â Â Â at java.lang.Thread.sleep(Native Method) Â Â Â Â at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101) Â Â Â Â at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940) Â Â Â Â at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003) Â Â 2) Thread[id=2970, name=TEST-OverseerTest.testShardLeaderChange-seed#[A819205877C572F6]-EventThread, state=WAITING, group=TGRP-OverseerTest] Â Â Â Â at sun.misc.Unsafe.park(Native Method) Â Â Â Â at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) Â Â Â Â at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) Â Â Â Â at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) Â Â Â Â at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.OverseerTest:
  1) Thread[id=2969, name=TEST-OverseerTest.testShardLeaderChange-seed#[A819205877C572F6]-SendThread(127.0.0.1:33880), state=TIMED_WAITING, group=TGRP-OverseerTest]
    at java.lang.Thread.sleep(Native Method)
    at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
    at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
  2) Thread[id=2970, name=TEST-OverseerTest.testShardLeaderChange-seed#[A819205877C572F6]-EventThread, state=WAITING, group=TGRP-OverseerTest]
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
    at __randomizedtesting.SeedInfo.seed([A819205877C572F6]:0)
FAILED: Â junit.framework.TestSuite.org.apache.solr.cloud.OverseerTest
Error Message:
There are still zombie threads that couldn't be terminated: Â Â 1) Thread[id=2969, name=TEST-OverseerTest.testShardLeaderChange-seed#[A819205877C572F6]-SendThread(127.0.0.1:33880), state=TIMED_WAITING, group=TGRP-OverseerTest] Â Â Â Â at java.lang.Thread.sleep(Native Method) Â Â Â Â at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101) Â Â Â Â at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940) Â Â Â Â at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
  1) Thread[id=2969, name=TEST-OverseerTest.testShardLeaderChange-seed#[A819205877C572F6]-SendThread(127.0.0.1:33880), state=TIMED_WAITING, group=TGRP-OverseerTest]
    at java.lang.Thread.sleep(Native Method)
    at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
    at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
    at __randomizedtesting.SeedInfo.seed([A819205877C572F6]:0)
REGRESSION: Â org.apache.solr.cloud.OverseerTest.testShardLeaderChange
Error Message:
KeeperErrorCode = NodeExists for /live_nodes/node1
Stack Trace:
org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /live_nodes/node1
    at __randomizedtesting.SeedInfo.seed([A819205877C572F6:764AA7AF6D5D8707]:0)
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
    at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
    at org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:432)
    at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
    at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:429)
    at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:386)
    at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:373)
    at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:350)
    at org.apache.solr.cloud.OverseerTest$MockZKController.<init>(OverseerTest.java:85)
    at org.apache.solr.cloud.OverseerTest.testShardLeaderChange(OverseerTest.java:746)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:483)
    at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
    at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
    at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
    at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
    at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
    at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
    at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
    at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
    at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
    at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
    at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
    at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
    at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
    at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
    at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
    at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
    at java.lang.Thread.run(Thread.java:745)
FAILED: Â org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch
Error Message:
shard2 is not consistent.  Got 153 from http://127.0.0.1:51995/collection1lastClient and got 132 from http://127.0.0.1:36739/collection1
Stack Trace:
java.lang.AssertionError: shard2 is not consistent.  Got 153 from http://127.0.0.1:51995/collection1lastClient and got 132 from http://127.0.0.1:36739/collection1
    at __randomizedtesting.SeedInfo.seed([A819205877C572F6:29FFAE40009A12CA]:0)
    at org.junit.Assert.fail(Assert.java:93)
    at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1139)
    at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1118)
    at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:150)
    at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:483)
    at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
    at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
    at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
    at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
    at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
    at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
    at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
    at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
    at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
    at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
    at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
    at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
    at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
    at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
    at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
    at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
    at java.lang.Thread.run(Thread.java:745)
Build Log:
[...truncated 11133 lines...]
  [junit4] Suite: org.apache.solr.cloud.OverseerTest
  [junit4]  2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.OverseerTest-A819205877C572F6-001/init-core-data-001
  [junit4]  2> 443102 T2522 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
  [junit4]  2> 443103 T2522 oas.SolrTestCaseJ4.initCore ####initCore
  [junit4]  2> 443103 T2522 oas.SolrTestCaseJ4.initCore ####initCore end
  [junit4]  2> 443107 T2522 oas.SolrTestCaseJ4.setUp ###Starting testStateChange
  [junit4]  2> 443107 T2522 oasc.ZkTestServer.run STARTING ZK TEST SERVER
  [junit4]  1> client port:0.0.0.0/0.0.0.0:0
  [junit4]  2> 443108 T2523 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
  [junit4]  2> 443208 T2522 oasc.ZkTestServer.run start zk server on port:45124
  [junit4]  2> 443209 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 443282 T2529 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ed72a5 name:ZooKeeperConnection Watcher:127.0.0.1:45124/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 443283 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 443284 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 443287 T2531 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6366c4 name:ZooKeeperConnection Watcher:127.0.0.1:45124 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 443287 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 443290 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 443292 T2533 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@60240b name:ZooKeeperConnection Watcher:127.0.0.1:45124 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 443292 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 443293 T2522 oascc.SolrZkClient.makePath makePath: /solr
  [junit4]  2> 443298 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes
  [junit4]  2> 443301 T2522 oascc.SolrZkClient.makePath makePath: /clusterstate.json
  [junit4]  2> 443304 T2522 oascc.SolrZkClient.makePath makePath: /aliases.json
  [junit4]  2> 443306 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
  [junit4]  2> 443311 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 443312 T2535 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@68e786 name:ZooKeeperConnection Watcher:127.0.0.1:45124/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 443313 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 443314 T2522 oascc.SolrZkClient.makePath makePath: /overseer_elect
  [junit4]  2> 443318 T2522 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
  [junit4]  2> 443324 T2522 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:45124_solr
  [junit4]  2> 443324 T2522 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
  [junit4]  2> 443328 T2522 oasc.Overseer.start Overseer (id=92053395863699459-127.0.0.1:45124_solr-n_0000000000) starting
  [junit4]  2> 443331 T2522 oascc.SolrZkClient.makePath makePath: /overseer/queue
  [junit4]  2> 443336 T2522 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
  [junit4]  2> 443340 T2522 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
  [junit4]  2> 443345 T2522 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
  [junit4]  2> 443350 T2522 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
  [junit4]  2> 443357 T2522 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
  [junit4]  2> 443365 T2537 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
  [junit4]  2> 443366 T2536 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
  [junit4]  2> 443368 T2522 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
  [junit4]  2> 443369 T2536 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
  [junit4]  2> 443370 T2536 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
  [junit4]  2>      "operation":"state",
  [junit4]  2>      "base_url":"http://127.0.0.1/solr",
  [junit4]  2>      "node_name":"node1",
  [junit4]  2>      "collection":"collection1",
  [junit4]  2>      "core":"core1",
  [junit4]  2>      "roles":"",
  [junit4]  2>      "state":"recovering"}
  [junit4]  2> 443370 T2536 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
  [junit4]  2> 443374 T2535 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
  [junit4]  2> 443375 T2529 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
  [junit4]  2> 443420 T2522 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
  [junit4]  2> 443422 T2535 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
  [junit4]  2> 443424 T2536 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
  [junit4]  2> 443425 T2536 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
  [junit4]  2>      "operation":"state",
  [junit4]  2>      "base_url":"http://127.0.0.1/solr",
  [junit4]  2>      "node_name":"node1",
  [junit4]  2>      "collection":"collection1",
  [junit4]  2>      "core":"core1",
  [junit4]  2>      "roles":"",
  [junit4]  2>      "state":"active"}
  [junit4]  2> 443426 T2536 oasc.Overseer$ClusterStateUpdater.updateState node=core_node1 is already registered
  [junit4]  2> 443426 T2536 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
  [junit4]  2> 443431 T2535 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
  [junit4]  2> 443534 T2529 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
  [junit4]  2> 443575 T2522 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:45124 45124
  [junit4]  2> 443988 T2523 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:45124 45124
  [junit4]  2> 443991 T2522 oas.SolrTestCaseJ4.tearDown ###Ending testStateChange
  [junit4]  2> 443992 T2536 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:45124_solr
  [junit4]  2> 443994 T2538 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
  [junit4]  2>     at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
  [junit4]  2>     at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
  [junit4]  2>     at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
  [junit4]  2>     at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
  [junit4]  2>     at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
  [junit4]  2>     at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
  [junit4]  2>     at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
  [junit4]  2>     at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:316)
  [junit4]  2>     at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
  [junit4]  2>     at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:288)
  [junit4]  2>
  [junit4]  2> 444000 T2522 oas.SolrTestCaseJ4.setUp ###Starting testShardAssignment
  [junit4]  2> 444001 T2522 oasc.ZkTestServer.run STARTING ZK TEST SERVER
  [junit4]  1> client port:0.0.0.0/0.0.0.0:0
  [junit4]  2> 444002 T2539 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
  [junit4]  2> 444101 T2522 oasc.ZkTestServer.run start zk server on port:57742
  [junit4]  2> 444103 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 444107 T2545 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@9fc604 name:ZooKeeperConnection Watcher:127.0.0.1:57742 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 444108 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 444111 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 444112 T2547 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@104f2e0 name:ZooKeeperConnection Watcher:127.0.0.1:57742 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 444113 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 444114 T2522 oascc.SolrZkClient.makePath makePath: /solr
  [junit4]  2> 444119 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 444121 T2549 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@896251 name:ZooKeeperConnection Watcher:127.0.0.1:57742/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 444121 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 444122 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes
  [junit4]  2> 444125 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 444127 T2551 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@10c3e42 name:ZooKeeperConnection Watcher:127.0.0.1:57742/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 444127 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 444128 T2522 oascc.SolrZkClient.makePath makePath: /overseer_elect
  [junit4]  2> 444132 T2522 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
  [junit4]  2> 444137 T2522 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:57742_solr
  [junit4]  2> 444138 T2522 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
  [junit4]  2> 444141 T2522 oasc.Overseer.start Overseer (id=92053395922223107-127.0.0.1:57742_solr-n_0000000000) starting
  [junit4]  2> 444144 T2522 oascc.SolrZkClient.makePath makePath: /overseer/queue
  [junit4]  2> 444149 T2522 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
  [junit4]  2> 444153 T2522 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
  [junit4]  2> 444157 T2522 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
  [junit4]  2> 444162 T2522 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
  [junit4]  2> 444166 T2522 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
  [junit4]  2> 444173 T2553 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
  [junit4]  2> 444175 T2522 oascc.SolrZkClient.makePath makePath: /clusterstate.json
  [junit4]  2> 444175 T2552 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
  [junit4]  2> 444177 T2522 oascc.SolrZkClient.makePath makePath: /aliases.json
  [junit4]  2> 444179 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
  [junit4]  2> 444183 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 444185 T2555 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1196b81 name:ZooKeeperConnection Watcher:127.0.0.1:57742/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 444185 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 444187 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
  [junit4]  2> 444190 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1
  [junit4]  2> 444194 T2549 oascc.ZkStateReader$3.process Updating live nodes... (1)
  [junit4]  2> 444194 T2555 oascc.ZkStateReader$3.process Updating live nodes... (1)
  [junit4]  2> 444197 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
  [junit4]  2> 444199 T2552 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
  [junit4]  2> 444199 T2552 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
  [junit4]  2>      "operation":"state",
  [junit4]  2>      "state":"active",
  [junit4]  2>      "node_name":"127.0.0.1",
  [junit4]  2>      "core":"core1",
  [junit4]  2>      "core_node_name":"node1",
  [junit4]  2>      "collection":"collection1",
  [junit4]  2>      "numShards":"3",
  [junit4]  2>      "base_url":"http://127.0.0.1/solr/"}
  [junit4]  2> 444200 T2552 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
  [junit4]  2> 444200 T2552 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
  [junit4]  2> 444204 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
  [junit4]  2> 444205 T2555 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
  [junit4]  2> 444205 T2549 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
  [junit4]  2> 444697 T2522 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
  [junit4]  2> 444712 T2522 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
  [junit4]  2> 444719 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
  [junit4]  2> 444721 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
  [junit4]  2> 444721 T2552 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
  [junit4]  2> 444727 T2552 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
  [junit4]  2>      "operation":"state",
  [junit4]  2>      "state":"active",
  [junit4]  2>      "node_name":"127.0.0.1",
  [junit4]  2>      "core":"core2",
  [junit4]  2>      "core_node_name":"node2",
  [junit4]  2>      "collection":"collection1",
  [junit4]  2>      "numShards":"3",
  [junit4]  2>      "base_url":"http://127.0.0.1/solr/"}
  [junit4]  2> 444728 T2552 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
  [junit4]  2> 444728 T2552 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
  [junit4]  2> 444731 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
  [junit4]  2> 444834 T2555 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
  [junit4]  2> 444834 T2549 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
  [junit4]  2> 445222 T2522 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
  [junit4]  2> 445233 T2522 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
  [junit4]  2> 445240 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
  [junit4]  2> 445240 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
  [junit4]  2> 445241 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
  [junit4]  2> 445242 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
  [junit4]  2> 445242 T2552 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
  [junit4]  2> 445248 T2552 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
  [junit4]  2>      "operation":"state",
  [junit4]  2>      "state":"active",
  [junit4]  2>      "node_name":"127.0.0.1",
  [junit4]  2>      "core":"core3",
  [junit4]  2>      "core_node_name":"node3",
  [junit4]  2>      "collection":"collection1",
  [junit4]  2>      "numShards":"3",
  [junit4]  2>      "base_url":"http://127.0.0.1/solr/"}
  [junit4]  2> 445248 T2552 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
  [junit4]  2> 445248 T2552 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
  [junit4]  2> 445251 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
  [junit4]  2> 445354 T2549 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
  [junit4]  2> 445354 T2555 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
  [junit4]  2> 445742 T2522 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
  [junit4]  2> 445753 T2522 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
  [junit4]  2> 445764 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
  [junit4]  2> 445765 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
  [junit4]  2> 445765 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
  [junit4]  2> 445766 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
  [junit4]  2> 445767 T2552 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
  [junit4]  2> 445771 T2552 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
  [junit4]  2>      "operation":"state",
  [junit4]  2>      "state":"active",
  [junit4]  2>      "node_name":"127.0.0.1",
  [junit4]  2>      "core":"core4",
  [junit4]  2>      "core_node_name":"node4",
  [junit4]  2>      "collection":"collection1",
  [junit4]  2>      "numShards":"3",
  [junit4]  2>      "base_url":"http://127.0.0.1/solr/"}
  [junit4]  2> 445771 T2552 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
  [junit4]  2> 445771 T2552 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
  [junit4]  2> 445773 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
  [junit4]  2> 445876 T2555 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
  [junit4]  2> 445876 T2549 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
  [junit4]  2> 446267 T2522 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
  [junit4]  2> 446276 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
  [junit4]  2> 446276 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
  [junit4]  2> 446276 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
  [junit4]  2> 446278 T2552 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
  [junit4]  2> 446279 T2552 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
  [junit4]  2>      "operation":"state",
  [junit4]  2>      "state":"active",
  [junit4]  2>      "node_name":"127.0.0.1",
  [junit4]  2>      "core":"core5",
  [junit4]  2>      "core_node_name":"node5",
  [junit4]  2>      "collection":"collection1",
  [junit4]  2>      "numShards":"3",
  [junit4]  2>      "base_url":"http://127.0.0.1/solr/"}
  [junit4]  2> 446280 T2552 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
  [junit4]  2> 446280 T2552 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
  [junit4]  2> 446283 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
  [junit4]  2> 446386 T2555 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
  [junit4]  2> 446386 T2549 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
  [junit4]  2> 446776 T2522 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
  [junit4]  2> 446787 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
  [junit4]  2> 446788 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
  [junit4]  2> 446788 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
  [junit4]  2> 446790 T2552 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
  [junit4]  2> 446791 T2552 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
  [junit4]  2>      "operation":"state",
  [junit4]  2>      "state":"active",
  [junit4]  2>      "node_name":"127.0.0.1",
  [junit4]  2>      "core":"core6",
  [junit4]  2>      "core_node_name":"node6",
  [junit4]  2>      "collection":"collection1",
  [junit4]  2>      "numShards":"3",
  [junit4]  2>      "base_url":"http://127.0.0.1/solr/"}
  [junit4]  2> 446791 T2552 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
  [junit4]  2> 446791 T2552 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
  [junit4]  2> 446795 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
  [junit4]  2> 446899 T2549 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
  [junit4]  2> 446899 T2555 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
  [junit4]  2> 447288 T2522 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
  [junit4]  2> 447298 T2555 oascc.ZkStateReader$3.process Updating live nodes... (0)
  [junit4]  2> 447301 T2522 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:57742 57742
  [junit4]  2> 447869 T2539 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:57742 57742
  [junit4]  2> 447871 T2522 oas.SolrTestCaseJ4.tearDown ###Ending testShardAssignment
  [junit4]  2> 447872 T2552 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:57742_solr
  [junit4]  2> 447875 T2556 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
  [junit4]  2>     at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
  [junit4]  2>     at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
  [junit4]  2>     at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
  [junit4]  2>     at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
  [junit4]  2>     at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
  [junit4]  2>     at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
  [junit4]  2>     at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
  [junit4]  2>     at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:316)
  [junit4]  2>     at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
  [junit4]  2>     at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:288)
  [junit4]  2>
  [junit4]  2> 447881 T2522 oas.SolrTestCaseJ4.setUp ###Starting testShardAssignmentBigger
  [junit4]  2> 447882 T2522 oasc.ZkTestServer.run STARTING ZK TEST SERVER
  [junit4]  1> client port:0.0.0.0/0.0.0.0:0
  [junit4]  2> 447883 T2557 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
  [junit4]  2> 447983 T2522 oasc.ZkTestServer.run start zk server on port:50146
  [junit4]  2> 447984 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 447988 T2563 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@daa195 name:ZooKeeperConnection Watcher:127.0.0.1:50146 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 447989 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 447993 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 447994 T2565 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f8aac name:ZooKeeperConnection Watcher:127.0.0.1:50146 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 447995 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 447995 T2522 oascc.SolrZkClient.makePath makePath: /solr
  [junit4]  2> 448000 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 448002 T2567 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@141291c name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 448002 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 448003 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes
  [junit4]  2> 448006 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 448008 T2569 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@49f99d name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 448008 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 448009 T2522 oascc.SolrZkClient.makePath makePath: /overseer_elect
  [junit4]  2> 448012 T2522 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
  [junit4]  2> 448018 T2522 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:50146_solr
  [junit4]  2> 448018 T2522 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
  [junit4]  2> 448021 T2522 oasc.Overseer.start Overseer (id=92053396176633859-127.0.0.1:50146_solr-n_0000000000) starting
  [junit4]  2> 448024 T2522 oascc.SolrZkClient.makePath makePath: /overseer/queue
  [junit4]  2> 448029 T2522 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
  [junit4]  2> 448034 T2522 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
  [junit4]  2> 448038 T2522 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
  [junit4]  2> 448043 T2522 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
  [junit4]  2> 448048 T2522 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
  [junit4]  2> 448055 T2571 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
  [junit4]  2> 448056 T2522 oascc.SolrZkClient.makePath makePath: /clusterstate.json
  [junit4]  2> 448056 T2570 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
  [junit4]  2> 448058 T2522 oascc.SolrZkClient.makePath makePath: /aliases.json
  [junit4]  2> 448061 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
  [junit4]  2> 448065 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 448066 T2573 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1242e08 name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 448067 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 448069 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
  [junit4]  2> 448072 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node0
  [junit4]  2> 448076 T2573 oascc.ZkStateReader$3.process Updating live nodes... (1)
  [junit4]  2> 448076 T2567 oascc.ZkStateReader$3.process Updating live nodes... (1)
  [junit4]  2> 448077 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 448078 T2575 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1fc812a name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 448078 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 448080 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
  [junit4]  2> 448083 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
  [junit4]  2> 448087 T2573 oascc.ZkStateReader$3.process Updating live nodes... (2)
  [junit4]  2> 448087 T2567 oascc.ZkStateReader$3.process Updating live nodes... (2)
  [junit4]  2> 448087 T2575 oascc.ZkStateReader$3.process Updating live nodes... (2)
  [junit4]  2> 448088 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 448089 T2577 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1714ec0 name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 448090 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 448091 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
  [junit4]  2> 448095 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node2
  [junit4]  2> 448099 T2573 oascc.ZkStateReader$3.process Updating live nodes... (3)
  [junit4]  2> 448099 T2577 oascc.ZkStateReader$3.process Updating live nodes... (3)
  [junit4]  2> 448099 T2575 oascc.ZkStateReader$3.process Updating live nodes... (3)
  [junit4]  2> 448100 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 448099 T2567 oascc.ZkStateReader$3.process Updating live nodes... (3)
  [junit4]  2> 448102 T2579 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@fbc5e2 name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 448102 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 448104 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
  [junit4]  2> 448107 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node3
  [junit4]  2> 448112 T2577 oascc.ZkStateReader$3.process Updating live nodes... (4)
  [junit4]  2> 448112 T2575 oascc.ZkStateReader$3.process Updating live nodes... (4)
  [junit4]  2> 448112 T2567 oascc.ZkStateReader$3.process Updating live nodes... (4)
  [junit4]  2> 448112 T2573 oascc.ZkStateReader$3.process Updating live nodes... (4)
  [junit4]  2> 448113 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 448113 T2579 oascc.ZkStateReader$3.process Updating live nodes... (4)
  [junit4]  2> 448115 T2581 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@75e1db name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 448115 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 448117 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
  [junit4]  2> 448121 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node4
  [junit4]  2> 448126 T2577 oascc.ZkStateReader$3.process Updating live nodes... (5)
  [junit4]  2> 448126 T2575 oascc.ZkStateReader$3.process Updating live nodes... (5)
  [junit4]  2> 448126 T2573 oascc.ZkStateReader$3.process Updating live nodes... (5)
  [junit4]  2> 448127 T2581 oascc.ZkStateReader$3.process Updating live nodes... (5)
  [junit4]  2> 448126 T2567 oascc.ZkStateReader$3.process Updating live nodes... (5)
  [junit4]  2> 448126 T2579 oascc.ZkStateReader$3.process Updating live nodes... (5)
  [junit4]  2> 448127 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 448129 T2583 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@11dc02e name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 448130 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 448132 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
  [junit4]  2> 448136 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node5
  [junit4]  2> 448139 T2575 oascc.ZkStateReader$3.process Updating live nodes... (6)
  [junit4]  2> 448140 T2577 oascc.ZkStateReader$3.process Updating live nodes... (6)
  [junit4]  2> 448140 T2581 oascc.ZkStateReader$3.process Updating live nodes... (6)
  [junit4]  2> 448140 T2573 oascc.ZkStateReader$3.process Updating live nodes... (6)
  [junit4]  2> 448140 T2579 oascc.ZkStateReader$3.process Updating live nodes... (6)
  [junit4]  2> 448141 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 448141 T2583 oascc.ZkStateReader$3.process Updating live nodes... (6)
  [junit4]  2> 448140 T2567 oascc.ZkStateReader$3.process Updating live nodes... (6)
  [junit4]  2> 448143 T2585 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6d9f03 name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 448143 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 448145 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
  [junit4]  2> 448148 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node6
  [junit4]  2> 448153 T2575 oascc.ZkStateReader$3.process Updating live nodes... (7)
  [junit4]  2> 448153 T2577 oascc.ZkStateReader$3.process Updating live nodes... (7)
  [junit4]  2> 448153 T2573 oascc.ZkStateReader$3.process Updating live nodes... (7)
  [junit4]  2> 448153 T2585 oascc.ZkStateReader$3.process Updating live nodes... (7)
  [junit4]  2> 448153 T2583 oascc.ZkStateReader$3.process Updating live nodes... (7)
  [junit4]  2> 448153 T2579 oascc.ZkStateReader$3.process Updating live nodes... (7)
  [junit4]  2> 448153 T2581 oascc.ZkStateReader$3.process Updating live nodes... (7)
  [junit4]  2> 448153 T2567 oascc.ZkStateReader$3.process Updating live nodes... (7)
  [junit4]  2> 448154 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 448156 T2587 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@958679 name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 448157 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 448159 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
  [junit4]  2> 448162 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node7
  [junit4]  2> 448167 T2567 oascc.ZkStateReader$3.process Updating live nodes... (8)
  [junit4]  2> 448167 T2577 oascc.ZkStateReader$3.process Updating live nodes... (8)
  [junit4]  2> 448168 T2585 oascc.ZkStateReader$3.process Updating live nodes... (8)
  [junit4]  2> 448168 T2583 oascc.ZkStateReader$3.process Updating live nodes... (8)
  [junit4]  2> 448168 T2581 oascc.ZkStateReader$3.process Updating live nodes... (8)
  [junit4]  2> 448168 T2579 oascc.ZkStateReader$3.process Updating live nodes... (8)
  [junit4]  2> 448168 T2573 oascc.ZkStateReader$3.process Updating live nodes... (8)
  [junit4]  2> 448167 T2587 oascc.ZkStateReader$3.process Updating live nodes... (8)
  [junit4]  2> 448167 T2575 oascc.ZkStateReader$3.process Updating live nodes... (8)
  [junit4]  2> 448169 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 448171 T2589 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@169d949 name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 448171 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 448174 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
  [junit4]  2> 448177 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node8
  [junit4]  2> 448182 T2589 oascc.ZkStateReader$3.process Updating live nodes... (9)
  [junit4]  2> 448183 T2585 oascc.ZkStateReader$3.process Updating live nodes... (9)
  [junit4]  2> 448183 T2583 oascc.ZkStateReader$3.process Updating live nodes... (9)
  [junit4]  2> 448183 T2587 oascc.ZkStateReader$3.process Updating live nodes... (9)
  [junit4]  2> 448183 T2573 oascc.ZkStateReader$3.process Updating live nodes... (9)
  [junit4]  2> 448183 T2579 oascc.ZkStateReader$3.process Updating live nodes... (9)
  [junit4]  2> 448183 T2581 oascc.ZkStateReader$3.process Updating live nodes... (9)
  [junit4]  2> 448183 T2575 oascc.ZkStateReader$3.process Updating live nodes... (9)
  [junit4]  2> 448183 T2577 oascc.ZkStateReader$3.process Updating live nodes... (9)
  [junit4]  2> 448182 T2567 oascc.ZkStateReader$3.process Updating live nodes... (9)
  [junit4]  2> 448185 T2591 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2e78fa name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 448184 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 448188 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 448190 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
  [junit4]  2> 448192 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node9
  [junit4]  2> 448195 T2589 oascc.ZkStateReader$3.process Updating live nodes... (10)
  [junit4]  2> 448195 T2577 oascc.ZkStateReader$3.process Updating live nodes... (10)
  [junit4]  2> 448195 T2567 oascc.ZkStateReader$3.process Updating live nodes... (10)
  [junit4]  2> 448196 T2573 oascc.ZkStateReader$3.process Updating live nodes... (10)
  [junit4]  2> 448195 T2575 oascc.ZkStateReader$3.process Updating live nodes... (10)
  [junit4]  2> 448195 T2587 oascc.ZkStateReader$3.process Updating live nodes... (10)
  [junit4]  2> 448195 T2579 oascc.ZkStateReader$3.process Updating live nodes... (10)
  [junit4]  2> 448195 T2583 oascc.ZkStateReader$3.process Updating live nodes... (10)
  [junit4]  2> 448196 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 448198 T2593 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@156c3b name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 448196 T2591 oascc.ZkStateReader$3.process Updating live nodes... (10)
  [junit4]  2> 448196 T2585 oascc.ZkStateReader$3.process Updating live nodes... (10)
  [junit4]  2> 448196 T2581 oascc.ZkStateReader$3.process Updating live nodes... (10)
  [junit4]  2> 448198 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 448200 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
  [junit4]  2> 448203 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node10
  [junit4]  2> 448207 T2575 oascc.ZkStateReader$3.process Updating live nodes... (11)
  [junit4]  2> 448207 T2583 oascc.ZkStateReader$3.process Updating live nodes... (11)
  [junit4]  2> 448208 T2579 oascc.ZkStateReader$3.process Updating live nodes... (11)
  [junit4]  2> 448207 T2577 oascc.ZkStateReader$3.process Updating live nodes... (11)
  [junit4]  2> 448207 T2573 oascc.ZkStateReader$3.process Updating live nodes... (11)
  [junit4]  2> 448207 T2589 oascc.ZkStateReader$3.process Updating live nodes... (11)
  [junit4]  2> 448207 T2587 oascc.ZkStateReader$3.process Updating live nodes... (11)
  [junit4]  2> 448209 T2585 oascc.ZkStateReader$3.process Updating live nodes... (11)
  [junit4]  2> 448208 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 448208 T2581 oascc.ZkStateReader$3.process Updating live nodes... (11)
  [junit4]  2> 448208 T2593 oascc.ZkStateReader$3.process Updating live nodes... (11)
  [junit4]  2> 448208 T2591 oascc.ZkStateReader$3.process Updating live nodes... (11)
  [junit4]  2> 448208 T2567 oascc.ZkStateReader$3.process Updating live nodes... (11)
  [junit4]  2> 448211 T2595 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f1978e name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 448212 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 448213 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
  [junit4]  2> 448216 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node11
  [junit4]  2> 448220 T2587 oascc.ZkStateReader$3.process Updating live nodes... (12)
  [junit4]  2> 448220 T2567 oascc.ZkStateReader$3.process Updating live nodes... (12)
  [junit4]  2> 448221 T2583 oascc.ZkStateReader$3.process Updating live nodes... (12)
  [junit4]  2> 448220 T2589 oascc.ZkStateReader$3.process Updating live nodes... (12)
  [junit4]  2> 448222 T2595 oascc.ZkStateReader$3.process Updating live nodes... (12)
  [junit4]  2> 448220 T2575 oascc.ZkStateReader$3.process Updating live nodes... (12)
  [junit4]  2> 448222 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 448222 T2593 oascc.ZkStateReader$3.process Updating live nodes... (12)
  [junit4]  2> 448222 T2581 oascc.ZkStateReader$3.process Updating live nodes... (12)
  [junit4]  2> 448224 T2597 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@175be88 name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 448221 T2591 oascc.ZkStateReader$3.process Updating live nodes... (12)
  [junit4]  2> 448221 T2573 oascc.ZkStateReader$3.process Updating live nodes... (12)
  [junit4]  2> 448221 T2577 oascc.ZkStateReader$3.process Updating live nodes... (12)
  [junit4]  2> 448221 T2585 oascc.ZkStateReader$3.process Updating live nodes... (12)
  [junit4]  2> 448221 T2579 oascc.ZkStateReader$3.process Updating live nodes... (12)
  [junit4]  2> 448224 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 448228 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
  [junit4]  2> 448232 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node12
  [junit4]  2> 448236 T2573 oascc.ZkStateReader$3.process Updating live nodes... (13)
  [junit4]  2> 448236 T2577 oascc.ZkStateReader$3.process Updating live nodes... (13)
  [junit4]  2> 448237 T2595 oascc.ZkStateReader$3.process Updating live nodes... (13)
  [junit4]  2> 448237 T2593 oascc.ZkStateReader$3.process Updating live nodes... (13)
  [junit4]  2> 448237 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 448238 T2581 oascc.ZkStateReader$3.process Updating live nodes... (13)
  [junit4]  2> 448238 T2597 oascc.ZkStateReader$3.process Updating live nodes... (13)
  [junit4]  2> 448237 T2583 oascc.ZkStateReader$3.process Updating live nodes... (13)
  [junit4]  2> 448239 T2599 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@dde0f4 name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 448237 T2589 oascc.ZkStateReader$3.process Updating live nodes... (13)
  [junit4]  2> 448237 T2575 oascc.ZkStateReader$3.process Updating live nodes... (13)
  [junit4]  2> 448237 T2587 oascc.ZkStateReader$3.process Updating live nodes... (13)
  [junit4]  2> 448237 T2585 oascc.ZkStateReader$3.process Updating live nodes... (13)
  [junit4]  2> 448236 T2567 oascc.ZkStateReader$3.process Updating live nodes... (13)
  [junit4]  2> 448236 T2579 oascc.ZkStateReader$3.process Updating live nodes... (13)
  [junit4]  2> 448236 T2591 oascc.ZkStateReader$3.process Updating live nodes... (13)
  [junit4]  2> 448240 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 448244 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
  [junit4]  2> 448248 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node13
  [junit4]  2> 448254 T2587 oascc.ZkStateReader$3.process Updating live nodes... (14)
  [junit4]  2> 448255 T2577 oascc.ZkStateReader$3.process Updating live nodes... (14)
  [junit4]  2> 448255 T2579 oascc.ZkStateReader$3.process Updating live nodes... (14)
  [junit4]  2> 448255 T2595 oascc.ZkStateReader$3.process Updating live nodes... (14)
  [junit4]  2> 448255 T2573 oascc.ZkStateReader$3.process Updating live nodes... (14)
  [junit4]  2> 448256 T2597 oascc.ZkStateReader$3.process Updating live nodes... (14)
  [junit4]  2> 448256 T2599 oascc.ZkStateReader$3.process Updating live nodes... (14)
  [junit4]  2> 448256 T2593 oascc.ZkStateReader$3.process Updating live nodes... (14)
  [junit4]  2> 448256 T2585 oascc.ZkStateReader$3.process Updating live nodes... (14)
  [junit4]  2> 448256 T2581 oascc.ZkStateReader$3.process Updating live nodes... (14)
  [junit4]  2> 448256 T2583 oascc.ZkStateReader$3.process Updating live nodes... (14)
  [junit4]  2> 448256 T2567 oascc.ZkStateReader$3.process Updating live nodes... (14)
  [junit4]  2> 448256 T2575 oascc.ZkStateReader$3.process Updating live nodes... (14)
  [junit4]  2> 448256 T2589 oascc.ZkStateReader$3.process Updating live nodes... (14)
  [junit4]  2> 448256 T2591 oascc.ZkStateReader$3.process Updating live nodes... (14)
  [junit4]  2> 448259 T2601 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@19e3760 name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 448257 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 448262 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 448264 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
  [junit4]  2> 448268 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node14
  [junit4]  2> 448273 T2573 oascc.ZkStateReader$3.process Updating live nodes... (15)
  [junit4]  2> 448273 T2587 oascc.ZkStateReader$3.process Updating live nodes... (15)
  [junit4]  2> 448273 T2579 oascc.ZkStateReader$3.process Updating live nodes... (15)
  [junit4]  2> 448273 T2591 oascc.ZkStateReader$3.process Updating live nodes... (15)
  [junit4]  2> 448273 T2577 oascc.ZkStateReader$3.process Updating live nodes... (15)
  [junit4]  2> 448273 T2595 oascc.ZkStateReader$3.process Updating live nodes... (15)
  [junit4]  2> 448275 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 448275 T2599 oascc.ZkStateReader$3.process Updating live nodes... (15)
  [junit4]  2> 448275 T2583 oascc.ZkStateReader$3.process Updating live nodes... (15)
  [junit4]  2> 448275 T2585 oascc.ZkStateReader$3.process Updating live nodes... (15)
  [junit4]  2> 448275 T2597 oascc.ZkStateReader$3.process Updating live nodes... (15)
  [junit4]  2> 448275 T2601 oascc.ZkStateReader$3.process Updating live nodes... (15)
  [junit4]  2> 448275 T2567 oascc.ZkStateReader$3.process Updating live nodes... (15)
  [junit4]  2> 448275 T2575 oascc.ZkStateReader$3.process Updating live nodes... (15)
  [junit4]  2> 448275 T2581 oascc.ZkStateReader$3.process Updating live nodes... (15)
  [junit4]  2> 448274 T2593 oascc.ZkStateReader$3.process Updating live nodes... (15)
  [junit4]  2> 448273 T2589 oascc.ZkStateReader$3.process Updating live nodes... (15)
  [junit4]  2> 448277 T2603 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1818cfa name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 448281 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 448283 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
  [junit4]  2> 448286 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node15
  [junit4]  2> 448291 T2577 oascc.ZkStateReader$3.process Updating live nodes... (16)
  [junit4]  2> 448292 T2573 oascc.ZkStateReader$3.process Updating live nodes... (16)
  [junit4]  2> 448291 T2587 oascc.ZkStateReader$3.process Updating live nodes... (16)
  [junit4]  2> 448293 T2579 oascc.ZkStateReader$3.process Updating live nodes... (16)
  [junit4]  2> 448293 T2575 oascc.ZkStateReader$3.process Updating live nodes... (16)
  [junit4]  2> 448293 T2601 oascc.ZkStateReader$3.process Updating live nodes... (16)
  [junit4]  2> 448294 T2599 oascc.ZkStateReader$3.process Updating live nodes... (16)
  [junit4]  2> 448293 T2591 oascc.ZkStateReader$3.process Updating live nodes... (16)
  [junit4]  2> 448293 T2597 oascc.ZkStateReader$3.process Updating live nodes... (16)
  [junit4]  2> 448293 T2583 oascc.ZkStateReader$3.process Updating live nodes... (16)
  [junit4]  2> 448293 T2567 oascc.ZkStateReader$3.process Updating live nodes... (16)
  [junit4]  2> 448293 T2589 oascc.ZkStateReader$3.process Updating live nodes... (16)
  [junit4]  2> 448295 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 448294 T2603 oascc.ZkStateReader$3.process Updating live nodes... (16)
  [junit4]  2> 448294 T2581 oascc.ZkStateReader$3.process Updating live nodes... (16)
  [junit4]  2> 448294 T2585 oascc.ZkStateReader$3.process Updating live nodes... (16)
  [junit4]  2> 448294 T2595 oascc.ZkStateReader$3.process Updating live nodes... (16)
  [junit4]  2> 448294 T2593 oascc.ZkStateReader$3.process Updating live nodes... (16)
  [junit4]  2> 448297 T2605 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@154e83c name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 448299 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 448300 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
  [junit4]  2> 448304 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node16
  [junit4]  2> 448313 T2587 oascc.ZkStateReader$3.process Updating live nodes... (17)
  [junit4]  2> 448313 T2589 oascc.ZkStateReader$3.process Updating live nodes... (17)
  [junit4]  2> 448314 T2595 oascc.ZkStateReader$3.process Updating live nodes... (17)
  [junit4]  2> 448315 T2575 oascc.ZkStateReader$3.process Updating live nodes... (17)
  [junit4]  2> 448314 T2585 oascc.ZkStateReader$3.process Updating live nodes... (17)
  [junit4]  2> 448316 T2591 oascc.ZkStateReader$3.process Updating live nodes... (17)
  [junit4]  2> 448313 T2573 oascc.ZkStateReader$3.process Updating live nodes... (17)
  [junit4]  2> 448313 T2577 oascc.ZkStateReader$3.process Updating live nodes... (17)
  [junit4]  2> 448313 T2593 oascc.ZkStateReader$3.process Updating live nodes... (17)
  [junit4]  2> 448317 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 448316 T2599 oascc.ZkStateReader$3.process Updating live nodes... (17)
  [junit4]  2> 448316 T2581 oascc.ZkStateReader$3.process Updating live nodes... (17)
  [junit4]  2> 448316 T2605 oascc.ZkStateReader$3.process Updating live nodes... (17)
  [junit4]  2> 448316 T2603 oascc.ZkStateReader$3.process Updating live nodes... (17)
  [junit4]  2> 448316 T2583 oascc.ZkStateReader$3.process Updating live nodes... (17)
  [junit4]  2> 448316 T2567 oascc.ZkStateReader$3.process Updating live nodes... (17)
  [junit4]  2> 448315 T2601 oascc.ZkStateReader$3.process Updating live nodes... (17)
  [junit4]  2> 448315 T2597 oascc.ZkStateReader$3.process Updating live nodes... (17)
  [junit4]  2> 448315 T2579 oascc.ZkStateReader$3.process Updating live nodes... (17)
  [junit4]  2> 448319 T2607 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d963a2 name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 448323 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 448324 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
  [junit4]  2> 448328 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node17
  [junit4]  2> 448333 T2573 oascc.ZkStateReader$3.process Updating live nodes... (18)
  [junit4]  2> 448333 T2577 oascc.ZkStateReader$3.process Updating live nodes... (18)
  [junit4]  2> 448334 T2601 oascc.ZkStateReader$3.process Updating live nodes... (18)
  [junit4]  2> 448334 T2607 oascc.ZkStateReader$3.process Updating live nodes... (18)
  [junit4]  2> 448335 T2567 oascc.ZkStateReader$3.process Updating live nodes... (18)
  [junit4]  2> 448335 T2583 oascc.ZkStateReader$3.process Updating live nodes... (18)
  [junit4]  2> 448335 T2603 oascc.ZkStateReader$3.process Updating live nodes... (18)
  [junit4]  2> 448334 T2589 oascc.ZkStateReader$3.process Updating live nodes... (18)
  [junit4]  2> 448336 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 448334 T2595 oascc.ZkStateReader$3.process Updating live nodes... (18)
  [junit4]  2> 448334 T2579 oascc.ZkStateReader$3.process Updating live nodes... (18)
  [junit4]  2> 448334 T2575 oascc.ZkStateReader$3.process Updating live nodes... (18)
  [junit4]  2> 448338 T2609 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@826a2e name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 448333 T2587 oascc.ZkStateReader$3.process Updating live nodes... (18)
  [junit4]  2> 448333 T2591 oascc.ZkStateReader$3.process Updating live nodes... (18)
  [junit4]  2> 448338 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 448336 T2585 oascc.ZkStateReader$3.process Updating live nodes... (18)
  [junit4]  2> 448335 T2593 oascc.ZkStateReader$3.process Updating live nodes... (18)
  [junit4]  2> 448335 T2599 oascc.ZkStateReader$3.process Updating live nodes... (18)
  [junit4]  2> 448335 T2581 oascc.ZkStateReader$3.process Updating live nodes... (18)
  [junit4]  2> 448335 T2605 oascc.ZkStateReader$3.process Updating live nodes... (18)
  [junit4]  2> 448335 T2597 oascc.ZkStateReader$3.process Updating live nodes... (18)
  [junit4]  2> 448341 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
  [junit4]  2> 448346 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node18
  [junit4]  2> 448350 T2573 oascc.ZkStateReader$3.process Updating live nodes... (19)
  [junit4]  2> 448350 T2577 oascc.ZkStateReader$3.process Updating live nodes... (19)
  [junit4]  2> 448351 T2607 oascc.ZkStateReader$3.process Updating live nodes... (19)
  [junit4]  2> 448352 T2587 oascc.ZkStateReader$3.process Updating live nodes... (19)
  [junit4]  2> 448352 T2593 oascc.ZkStateReader$3.process Updating live nodes... (19)
  [junit4]  2> 448351 T2601 oascc.ZkStateReader$3.process Updating live nodes... (19)
  [junit4]  2> 448353 T2579 oascc.ZkStateReader$3.process Updating live nodes... (19)
  [junit4]  2> 448353 T2581 oascc.ZkStateReader$3.process Updating live nodes... (19)
  [junit4]  2> 448351 T2595 oascc.ZkStateReader$3.process Updating live nodes... (19)
  [junit4]  2> 448353 T2605 oascc.ZkStateReader$3.process Updating live nodes... (19)
  [junit4]  2> 448354 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 448353 T2609 oascc.ZkStateReader$3.process Updating live nodes... (19)
  [junit4]  2> 448353 T2603 oascc.ZkStateReader$3.process Updating live nodes... (19)
  [junit4]  2> 448353 T2597 oascc.ZkStateReader$3.process Updating live nodes... (19)
  [junit4]  2> 448353 T2589 oascc.ZkStateReader$3.process Updating live nodes... (19)
  [junit4]  2> 448353 T2567 oascc.ZkStateReader$3.process Updating live nodes... (19)
  [junit4]  2> 448353 T2591 oascc.ZkStateReader$3.process Updating live nodes... (19)
  [junit4]  2> 448353 T2575 oascc.ZkStateReader$3.process Updating live nodes... (19)
  [junit4]  2> 448352 T2599 oascc.ZkStateReader$3.process Updating live nodes... (19)
  [junit4]  2> 448352 T2583 oascc.ZkStateReader$3.process Updating live nodes... (19)
  [junit4]  2> 448352 T2585 oascc.ZkStateReader$3.process Updating live nodes... (19)
  [junit4]  2> 448356 T2611 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b5c498 name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 448359 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 448362 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
  [junit4]  2> 448366 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node19
  [junit4]  2> 448371 T2587 oascc.ZkStateReader$3.process Updating live nodes... (20)
  [junit4]  2> 448371 T2591 oascc.ZkStateReader$3.process Updating live nodes... (20)
  [junit4]  2> 448372 T2579 oascc.ZkStateReader$3.process Updating live nodes... (20)
  [junit4]  2> 448372 T2575 oascc.ZkStateReader$3.process Updating live nodes... (20)
  [junit4]  2> 448372 T2577 oascc.ZkStateReader$3.process Updating live nodes... (20)
  [junit4]  2> 448373 T2581 oascc.ZkStateReader$3.process Updating live nodes... (20)
  [junit4]  2> 448372 T2589 oascc.ZkStateReader$3.process Updating live nodes... (20)
  [junit4]  2> 448372 T2573 oascc.ZkStateReader$3.process Updating live nodes... (20)
  [junit4]  2> 448372 T2595 oascc.ZkStateReader$3.process Updating live nodes... (20)
  [junit4]  2> 448374 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 448373 T2609 oascc.ZkStateReader$3.process Updating live nodes... (20)
  [junit4]  2> 448373 T2611 oascc.ZkStateReader$3.process Updating live nodes... (20)
  [junit4]  2> 448373 T2603 oascc.ZkStateReader$3.process Updating live nodes... (20)
  [junit4]  2> 448373 T2585 oascc.ZkStateReader$3.process Updating live nodes... (20)
  [junit4]  2> 448373 T2593 oascc.ZkStateReader$3.process Updating live nodes... (20)
  [junit4]  2> 448373 T2567 oascc.ZkStateReader$3.process Updating live nodes... (20)
  [junit4]  2> 448373 T2607 oascc.ZkStateReader$3.process Updating live nodes... (20)
  [junit4]  2> 448372 T2599 oascc.ZkStateReader$3.process Updating live nodes... (20)
  [junit4]  2> 448372 T2597 oascc.ZkStateReader$3.process Updating live nodes... (20)
  [junit4]  2> 448372 T2601 oascc.ZkStateReader$3.process Updating live nodes... (20)
  [junit4]  2> 448372 T2583 oascc.ZkStateReader$3.process Updating live nodes... (20)
  [junit4]  2> 448372 T2605 oascc.ZkStateReader$3.process Updating live nodes... (20)
  [junit4]  2> 448376 T2613 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@18ed79a name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 448380 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 448382 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
  [junit4]  2> 448385 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node20
  [junit4]  2> 448390 T2577 oascc.ZkStateReader$3.process Updating live nodes... (21)
  [junit4]  2> 448390 T2579 oascc.ZkStateReader$3.process Updating live nodes... (21)
  [junit4]  2> 448391 T2589 oascc.ZkStateReader$3.process Updating live nodes... (21)
  [junit4]  2> 448391 T2601 oascc.ZkStateReader$3.process Updating live nodes... (21)
  [junit4]  2> 448391 T2599 oascc.ZkStateReader$3.process Updating live nodes... (21)
  [junit4]  2> 448391 T2573 oascc.ZkStateReader$3.process Updating live nodes... (21)
  [junit4]  2> 448392 T2581 oascc.ZkStateReader$3.process Updating live nodes... (21)
  [junit4]  2> 448391 T2605 oascc.ZkStateReader$3.process Updating live nodes... (21)
  [junit4]  2> 448390 T2591 oascc.ZkStateReader$3.process Updating live nodes... (21)
  [junit4]  2> 448390 T2595 oascc.ZkStateReader$3.process Updating live nodes... (21)
  [junit4]  2> 448393 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 448392 T2603 oascc.ZkStateReader$3.process Updating live nodes... (21)
  [junit4]  2> 448392 T2609 oascc.ZkStateReader$3.process Updating live nodes... (21)
  [junit4]  2> 448392 T2611 oascc.ZkStateReader$3.process Updating live nodes... (21)
  [junit4]  2> 448392 T2593 oascc.ZkStateReader$3.process Updating live nodes... (21)
  [junit4]  2> 448392 T2585 oascc.ZkStateReader$3.process Updating live nodes... (21)
  [junit4]  2> 448392 T2567 oascc.ZkStateReader$3.process Updating live nodes... (21)
  [junit4]  2> 448392 T2575 oascc.ZkStateReader$3.process Updating live nodes... (21)
  [junit4]  2> 448391 T2613 oascc.ZkStateReader$3.process Updating live nodes... (21)
  [junit4]  2> 448391 T2583 oascc.ZkStateReader$3.process Updating live nodes... (21)
  [junit4]  2> 448391 T2607 oascc.ZkStateReader$3.process Updating live nodes... (21)
  [junit4]  2> 448391 T2597 oascc.ZkStateReader$3.process Updating live nodes... (21)
  [junit4]  2> 448391 T2587 oascc.ZkStateReader$3.process Updating live nodes... (21)
  [junit4]  2> 448395 T2615 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5f6bba name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 448399 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 448401 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
  [junit4]  2> 448404 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node21
  [junit4]  2> 448409 T2587 oascc.ZkStateReader$3.process Updating live nodes... (22)
  [junit4]  2> 448410 T2573 oascc.ZkStateReader$3.process Updating live nodes... (22)
  [junit4]  2> 448410 T2595 oascc.ZkStateReader$3.process Updating live nodes... (22)
  [junit4]  2> 448410 T2577 oascc.ZkStateReader$3.process Updating live nodes... (22)
  [junit4]  2> 448409 T2591 oascc.ZkStateReader$3.process Updating live nodes... (22)
  [junit4]  2> 448412 T2607 oascc.ZkStateReader$3.process Updating live nodes... (22)
  [junit4]  2> 448412 T2593 oascc.ZkStateReader$3.process Updating live nodes... (22)
  [junit4]  2> 448412 T2599 oascc.ZkStateReader$3.process Updating live nodes... (22)
  [junit4]  2> 448413 T2603 oascc.ZkStateReader$3.process Updating live nodes... (22)
  [junit4]  2> 448413 T2609 oascc.ZkStateReader$3.process Updating live nodes... (22)
  [junit4]  2> 448412 T2567 oascc.ZkStateReader$3.process Updating live nodes... (22)
  [junit4]  2> 448412 T2613 oascc.ZkStateReader$3.process Updating live nodes... (22)
  [junit4]  2> 448412 T2575 oascc.ZkStateReader$3.process Updating live nodes... (22)
  [junit4]  2> 448411 T2583 oascc.ZkStateReader$3.process Updating live nodes... (22)
  [junit4]  2> 448411 T2601 oascc.ZkStateReader$3.process Updating live nodes... (22)
  [junit4]  2> 448411 T2597 oascc.ZkStateReader$3.process Updating live nodes... (22)
  [junit4]  2> 448411 T2579 oascc.ZkStateReader$3.process Updating live nodes... (22)
  [junit4]  2> 448411 T2605 oascc.ZkStateReader$3.process Updating live nodes... (22)
  [junit4]  2> 448411 T2589 oascc.ZkStateReader$3.process Updating live nodes... (22)
  [junit4]  2> 448416 T2617 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@13fabc5 name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 448414 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 448413 T2615 oascc.ZkStateReader$3.process Updating live nodes... (22)
  [junit4]  2> 448413 T2611 oascc.ZkStateReader$3.process Updating live nodes... (22)
  [junit4]  2> 448412 T2581 oascc.ZkStateReader$3.process Updating live nodes... (22)
  [junit4]  2> 448412 T2585 oascc.ZkStateReader$3.process Updating live nodes... (22)
  [junit4]  2> 448418 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 448421 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
  [junit4]  2> 448426 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node22
  [junit4]  2> 448431 T2587 oascc.ZkStateReader$3.process Updating live nodes... (23)
  [junit4]  2> 448432 T2577 oascc.ZkStateReader$3.process Updating live nodes... (23)
  [junit4]  2> 448432 T2579 oascc.ZkStateReader$3.process Updating live nodes... (23)
  [junit4]  2> 448432 T2605 oascc.ZkStateReader$3.process Updating live nodes... (23)
  [junit4]  2> 448433 T2611 oascc.ZkStateReader$3.process Updating live nodes... (23)
  [junit4]  2> 448432 T2595 oascc.ZkStateReader$3.process Updating live nodes... (23)
  [junit4]  2> 448432 T2591 oascc.ZkStateReader$3.process Updating live nodes... (23)
  [junit4]  2> 448434 T2609 oascc.ZkStateReader$3.process Updating live nodes... (23)
  [junit4]  2> 448432 T2573 oascc.ZkStateReader$3.process Updating live nodes... (23)
  [junit4]  2> 448435 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 448434 T2603 oascc.ZkStateReader$3.process Updating live nodes... (23)
  [junit4]  2> 448434 T2617 oascc.ZkStateReader$3.process Updating live nodes... (23)
  [junit4]  2> 448434 T2581 oascc.ZkStateReader$3.process Updating live nodes... (23)
  [junit4]  2> 448437 T2619 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@71217e name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null
[...truncated too long message...]
queue/qn-
  [junit4]  2>     at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
  [junit4]  2>     at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
  [junit4]  2>     at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
  [junit4]  2>     at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:243)
  [junit4]  2>     at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:240)
  [junit4]  2>     at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
  [junit4]  2>     at org.apache.solr.common.cloud.SolrZkClient.create(SolrZkClient.java:240)
  [junit4]  2>     at org.apache.solr.cloud.DistributedQueue.createData(DistributedQueue.java:346)
  [junit4]  2>     at org.apache.solr.cloud.DistributedQueue.offer(DistributedQueue.java:330)
  [junit4]  2>     at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:198)
  [junit4]  2>     at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:163)
  [junit4]  2>     at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:125)
  [junit4]  2>     at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:314)
  [junit4]  2>     at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:221)
  [junit4]  2>     at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:452)
  [junit4]  2>     at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:216)
  [junit4]  2>     at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:163)
  [junit4]  2>     at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:125)
  [junit4]  2>     at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
  [junit4]  2>     at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:358)
  [junit4]  2>     at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
  [junit4]  2>     at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
  [junit4]  2>
  [junit4]  2> 750036 T3044 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
  [junit4]  2> 750037 T3044 oasc.LeaderElector$ElectionWatcher.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
  [junit4]  2>     at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
  [junit4]  2>     at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
  [junit4]  2>     at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
  [junit4]  2>     at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
  [junit4]  2>     at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
  [junit4]  2>     at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
  [junit4]  2>     at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
  [junit4]  2>     at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
  [junit4]  2>     at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
  [junit4]  2>     at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:358)
  [junit4]  2>     at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
  [junit4]  2>     at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
  [junit4]  2>
  [junit4]  2> 750038 T3044 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
  [junit4]  2> 750038 T3044 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
  [junit4]  2> 750121 T3116 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
  [junit4]  2> 750206 T2787 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
  [junit4]  2> 750206 T2787 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
  [junit4]  2> 750207 T2787 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
  [junit4]  2> 750207 T2787 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
  [junit4]  2> 750207 T2787 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
  [junit4]  2> 750206 T3156 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
  [junit4]  2>     at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
  [junit4]  2>     at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
  [junit4]  2>     at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
  [junit4]  2>     at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
  [junit4]  2>     at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
  [junit4]  2>     at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
  [junit4]  2>     at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
  [junit4]  2>     at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:316)
  [junit4]  2>     at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
  [junit4]  2>     at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:288)
  [junit4]  2>
  [junit4]  2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-A819205877C572F6-001
  [junit4]  2> NOTE: test params are: codec=Lucene41, sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=ar_JO, timezone=Africa/Ndjamena
  [junit4]  2> NOTE: Linux 3.8.0-42-generic i386/Oracle Corporation 1.8.0_05 (32-bit)/cpus=8,threads=1,free=118175464,total=317980672
  [junit4]  2> NOTE: All tests run in this JVM: [TestRecovery, OpenCloseCoreStressTest, HardAutoCommitTest, DeleteReplicaTest, IndexSchemaRuntimeFieldTest, TestXIncludeConfig, TestFreeTextSuggestions, CollectionsAPIDistributedZkTest, TestFieldTypeResource, BadCopyFieldTest, CopyFieldTest, TestCSVLoader, UniqFieldsUpdateProcessorFactoryTest, TestIndexingPerformance, TestDocumentBuilder, AutoCommitTest, AtomicUpdatesTest, TestQueryTypes, TestCollapseQParserPlugin, BasicDistributedZk2Test, TestWriterPerf, TestCoreContainer, TestComplexPhraseQParserPlugin, TestDistribDocBasedVersion, TestFastLRUCache, TestDynamicFieldResource, DOMUtilTest, QueryResultKeyTest, DistribDocExpirationUpdateProcessorTest, UpdateRequestProcessorFactoryTest, TimeZoneUtilsTest, SchemaVersionSpecificBehaviorTest, PreAnalyzedUpdateProcessorTest, SOLR749Test, TestNonDefinedSimilarityFactory, TestFileDictionaryLookup, MultiThreadedOCPTest, TestRTGBase, QueryParsingTest, OutputWriterTest, LoggingHandlerTest, TestInfoStreamLogging, TestPseudoReturnFields, TestFastWriter, TestReplicationHandlerBackup, AssignTest, TermVectorComponentTest, SolrRequestParserTest, TestPerFieldSimilarity, TestAtomicUpdateErrorCases, TestManagedResource, TermVectorComponentDistributedTest, TestReRankQParserPlugin, TestSerializedLuceneMatchVersion, TestChildDocTransformer, TestNoOpRegenerator, TestGroupingSearch, TestFieldTypeCollectionResource, TestBinaryField, StatsComponentTest, CursorMarkTest, BasicZkTest, SuggesterFSTTest, BJQParserTest, CurrencyFieldXmlFileTest, DistributedQueryElevationComponentTest, TestDistributedMissingSort, SearchHandlerTest, TestLFUCache, TestSolrJ, FileBasedSpellCheckerTest, TestRealTimeGet, RequestHandlersTest, AnalysisAfterCoreReloadTest, SolrInfoMBeanTest, TestDistributedGrouping, TestExtendedDismaxParser, URLClassifyProcessorTest, SuggestComponentTest, ChaosMonkeySafeLeaderTest]
  [junit4] Completed on J1 in 74.71s, 1 test, 1 failure <<< FAILURES!
[...truncated 59 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1299: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:923: There were test failures: 405 suites, 1656 tests, 3 suite-level errors, 1 error, 1 failure, 954 ignored (13 assumptions)
Total time: 65 minutes 33 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 32bit/jdk1.8.0_05 -client -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
---------------------------------------------------------------------
To unsubscribe, e-mail:Â dev-unsubscribe@lucene.apache.org
For additional commands, e-mail:Â dev-help@lucene.apache.org
--
Regards,
Shalin Shekhar Mangar.
Re: [JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.8.0_05) - Build #
10616 - Still Failing!
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
I was looking into the OverseerTest failure and the following caught my
attention.
This error can be due to a slow machine but it also happens on fast
machines if you try to do a lot of writes very fast on ZooKeeper which is
what the testShardLeaderChange does. Perhaps we should add a small wait
between operations?
[junit4] 2> 472241 T2893 oazsp.FileTxnLog.commit WARN fsync-ing
the write ahead log in SyncThread:0 took 11588ms which will adversely
effect operation latency. See the ZooKeeper troubleshooting guide
On Sun, Jul 6, 2014 at 10:46 AM, Policeman Jenkins Server <
jenkins@thetaphi.de> wrote:
> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/10616/
> Java: 32bit/jdk1.8.0_05 -client -XX:+UseParallelGC
>
> 5 tests failed.
> FAILED: junit.framework.TestSuite.org.apache.solr.cloud.OverseerTest
>
> Error Message:
> ERROR: SolrZkClient opens=165 closes=164
>
> Stack Trace:
> java.lang.AssertionError: ERROR: SolrZkClient opens=165 closes=164
> at __randomizedtesting.SeedInfo.seed([A819205877C572F6]:0)
> at org.junit.Assert.fail(Assert.java:93)
> at
> org.apache.solr.SolrTestCaseJ4.endTrackingZkClients(SolrTestCaseJ4.java:436)
> at
> org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:182)
> at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:790)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
> at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
> at
> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
> at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
> at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
> at
> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
> at java.lang.Thread.run(Thread.java:745)
>
>
> FAILED: junit.framework.TestSuite.org.apache.solr.cloud.OverseerTest
>
> Error Message:
> 2 threads leaked from SUITE scope at org.apache.solr.cloud.OverseerTest:
> 1) Thread[id=2969,
> name=TEST-OverseerTest.testShardLeaderChange-seed#[A819205877C572F6]-SendThread(
> 127.0.0.1:33880), state=TIMED_WAITING, group=TGRP-OverseerTest]
> at java.lang.Thread.sleep(Native Method) at
> org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003) 2)
> Thread[id=2970,
> name=TEST-OverseerTest.testShardLeaderChange-seed#[A819205877C572F6]-EventThread,
> state=WAITING, group=TGRP-OverseerTest] at
> sun.misc.Unsafe.park(Native Method) at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
>
> Stack Trace:
> com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from
> SUITE scope at org.apache.solr.cloud.OverseerTest:
> 1) Thread[id=2969,
> name=TEST-OverseerTest.testShardLeaderChange-seed#[A819205877C572F6]-SendThread(
> 127.0.0.1:33880), state=TIMED_WAITING, group=TGRP-OverseerTest]
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
> 2) Thread[id=2970,
> name=TEST-OverseerTest.testShardLeaderChange-seed#[A819205877C572F6]-EventThread,
> state=WAITING, group=TGRP-OverseerTest]
> at sun.misc.Unsafe.park(Native Method)
> at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
> at __randomizedtesting.SeedInfo.seed([A819205877C572F6]:0)
>
>
> FAILED: junit.framework.TestSuite.org.apache.solr.cloud.OverseerTest
>
> Error Message:
> There are still zombie threads that couldn't be terminated: 1)
> Thread[id=2969,
> name=TEST-OverseerTest.testShardLeaderChange-seed#[A819205877C572F6]-SendThread(
> 127.0.0.1:33880), state=TIMED_WAITING, group=TGRP-OverseerTest]
> at java.lang.Thread.sleep(Native Method) at
> org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
>
> Stack Trace:
> com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie
> threads that couldn't be terminated:
> 1) Thread[id=2969,
> name=TEST-OverseerTest.testShardLeaderChange-seed#[A819205877C572F6]-SendThread(
> 127.0.0.1:33880), state=TIMED_WAITING, group=TGRP-OverseerTest]
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
> at __randomizedtesting.SeedInfo.seed([A819205877C572F6]:0)
>
>
> REGRESSION: org.apache.solr.cloud.OverseerTest.testShardLeaderChange
>
> Error Message:
> KeeperErrorCode = NodeExists for /live_nodes/node1
>
> Stack Trace:
> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode
> = NodeExists for /live_nodes/node1
> at
> __randomizedtesting.SeedInfo.seed([A819205877C572F6:764AA7AF6D5D8707]:0)
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
> at
> org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:432)
> at
> org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
> at
> org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:429)
> at
> org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:386)
> at
> org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:373)
> at
> org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:350)
> at
> org.apache.solr.cloud.OverseerTest$MockZKController.<init>(OverseerTest.java:85)
> at
> org.apache.solr.cloud.OverseerTest.testShardLeaderChange(OverseerTest.java:746)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
> at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
> at
> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
> at
> org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
> at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
> at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> at
> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
> at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
> at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
> at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
> at
> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
> at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
> at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
> at
> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
> at java.lang.Thread.run(Thread.java:745)
>
>
> FAILED: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch
>
> Error Message:
> shard2 is not consistent. Got 153 from
> http://127.0.0.1:51995/collection1lastClient and got 132 from
> http://127.0.0.1:36739/collection1
>
> Stack Trace:
> java.lang.AssertionError: shard2 is not consistent. Got 153 from
> http://127.0.0.1:51995/collection1lastClient and got 132 from
> http://127.0.0.1:36739/collection1
> at
> __randomizedtesting.SeedInfo.seed([A819205877C572F6:29FFAE40009A12CA]:0)
> at org.junit.Assert.fail(Assert.java:93)
> at
> org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1139)
> at
> org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1118)
> at
> org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:150)
> at
> org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
> at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
> at
> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
> at
> org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
> at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
> at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> at
> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
> at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
> at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
> at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
> at
> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
> at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
> at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
> at
> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
> at java.lang.Thread.run(Thread.java:745)
>
>
>
>
> Build Log:
> [...truncated 11133 lines...]
> [junit4] Suite: org.apache.solr.cloud.OverseerTest
> [junit4] 2> Creating dataDir:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.OverseerTest-A819205877C572F6-001/init-core-data-001
> [junit4] 2> 443102 T2522 oas.SolrTestCaseJ4.buildSSLConfig Randomized
> ssl (false) and clientAuth (false)
> [junit4] 2> 443103 T2522 oas.SolrTestCaseJ4.initCore ####initCore
> [junit4] 2> 443103 T2522 oas.SolrTestCaseJ4.initCore ####initCore end
> [junit4] 2> 443107 T2522 oas.SolrTestCaseJ4.setUp ###Starting
> testStateChange
> [junit4] 2> 443107 T2522 oasc.ZkTestServer.run STARTING ZK TEST SERVER
> [junit4] 1> client port:0.0.0.0/0.0.0.0:0
> [junit4] 2> 443108 T2523 oasc.ZkTestServer$ZKServerMain.runFromConfig
> Starting server
> [junit4] 2> 443208 T2522 oasc.ZkTestServer.run start zk server on
> port:45124
> [junit4] 2> 443209 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 443282 T2529 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@ed72a5
> name:ZooKeeperConnection Watcher:127.0.0.1:45124/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 443283 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 443284 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 443287 T2531 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@6366c4
> name:ZooKeeperConnection Watcher:127.0.0.1:45124 got event WatchedEvent
> state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 443287 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 443290 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 443292 T2533 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@60240b
> name:ZooKeeperConnection Watcher:127.0.0.1:45124 got event WatchedEvent
> state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 443292 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 443293 T2522 oascc.SolrZkClient.makePath makePath: /solr
> [junit4] 2> 443298 T2522 oascc.SolrZkClient.makePath makePath:
> /live_nodes
> [junit4] 2> 443301 T2522 oascc.SolrZkClient.makePath makePath:
> /clusterstate.json
> [junit4] 2> 443304 T2522 oascc.SolrZkClient.makePath makePath:
> /aliases.json
> [junit4] 2> 443306 T2522
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
> [junit4] 2> 443311 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 443312 T2535 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@68e786
> name:ZooKeeperConnection Watcher:127.0.0.1:45124/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 443313 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 443314 T2522 oascc.SolrZkClient.makePath makePath:
> /overseer_elect
> [junit4] 2> 443318 T2522 oascc.SolrZkClient.makePath makePath:
> /overseer_elect/election
> [junit4] 2> 443324 T2522
> oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader
> 127.0.0.1:45124_solr
> [junit4] 2> 443324 T2522 oascc.SolrZkClient.makePath makePath:
> /overseer_elect/leader
> [junit4] 2> 443328 T2522 oasc.Overseer.start Overseer
> (id=92053395863699459-127.0.0.1:45124_solr-n_0000000000) starting
> [junit4] 2> 443331 T2522 oascc.SolrZkClient.makePath makePath:
> /overseer/queue
> [junit4] 2> 443336 T2522 oascc.SolrZkClient.makePath makePath:
> /overseer/queue-work
> [junit4] 2> 443340 T2522 oascc.SolrZkClient.makePath makePath:
> /overseer/collection-map-failure
> [junit4] 2> 443345 T2522 oascc.SolrZkClient.makePath makePath:
> /overseer/collection-map-running
> [junit4] 2> 443350 T2522 oascc.SolrZkClient.makePath makePath:
> /overseer/collection-map-completed
> [junit4] 2> 443357 T2522 oascc.SolrZkClient.makePath makePath:
> /overseer/collection-queue-work
> [junit4] 2> 443365 T2537 oasc.OverseerCollectionProcessor.run Process
> current queue of collection creations
> [junit4] 2> 443366 T2536 oasc.Overseer$ClusterStateUpdater.run
> Starting to work on the main queue
> [junit4] 2> 443368 T2522 oascc.ZkStateReader.updateClusterState
> Updating cloud state from ZooKeeper...
> [junit4] 2> 443369 T2536 oascc.ZkStateReader.updateClusterState
> Updating cloud state from ZooKeeper...
> [junit4] 2> 443370 T2536
> oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null
> message={
> [junit4] 2> "operation":"state",
> [junit4] 2> "base_url":"http://127.0.0.1/solr",
> [junit4] 2> "node_name":"node1",
> [junit4] 2> "collection":"collection1",
> [junit4] 2> "core":"core1",
> [junit4] 2> "roles":"",
> [junit4] 2> "state":"recovering"}
> [junit4] 2> 443370 T2536
> oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard
> shard=shard1
> [junit4] 2> 443374 T2535
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
> [junit4] 2> 443375 T2529 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
> [junit4] 2> 443420 T2522 oascc.ZkStateReader.updateClusterState
> Updating cloud state from ZooKeeper...
> [junit4] 2> 443422 T2535
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
> [junit4] 2> 443424 T2536 oascc.ZkStateReader.updateClusterState
> Updating cloud state from ZooKeeper...
> [junit4] 2> 443425 T2536
> oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null
> message={
> [junit4] 2> "operation":"state",
> [junit4] 2> "base_url":"http://127.0.0.1/solr",
> [junit4] 2> "node_name":"node1",
> [junit4] 2> "collection":"collection1",
> [junit4] 2> "core":"core1",
> [junit4] 2> "roles":"",
> [junit4] 2> "state":"active"}
> [junit4] 2> 443426 T2536
> oasc.Overseer$ClusterStateUpdater.updateState node=core_node1 is already
> registered
> [junit4] 2> 443426 T2536
> oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already
> registered
> [junit4] 2> 443431 T2535
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
> [junit4] 2> 443534 T2529 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
> [junit4] 2> 443575 T2522 oasc.ZkTestServer.send4LetterWord connecting
> to 127.0.0.1:45124 45124
> [junit4] 2> 443988 T2523 oasc.ZkTestServer.send4LetterWord connecting
> to 127.0.0.1:45124 45124
> [junit4] 2> 443991 T2522 oas.SolrTestCaseJ4.tearDown ###Ending
> testStateChange
> [junit4] 2> 443992 T2536 oasc.Overseer$ClusterStateUpdater.run
> Overseer Loop exiting : 127.0.0.1:45124_solr
> [junit4] 2> 443994 T2538
> oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not
> read the data org.apache.zookeeper.KeeperException$SessionExpiredException:
> KeeperErrorCode = Session expired for /overseer_elect/leader
> [junit4] 2> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
> [junit4] 2> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> [junit4] 2> at
> org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
> [junit4] 2> at
> org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
> [junit4] 2> at
> org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
> [junit4] 2> at
> org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
> [junit4] 2> at
> org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
> [junit4] 2> at
> org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:316)
> [junit4] 2> at
> org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
> [junit4] 2> at
> org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:288)
> [junit4] 2>
> [junit4] 2> 444000 T2522 oas.SolrTestCaseJ4.setUp ###Starting
> testShardAssignment
> [junit4] 2> 444001 T2522 oasc.ZkTestServer.run STARTING ZK TEST SERVER
> [junit4] 1> client port:0.0.0.0/0.0.0.0:0
> [junit4] 2> 444002 T2539 oasc.ZkTestServer$ZKServerMain.runFromConfig
> Starting server
> [junit4] 2> 444101 T2522 oasc.ZkTestServer.run start zk server on
> port:57742
> [junit4] 2> 444103 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 444107 T2545 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@9fc604
> name:ZooKeeperConnection Watcher:127.0.0.1:57742 got event WatchedEvent
> state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 444108 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 444111 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 444112 T2547 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@104f2e0
> name:ZooKeeperConnection Watcher:127.0.0.1:57742 got event WatchedEvent
> state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 444113 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 444114 T2522 oascc.SolrZkClient.makePath makePath: /solr
> [junit4] 2> 444119 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 444121 T2549 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@896251
> name:ZooKeeperConnection Watcher:127.0.0.1:57742/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 444121 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 444122 T2522 oascc.SolrZkClient.makePath makePath:
> /live_nodes
> [junit4] 2> 444125 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 444127 T2551 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@10c3e42
> name:ZooKeeperConnection Watcher:127.0.0.1:57742/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 444127 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 444128 T2522 oascc.SolrZkClient.makePath makePath:
> /overseer_elect
> [junit4] 2> 444132 T2522 oascc.SolrZkClient.makePath makePath:
> /overseer_elect/election
> [junit4] 2> 444137 T2522
> oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader
> 127.0.0.1:57742_solr
> [junit4] 2> 444138 T2522 oascc.SolrZkClient.makePath makePath:
> /overseer_elect/leader
> [junit4] 2> 444141 T2522 oasc.Overseer.start Overseer
> (id=92053395922223107-127.0.0.1:57742_solr-n_0000000000) starting
> [junit4] 2> 444144 T2522 oascc.SolrZkClient.makePath makePath:
> /overseer/queue
> [junit4] 2> 444149 T2522 oascc.SolrZkClient.makePath makePath:
> /overseer/queue-work
> [junit4] 2> 444153 T2522 oascc.SolrZkClient.makePath makePath:
> /overseer/collection-map-failure
> [junit4] 2> 444157 T2522 oascc.SolrZkClient.makePath makePath:
> /overseer/collection-map-running
> [junit4] 2> 444162 T2522 oascc.SolrZkClient.makePath makePath:
> /overseer/collection-map-completed
> [junit4] 2> 444166 T2522 oascc.SolrZkClient.makePath makePath:
> /overseer/collection-queue-work
> [junit4] 2> 444173 T2553 oasc.OverseerCollectionProcessor.run Process
> current queue of collection creations
> [junit4] 2> 444175 T2522 oascc.SolrZkClient.makePath makePath:
> /clusterstate.json
> [junit4] 2> 444175 T2552 oasc.Overseer$ClusterStateUpdater.run
> Starting to work on the main queue
> [junit4] 2> 444177 T2522 oascc.SolrZkClient.makePath makePath:
> /aliases.json
> [junit4] 2> 444179 T2522
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
> [junit4] 2> 444183 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 444185 T2555 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@1196b81
> name:ZooKeeperConnection Watcher:127.0.0.1:57742/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 444185 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 444187 T2522
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
> [junit4] 2> 444190 T2522 oascc.SolrZkClient.makePath makePath:
> /live_nodes/127.0.0.1
> [junit4] 2> 444194 T2549 oascc.ZkStateReader$3.process Updating live
> nodes... (1)
> [junit4] 2> 444194 T2555 oascc.ZkStateReader$3.process Updating live
> nodes... (1)
> [junit4] 2> 444197 T2551
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
> [junit4] 2> 444199 T2552 oascc.ZkStateReader.updateClusterState
> Updating cloud state from ZooKeeper...
> [junit4] 2> 444199 T2552
> oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3
> message={
> [junit4] 2> "operation":"state",
> [junit4] 2> "state":"active",
> [junit4] 2> "node_name":"127.0.0.1",
> [junit4] 2> "core":"core1",
> [junit4] 2> "core_node_name":"node1",
> [junit4] 2> "collection":"collection1",
> [junit4] 2> "numShards":"3",
> [junit4] 2> "base_url":"http://127.0.0.1/solr/"}
> [junit4] 2> 444200 T2552
> oasc.Overseer$ClusterStateUpdater.createCollection Create collection
> collection1 with shards [shard1, shard2, shard3]
> [junit4] 2> 444200 T2552
> oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard
> shard=shard2
> [junit4] 2> 444204 T2551
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
> [junit4] 2> 444205 T2555 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> [junit4] 2> 444205 T2549 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> [junit4] 2> 444697 T2522 oascc.SolrZkClient.makePath makePath:
> /collections/collection1/leader_elect/shard2/election
> [junit4] 2> 444712 T2522 oascc.SolrZkClient.makePath makePath:
> /collections/collection1/leaders/shard2
> [junit4] 2> 444719 T2551
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
> [junit4] 2> 444721 T2551
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
> [junit4] 2> 444721 T2552 oascc.ZkStateReader.updateClusterState
> Updating cloud state from ZooKeeper...
> [junit4] 2> 444727 T2552
> oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3
> message={
> [junit4] 2> "operation":"state",
> [junit4] 2> "state":"active",
> [junit4] 2> "node_name":"127.0.0.1",
> [junit4] 2> "core":"core2",
> [junit4] 2> "core_node_name":"node2",
> [junit4] 2> "collection":"collection1",
> [junit4] 2> "numShards":"3",
> [junit4] 2> "base_url":"http://127.0.0.1/solr/"}
> [junit4] 2> 444728 T2552
> oasc.Overseer$ClusterStateUpdater.updateState Collection already exists
> with numShards=3
> [junit4] 2> 444728 T2552
> oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard
> shard=shard3
> [junit4] 2> 444731 T2551
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
> [junit4] 2> 444834 T2555 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> [junit4] 2> 444834 T2549 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> [junit4] 2> 445222 T2522 oascc.SolrZkClient.makePath makePath:
> /collections/collection1/leader_elect/shard3/election
> [junit4] 2> 445233 T2522 oascc.SolrZkClient.makePath makePath:
> /collections/collection1/leaders/shard3
> [junit4] 2> 445240 T2551
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
> [junit4] 2> 445240 T2551
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
> [junit4] 2> 445241 T2551
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
> [junit4] 2> 445242 T2551
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
> [junit4] 2> 445242 T2552 oascc.ZkStateReader.updateClusterState
> Updating cloud state from ZooKeeper...
> [junit4] 2> 445248 T2552
> oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3
> message={
> [junit4] 2> "operation":"state",
> [junit4] 2> "state":"active",
> [junit4] 2> "node_name":"127.0.0.1",
> [junit4] 2> "core":"core3",
> [junit4] 2> "core_node_name":"node3",
> [junit4] 2> "collection":"collection1",
> [junit4] 2> "numShards":"3",
> [junit4] 2> "base_url":"http://127.0.0.1/solr/"}
> [junit4] 2> 445248 T2552
> oasc.Overseer$ClusterStateUpdater.updateState Collection already exists
> with numShards=3
> [junit4] 2> 445248 T2552
> oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard
> shard=shard1
> [junit4] 2> 445251 T2551
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
> [junit4] 2> 445354 T2549 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> [junit4] 2> 445354 T2555 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> [junit4] 2> 445742 T2522 oascc.SolrZkClient.makePath makePath:
> /collections/collection1/leader_elect/shard1/election
> [junit4] 2> 445753 T2522 oascc.SolrZkClient.makePath makePath:
> /collections/collection1/leaders/shard1
> [junit4] 2> 445764 T2551
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
> [junit4] 2> 445765 T2551
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
> [junit4] 2> 445765 T2551
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
> [junit4] 2> 445766 T2551
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
> [junit4] 2> 445767 T2552 oascc.ZkStateReader.updateClusterState
> Updating cloud state from ZooKeeper...
> [junit4] 2> 445771 T2552
> oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3
> message={
> [junit4] 2> "operation":"state",
> [junit4] 2> "state":"active",
> [junit4] 2> "node_name":"127.0.0.1",
> [junit4] 2> "core":"core4",
> [junit4] 2> "core_node_name":"node4",
> [junit4] 2> "collection":"collection1",
> [junit4] 2> "numShards":"3",
> [junit4] 2> "base_url":"http://127.0.0.1/solr/"}
> [junit4] 2> 445771 T2552
> oasc.Overseer$ClusterStateUpdater.updateState Collection already exists
> with numShards=3
> [junit4] 2> 445771 T2552
> oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard
> shard=shard2
> [junit4] 2> 445773 T2551
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
> [junit4] 2> 445876 T2555 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> [junit4] 2> 445876 T2549 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> [junit4] 2> 446267 T2522 oascc.SolrZkClient.makePath makePath:
> /collections/collection1/leader_elect/shard2/election
> [junit4] 2> 446276 T2551
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
> [junit4] 2> 446276 T2551
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
> [junit4] 2> 446276 T2551
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
> [junit4] 2> 446278 T2552 oascc.ZkStateReader.updateClusterState
> Updating cloud state from ZooKeeper...
> [junit4] 2> 446279 T2552
> oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3
> message={
> [junit4] 2> "operation":"state",
> [junit4] 2> "state":"active",
> [junit4] 2> "node_name":"127.0.0.1",
> [junit4] 2> "core":"core5",
> [junit4] 2> "core_node_name":"node5",
> [junit4] 2> "collection":"collection1",
> [junit4] 2> "numShards":"3",
> [junit4] 2> "base_url":"http://127.0.0.1/solr/"}
> [junit4] 2> 446280 T2552
> oasc.Overseer$ClusterStateUpdater.updateState Collection already exists
> with numShards=3
> [junit4] 2> 446280 T2552
> oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard
> shard=shard3
> [junit4] 2> 446283 T2551
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
> [junit4] 2> 446386 T2555 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> [junit4] 2> 446386 T2549 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> [junit4] 2> 446776 T2522 oascc.SolrZkClient.makePath makePath:
> /collections/collection1/leader_elect/shard3/election
> [junit4] 2> 446787 T2551
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
> [junit4] 2> 446788 T2551
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
> [junit4] 2> 446788 T2551
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
> [junit4] 2> 446790 T2552 oascc.ZkStateReader.updateClusterState
> Updating cloud state from ZooKeeper...
> [junit4] 2> 446791 T2552
> oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3
> message={
> [junit4] 2> "operation":"state",
> [junit4] 2> "state":"active",
> [junit4] 2> "node_name":"127.0.0.1",
> [junit4] 2> "core":"core6",
> [junit4] 2> "core_node_name":"node6",
> [junit4] 2> "collection":"collection1",
> [junit4] 2> "numShards":"3",
> [junit4] 2> "base_url":"http://127.0.0.1/solr/"}
> [junit4] 2> 446791 T2552
> oasc.Overseer$ClusterStateUpdater.updateState Collection already exists
> with numShards=3
> [junit4] 2> 446791 T2552
> oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard
> shard=shard1
> [junit4] 2> 446795 T2551
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
> [junit4] 2> 446899 T2549 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> [junit4] 2> 446899 T2555 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> [junit4] 2> 447288 T2522 oascc.SolrZkClient.makePath makePath:
> /collections/collection1/leader_elect/shard1/election
> [junit4] 2> 447298 T2555 oascc.ZkStateReader$3.process Updating live
> nodes... (0)
> [junit4] 2> 447301 T2522 oasc.ZkTestServer.send4LetterWord connecting
> to 127.0.0.1:57742 57742
> [junit4] 2> 447869 T2539 oasc.ZkTestServer.send4LetterWord connecting
> to 127.0.0.1:57742 57742
> [junit4] 2> 447871 T2522 oas.SolrTestCaseJ4.tearDown ###Ending
> testShardAssignment
> [junit4] 2> 447872 T2552 oasc.Overseer$ClusterStateUpdater.run
> Overseer Loop exiting : 127.0.0.1:57742_solr
> [junit4] 2> 447875 T2556
> oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not
> read the data org.apache.zookeeper.KeeperException$SessionExpiredException:
> KeeperErrorCode = Session expired for /overseer_elect/leader
> [junit4] 2> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
> [junit4] 2> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> [junit4] 2> at
> org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
> [junit4] 2> at
> org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
> [junit4] 2> at
> org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
> [junit4] 2> at
> org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
> [junit4] 2> at
> org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
> [junit4] 2> at
> org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:316)
> [junit4] 2> at
> org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
> [junit4] 2> at
> org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:288)
> [junit4] 2>
> [junit4] 2> 447881 T2522 oas.SolrTestCaseJ4.setUp ###Starting
> testShardAssignmentBigger
> [junit4] 2> 447882 T2522 oasc.ZkTestServer.run STARTING ZK TEST SERVER
> [junit4] 1> client port:0.0.0.0/0.0.0.0:0
> [junit4] 2> 447883 T2557 oasc.ZkTestServer$ZKServerMain.runFromConfig
> Starting server
> [junit4] 2> 447983 T2522 oasc.ZkTestServer.run start zk server on
> port:50146
> [junit4] 2> 447984 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 447988 T2563 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@daa195
> name:ZooKeeperConnection Watcher:127.0.0.1:50146 got event WatchedEvent
> state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 447989 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 447993 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 447994 T2565 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@f8aac
> name:ZooKeeperConnection Watcher:127.0.0.1:50146 got event WatchedEvent
> state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 447995 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 447995 T2522 oascc.SolrZkClient.makePath makePath: /solr
> [junit4] 2> 448000 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 448002 T2567 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@141291c
> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 448002 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 448003 T2522 oascc.SolrZkClient.makePath makePath:
> /live_nodes
> [junit4] 2> 448006 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 448008 T2569 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@49f99d
> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 448008 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 448009 T2522 oascc.SolrZkClient.makePath makePath:
> /overseer_elect
> [junit4] 2> 448012 T2522 oascc.SolrZkClient.makePath makePath:
> /overseer_elect/election
> [junit4] 2> 448018 T2522
> oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader
> 127.0.0.1:50146_solr
> [junit4] 2> 448018 T2522 oascc.SolrZkClient.makePath makePath:
> /overseer_elect/leader
> [junit4] 2> 448021 T2522 oasc.Overseer.start Overseer
> (id=92053396176633859-127.0.0.1:50146_solr-n_0000000000) starting
> [junit4] 2> 448024 T2522 oascc.SolrZkClient.makePath makePath:
> /overseer/queue
> [junit4] 2> 448029 T2522 oascc.SolrZkClient.makePath makePath:
> /overseer/queue-work
> [junit4] 2> 448034 T2522 oascc.SolrZkClient.makePath makePath:
> /overseer/collection-map-failure
> [junit4] 2> 448038 T2522 oascc.SolrZkClient.makePath makePath:
> /overseer/collection-map-running
> [junit4] 2> 448043 T2522 oascc.SolrZkClient.makePath makePath:
> /overseer/collection-map-completed
> [junit4] 2> 448048 T2522 oascc.SolrZkClient.makePath makePath:
> /overseer/collection-queue-work
> [junit4] 2> 448055 T2571 oasc.OverseerCollectionProcessor.run Process
> current queue of collection creations
> [junit4] 2> 448056 T2522 oascc.SolrZkClient.makePath makePath:
> /clusterstate.json
> [junit4] 2> 448056 T2570 oasc.Overseer$ClusterStateUpdater.run
> Starting to work on the main queue
> [junit4] 2> 448058 T2522 oascc.SolrZkClient.makePath makePath:
> /aliases.json
> [junit4] 2> 448061 T2522
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
> [junit4] 2> 448065 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 448066 T2573 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@1242e08
> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 448067 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 448069 T2522
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
> [junit4] 2> 448072 T2522 oascc.SolrZkClient.makePath makePath:
> /live_nodes/node0
> [junit4] 2> 448076 T2573 oascc.ZkStateReader$3.process Updating live
> nodes... (1)
> [junit4] 2> 448076 T2567 oascc.ZkStateReader$3.process Updating live
> nodes... (1)
> [junit4] 2> 448077 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 448078 T2575 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@1fc812a
> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 448078 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 448080 T2522
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
> [junit4] 2> 448083 T2522 oascc.SolrZkClient.makePath makePath:
> /live_nodes/node1
> [junit4] 2> 448087 T2573 oascc.ZkStateReader$3.process Updating live
> nodes... (2)
> [junit4] 2> 448087 T2567 oascc.ZkStateReader$3.process Updating live
> nodes... (2)
> [junit4] 2> 448087 T2575 oascc.ZkStateReader$3.process Updating live
> nodes... (2)
> [junit4] 2> 448088 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 448089 T2577 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@1714ec0
> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 448090 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 448091 T2522
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
> [junit4] 2> 448095 T2522 oascc.SolrZkClient.makePath makePath:
> /live_nodes/node2
> [junit4] 2> 448099 T2573 oascc.ZkStateReader$3.process Updating live
> nodes... (3)
> [junit4] 2> 448099 T2577 oascc.ZkStateReader$3.process Updating live
> nodes... (3)
> [junit4] 2> 448099 T2575 oascc.ZkStateReader$3.process Updating live
> nodes... (3)
> [junit4] 2> 448100 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 448099 T2567 oascc.ZkStateReader$3.process Updating live
> nodes... (3)
> [junit4] 2> 448102 T2579 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@fbc5e2
> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 448102 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 448104 T2522
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
> [junit4] 2> 448107 T2522 oascc.SolrZkClient.makePath makePath:
> /live_nodes/node3
> [junit4] 2> 448112 T2577 oascc.ZkStateReader$3.process Updating live
> nodes... (4)
> [junit4] 2> 448112 T2575 oascc.ZkStateReader$3.process Updating live
> nodes... (4)
> [junit4] 2> 448112 T2567 oascc.ZkStateReader$3.process Updating live
> nodes... (4)
> [junit4] 2> 448112 T2573 oascc.ZkStateReader$3.process Updating live
> nodes... (4)
> [junit4] 2> 448113 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 448113 T2579 oascc.ZkStateReader$3.process Updating live
> nodes... (4)
> [junit4] 2> 448115 T2581 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@75e1db
> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 448115 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 448117 T2522
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
> [junit4] 2> 448121 T2522 oascc.SolrZkClient.makePath makePath:
> /live_nodes/node4
> [junit4] 2> 448126 T2577 oascc.ZkStateReader$3.process Updating live
> nodes... (5)
> [junit4] 2> 448126 T2575 oascc.ZkStateReader$3.process Updating live
> nodes... (5)
> [junit4] 2> 448126 T2573 oascc.ZkStateReader$3.process Updating live
> nodes... (5)
> [junit4] 2> 448127 T2581 oascc.ZkStateReader$3.process Updating live
> nodes... (5)
> [junit4] 2> 448126 T2567 oascc.ZkStateReader$3.process Updating live
> nodes... (5)
> [junit4] 2> 448126 T2579 oascc.ZkStateReader$3.process Updating live
> nodes... (5)
> [junit4] 2> 448127 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 448129 T2583 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@11dc02e
> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 448130 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 448132 T2522
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
> [junit4] 2> 448136 T2522 oascc.SolrZkClient.makePath makePath:
> /live_nodes/node5
> [junit4] 2> 448139 T2575 oascc.ZkStateReader$3.process Updating live
> nodes... (6)
> [junit4] 2> 448140 T2577 oascc.ZkStateReader$3.process Updating live
> nodes... (6)
> [junit4] 2> 448140 T2581 oascc.ZkStateReader$3.process Updating live
> nodes... (6)
> [junit4] 2> 448140 T2573 oascc.ZkStateReader$3.process Updating live
> nodes... (6)
> [junit4] 2> 448140 T2579 oascc.ZkStateReader$3.process Updating live
> nodes... (6)
> [junit4] 2> 448141 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 448141 T2583 oascc.ZkStateReader$3.process Updating live
> nodes... (6)
> [junit4] 2> 448140 T2567 oascc.ZkStateReader$3.process Updating live
> nodes... (6)
> [junit4] 2> 448143 T2585 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@6d9f03
> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 448143 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 448145 T2522
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
> [junit4] 2> 448148 T2522 oascc.SolrZkClient.makePath makePath:
> /live_nodes/node6
> [junit4] 2> 448153 T2575 oascc.ZkStateReader$3.process Updating live
> nodes... (7)
> [junit4] 2> 448153 T2577 oascc.ZkStateReader$3.process Updating live
> nodes... (7)
> [junit4] 2> 448153 T2573 oascc.ZkStateReader$3.process Updating live
> nodes... (7)
> [junit4] 2> 448153 T2585 oascc.ZkStateReader$3.process Updating live
> nodes... (7)
> [junit4] 2> 448153 T2583 oascc.ZkStateReader$3.process Updating live
> nodes... (7)
> [junit4] 2> 448153 T2579 oascc.ZkStateReader$3.process Updating live
> nodes... (7)
> [junit4] 2> 448153 T2581 oascc.ZkStateReader$3.process Updating live
> nodes... (7)
> [junit4] 2> 448153 T2567 oascc.ZkStateReader$3.process Updating live
> nodes... (7)
> [junit4] 2> 448154 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 448156 T2587 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@958679
> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 448157 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 448159 T2522
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
> [junit4] 2> 448162 T2522 oascc.SolrZkClient.makePath makePath:
> /live_nodes/node7
> [junit4] 2> 448167 T2567 oascc.ZkStateReader$3.process Updating live
> nodes... (8)
> [junit4] 2> 448167 T2577 oascc.ZkStateReader$3.process Updating live
> nodes... (8)
> [junit4] 2> 448168 T2585 oascc.ZkStateReader$3.process Updating live
> nodes... (8)
> [junit4] 2> 448168 T2583 oascc.ZkStateReader$3.process Updating live
> nodes... (8)
> [junit4] 2> 448168 T2581 oascc.ZkStateReader$3.process Updating live
> nodes... (8)
> [junit4] 2> 448168 T2579 oascc.ZkStateReader$3.process Updating live
> nodes... (8)
> [junit4] 2> 448168 T2573 oascc.ZkStateReader$3.process Updating live
> nodes... (8)
> [junit4] 2> 448167 T2587 oascc.ZkStateReader$3.process Updating live
> nodes... (8)
> [junit4] 2> 448167 T2575 oascc.ZkStateReader$3.process Updating live
> nodes... (8)
> [junit4] 2> 448169 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 448171 T2589 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@169d949
> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 448171 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 448174 T2522
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
> [junit4] 2> 448177 T2522 oascc.SolrZkClient.makePath makePath:
> /live_nodes/node8
> [junit4] 2> 448182 T2589 oascc.ZkStateReader$3.process Updating live
> nodes... (9)
> [junit4] 2> 448183 T2585 oascc.ZkStateReader$3.process Updating live
> nodes... (9)
> [junit4] 2> 448183 T2583 oascc.ZkStateReader$3.process Updating live
> nodes... (9)
> [junit4] 2> 448183 T2587 oascc.ZkStateReader$3.process Updating live
> nodes... (9)
> [junit4] 2> 448183 T2573 oascc.ZkStateReader$3.process Updating live
> nodes... (9)
> [junit4] 2> 448183 T2579 oascc.ZkStateReader$3.process Updating live
> nodes... (9)
> [junit4] 2> 448183 T2581 oascc.ZkStateReader$3.process Updating live
> nodes... (9)
> [junit4] 2> 448183 T2575 oascc.ZkStateReader$3.process Updating live
> nodes... (9)
> [junit4] 2> 448183 T2577 oascc.ZkStateReader$3.process Updating live
> nodes... (9)
> [junit4] 2> 448182 T2567 oascc.ZkStateReader$3.process Updating live
> nodes... (9)
> [junit4] 2> 448185 T2591 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@2e78fa
> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 448184 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 448188 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 448190 T2522
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
> [junit4] 2> 448192 T2522 oascc.SolrZkClient.makePath makePath:
> /live_nodes/node9
> [junit4] 2> 448195 T2589 oascc.ZkStateReader$3.process Updating live
> nodes... (10)
> [junit4] 2> 448195 T2577 oascc.ZkStateReader$3.process Updating live
> nodes... (10)
> [junit4] 2> 448195 T2567 oascc.ZkStateReader$3.process Updating live
> nodes... (10)
> [junit4] 2> 448196 T2573 oascc.ZkStateReader$3.process Updating live
> nodes... (10)
> [junit4] 2> 448195 T2575 oascc.ZkStateReader$3.process Updating live
> nodes... (10)
> [junit4] 2> 448195 T2587 oascc.ZkStateReader$3.process Updating live
> nodes... (10)
> [junit4] 2> 448195 T2579 oascc.ZkStateReader$3.process Updating live
> nodes... (10)
> [junit4] 2> 448195 T2583 oascc.ZkStateReader$3.process Updating live
> nodes... (10)
> [junit4] 2> 448196 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 448198 T2593 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@156c3b
> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 448196 T2591 oascc.ZkStateReader$3.process Updating live
> nodes... (10)
> [junit4] 2> 448196 T2585 oascc.ZkStateReader$3.process Updating live
> nodes... (10)
> [junit4] 2> 448196 T2581 oascc.ZkStateReader$3.process Updating live
> nodes... (10)
> [junit4] 2> 448198 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 448200 T2522
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
> [junit4] 2> 448203 T2522 oascc.SolrZkClient.makePath makePath:
> /live_nodes/node10
> [junit4] 2> 448207 T2575 oascc.ZkStateReader$3.process Updating live
> nodes... (11)
> [junit4] 2> 448207 T2583 oascc.ZkStateReader$3.process Updating live
> nodes... (11)
> [junit4] 2> 448208 T2579 oascc.ZkStateReader$3.process Updating live
> nodes... (11)
> [junit4] 2> 448207 T2577 oascc.ZkStateReader$3.process Updating live
> nodes... (11)
> [junit4] 2> 448207 T2573 oascc.ZkStateReader$3.process Updating live
> nodes... (11)
> [junit4] 2> 448207 T2589 oascc.ZkStateReader$3.process Updating live
> nodes... (11)
> [junit4] 2> 448207 T2587 oascc.ZkStateReader$3.process Updating live
> nodes... (11)
> [junit4] 2> 448209 T2585 oascc.ZkStateReader$3.process Updating live
> nodes... (11)
> [junit4] 2> 448208 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 448208 T2581 oascc.ZkStateReader$3.process Updating live
> nodes... (11)
> [junit4] 2> 448208 T2593 oascc.ZkStateReader$3.process Updating live
> nodes... (11)
> [junit4] 2> 448208 T2591 oascc.ZkStateReader$3.process Updating live
> nodes... (11)
> [junit4] 2> 448208 T2567 oascc.ZkStateReader$3.process Updating live
> nodes... (11)
> [junit4] 2> 448211 T2595 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@f1978e
> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 448212 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 448213 T2522
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
> [junit4] 2> 448216 T2522 oascc.SolrZkClient.makePath makePath:
> /live_nodes/node11
> [junit4] 2> 448220 T2587 oascc.ZkStateReader$3.process Updating live
> nodes... (12)
> [junit4] 2> 448220 T2567 oascc.ZkStateReader$3.process Updating live
> nodes... (12)
> [junit4] 2> 448221 T2583 oascc.ZkStateReader$3.process Updating live
> nodes... (12)
> [junit4] 2> 448220 T2589 oascc.ZkStateReader$3.process Updating live
> nodes... (12)
> [junit4] 2> 448222 T2595 oascc.ZkStateReader$3.process Updating live
> nodes... (12)
> [junit4] 2> 448220 T2575 oascc.ZkStateReader$3.process Updating live
> nodes... (12)
> [junit4] 2> 448222 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 448222 T2593 oascc.ZkStateReader$3.process Updating live
> nodes... (12)
> [junit4] 2> 448222 T2581 oascc.ZkStateReader$3.process Updating live
> nodes... (12)
> [junit4] 2> 448224 T2597 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@175be88
> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 448221 T2591 oascc.ZkStateReader$3.process Updating live
> nodes... (12)
> [junit4] 2> 448221 T2573 oascc.ZkStateReader$3.process Updating live
> nodes... (12)
> [junit4] 2> 448221 T2577 oascc.ZkStateReader$3.process Updating live
> nodes... (12)
> [junit4] 2> 448221 T2585 oascc.ZkStateReader$3.process Updating live
> nodes... (12)
> [junit4] 2> 448221 T2579 oascc.ZkStateReader$3.process Updating live
> nodes... (12)
> [junit4] 2> 448224 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 448228 T2522
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
> [junit4] 2> 448232 T2522 oascc.SolrZkClient.makePath makePath:
> /live_nodes/node12
> [junit4] 2> 448236 T2573 oascc.ZkStateReader$3.process Updating live
> nodes... (13)
> [junit4] 2> 448236 T2577 oascc.ZkStateReader$3.process Updating live
> nodes... (13)
> [junit4] 2> 448237 T2595 oascc.ZkStateReader$3.process Updating live
> nodes... (13)
> [junit4] 2> 448237 T2593 oascc.ZkStateReader$3.process Updating live
> nodes... (13)
> [junit4] 2> 448237 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 448238 T2581 oascc.ZkStateReader$3.process Updating live
> nodes... (13)
> [junit4] 2> 448238 T2597 oascc.ZkStateReader$3.process Updating live
> nodes... (13)
> [junit4] 2> 448237 T2583 oascc.ZkStateReader$3.process Updating live
> nodes... (13)
> [junit4] 2> 448239 T2599 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@dde0f4
> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 448237 T2589 oascc.ZkStateReader$3.process Updating live
> nodes... (13)
> [junit4] 2> 448237 T2575 oascc.ZkStateReader$3.process Updating live
> nodes... (13)
> [junit4] 2> 448237 T2587 oascc.ZkStateReader$3.process Updating live
> nodes... (13)
> [junit4] 2> 448237 T2585 oascc.ZkStateReader$3.process Updating live
> nodes... (13)
> [junit4] 2> 448236 T2567 oascc.ZkStateReader$3.process Updating live
> nodes... (13)
> [junit4] 2> 448236 T2579 oascc.ZkStateReader$3.process Updating live
> nodes... (13)
> [junit4] 2> 448236 T2591 oascc.ZkStateReader$3.process Updating live
> nodes... (13)
> [junit4] 2> 448240 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 448244 T2522
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
> [junit4] 2> 448248 T2522 oascc.SolrZkClient.makePath makePath:
> /live_nodes/node13
> [junit4] 2> 448254 T2587 oascc.ZkStateReader$3.process Updating live
> nodes... (14)
> [junit4] 2> 448255 T2577 oascc.ZkStateReader$3.process Updating live
> nodes... (14)
> [junit4] 2> 448255 T2579 oascc.ZkStateReader$3.process Updating live
> nodes... (14)
> [junit4] 2> 448255 T2595 oascc.ZkStateReader$3.process Updating live
> nodes... (14)
> [junit4] 2> 448255 T2573 oascc.ZkStateReader$3.process Updating live
> nodes... (14)
> [junit4] 2> 448256 T2597 oascc.ZkStateReader$3.process Updating live
> nodes... (14)
> [junit4] 2> 448256 T2599 oascc.ZkStateReader$3.process Updating live
> nodes... (14)
> [junit4] 2> 448256 T2593 oascc.ZkStateReader$3.process Updating live
> nodes... (14)
> [junit4] 2> 448256 T2585 oascc.ZkStateReader$3.process Updating live
> nodes... (14)
> [junit4] 2> 448256 T2581 oascc.ZkStateReader$3.process Updating live
> nodes... (14)
> [junit4] 2> 448256 T2583 oascc.ZkStateReader$3.process Updating live
> nodes... (14)
> [junit4] 2> 448256 T2567 oascc.ZkStateReader$3.process Updating live
> nodes... (14)
> [junit4] 2> 448256 T2575 oascc.ZkStateReader$3.process Updating live
> nodes... (14)
> [junit4] 2> 448256 T2589 oascc.ZkStateReader$3.process Updating live
> nodes... (14)
> [junit4] 2> 448256 T2591 oascc.ZkStateReader$3.process Updating live
> nodes... (14)
> [junit4] 2> 448259 T2601 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@19e3760
> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 448257 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 448262 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 448264 T2522
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
> [junit4] 2> 448268 T2522 oascc.SolrZkClient.makePath makePath:
> /live_nodes/node14
> [junit4] 2> 448273 T2573 oascc.ZkStateReader$3.process Updating live
> nodes... (15)
> [junit4] 2> 448273 T2587 oascc.ZkStateReader$3.process Updating live
> nodes... (15)
> [junit4] 2> 448273 T2579 oascc.ZkStateReader$3.process Updating live
> nodes... (15)
> [junit4] 2> 448273 T2591 oascc.ZkStateReader$3.process Updating live
> nodes... (15)
> [junit4] 2> 448273 T2577 oascc.ZkStateReader$3.process Updating live
> nodes... (15)
> [junit4] 2> 448273 T2595 oascc.ZkStateReader$3.process Updating live
> nodes... (15)
> [junit4] 2> 448275 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 448275 T2599 oascc.ZkStateReader$3.process Updating live
> nodes... (15)
> [junit4] 2> 448275 T2583 oascc.ZkStateReader$3.process Updating live
> nodes... (15)
> [junit4] 2> 448275 T2585 oascc.ZkStateReader$3.process Updating live
> nodes... (15)
> [junit4] 2> 448275 T2597 oascc.ZkStateReader$3.process Updating live
> nodes... (15)
> [junit4] 2> 448275 T2601 oascc.ZkStateReader$3.process Updating live
> nodes... (15)
> [junit4] 2> 448275 T2567 oascc.ZkStateReader$3.process Updating live
> nodes... (15)
> [junit4] 2> 448275 T2575 oascc.ZkStateReader$3.process Updating live
> nodes... (15)
> [junit4] 2> 448275 T2581 oascc.ZkStateReader$3.process Updating live
> nodes... (15)
> [junit4] 2> 448274 T2593 oascc.ZkStateReader$3.process Updating live
> nodes... (15)
> [junit4] 2> 448273 T2589 oascc.ZkStateReader$3.process Updating live
> nodes... (15)
> [junit4] 2> 448277 T2603 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@1818cfa
> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 448281 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 448283 T2522
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
> [junit4] 2> 448286 T2522 oascc.SolrZkClient.makePath makePath:
> /live_nodes/node15
> [junit4] 2> 448291 T2577 oascc.ZkStateReader$3.process Updating live
> nodes... (16)
> [junit4] 2> 448292 T2573 oascc.ZkStateReader$3.process Updating live
> nodes... (16)
> [junit4] 2> 448291 T2587 oascc.ZkStateReader$3.process Updating live
> nodes... (16)
> [junit4] 2> 448293 T2579 oascc.ZkStateReader$3.process Updating live
> nodes... (16)
> [junit4] 2> 448293 T2575 oascc.ZkStateReader$3.process Updating live
> nodes... (16)
> [junit4] 2> 448293 T2601 oascc.ZkStateReader$3.process Updating live
> nodes... (16)
> [junit4] 2> 448294 T2599 oascc.ZkStateReader$3.process Updating live
> nodes... (16)
> [junit4] 2> 448293 T2591 oascc.ZkStateReader$3.process Updating live
> nodes... (16)
> [junit4] 2> 448293 T2597 oascc.ZkStateReader$3.process Updating live
> nodes... (16)
> [junit4] 2> 448293 T2583 oascc.ZkStateReader$3.process Updating live
> nodes... (16)
> [junit4] 2> 448293 T2567 oascc.ZkStateReader$3.process Updating live
> nodes... (16)
> [junit4] 2> 448293 T2589 oascc.ZkStateReader$3.process Updating live
> nodes... (16)
> [junit4] 2> 448295 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 448294 T2603 oascc.ZkStateReader$3.process Updating live
> nodes... (16)
> [junit4] 2> 448294 T2581 oascc.ZkStateReader$3.process Updating live
> nodes... (16)
> [junit4] 2> 448294 T2585 oascc.ZkStateReader$3.process Updating live
> nodes... (16)
> [junit4] 2> 448294 T2595 oascc.ZkStateReader$3.process Updating live
> nodes... (16)
> [junit4] 2> 448294 T2593 oascc.ZkStateReader$3.process Updating live
> nodes... (16)
> [junit4] 2> 448297 T2605 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@154e83c
> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 448299 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 448300 T2522
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
> [junit4] 2> 448304 T2522 oascc.SolrZkClient.makePath makePath:
> /live_nodes/node16
> [junit4] 2> 448313 T2587 oascc.ZkStateReader$3.process Updating live
> nodes... (17)
> [junit4] 2> 448313 T2589 oascc.ZkStateReader$3.process Updating live
> nodes... (17)
> [junit4] 2> 448314 T2595 oascc.ZkStateReader$3.process Updating live
> nodes... (17)
> [junit4] 2> 448315 T2575 oascc.ZkStateReader$3.process Updating live
> nodes... (17)
> [junit4] 2> 448314 T2585 oascc.ZkStateReader$3.process Updating live
> nodes... (17)
> [junit4] 2> 448316 T2591 oascc.ZkStateReader$3.process Updating live
> nodes... (17)
> [junit4] 2> 448313 T2573 oascc.ZkStateReader$3.process Updating live
> nodes... (17)
> [junit4] 2> 448313 T2577 oascc.ZkStateReader$3.process Updating live
> nodes... (17)
> [junit4] 2> 448313 T2593 oascc.ZkStateReader$3.process Updating live
> nodes... (17)
> [junit4] 2> 448317 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 448316 T2599 oascc.ZkStateReader$3.process Updating live
> nodes... (17)
> [junit4] 2> 448316 T2581 oascc.ZkStateReader$3.process Updating live
> nodes... (17)
> [junit4] 2> 448316 T2605 oascc.ZkStateReader$3.process Updating live
> nodes... (17)
> [junit4] 2> 448316 T2603 oascc.ZkStateReader$3.process Updating live
> nodes... (17)
> [junit4] 2> 448316 T2583 oascc.ZkStateReader$3.process Updating live
> nodes... (17)
> [junit4] 2> 448316 T2567 oascc.ZkStateReader$3.process Updating live
> nodes... (17)
> [junit4] 2> 448315 T2601 oascc.ZkStateReader$3.process Updating live
> nodes... (17)
> [junit4] 2> 448315 T2597 oascc.ZkStateReader$3.process Updating live
> nodes... (17)
> [junit4] 2> 448315 T2579 oascc.ZkStateReader$3.process Updating live
> nodes... (17)
> [junit4] 2> 448319 T2607 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@d963a2
> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 448323 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 448324 T2522
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
> [junit4] 2> 448328 T2522 oascc.SolrZkClient.makePath makePath:
> /live_nodes/node17
> [junit4] 2> 448333 T2573 oascc.ZkStateReader$3.process Updating live
> nodes... (18)
> [junit4] 2> 448333 T2577 oascc.ZkStateReader$3.process Updating live
> nodes... (18)
> [junit4] 2> 448334 T2601 oascc.ZkStateReader$3.process Updating live
> nodes... (18)
> [junit4] 2> 448334 T2607 oascc.ZkStateReader$3.process Updating live
> nodes... (18)
> [junit4] 2> 448335 T2567 oascc.ZkStateReader$3.process Updating live
> nodes... (18)
> [junit4] 2> 448335 T2583 oascc.ZkStateReader$3.process Updating live
> nodes... (18)
> [junit4] 2> 448335 T2603 oascc.ZkStateReader$3.process Updating live
> nodes... (18)
> [junit4] 2> 448334 T2589 oascc.ZkStateReader$3.process Updating live
> nodes... (18)
> [junit4] 2> 448336 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 448334 T2595 oascc.ZkStateReader$3.process Updating live
> nodes... (18)
> [junit4] 2> 448334 T2579 oascc.ZkStateReader$3.process Updating live
> nodes... (18)
> [junit4] 2> 448334 T2575 oascc.ZkStateReader$3.process Updating live
> nodes... (18)
> [junit4] 2> 448338 T2609 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@826a2e
> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 448333 T2587 oascc.ZkStateReader$3.process Updating live
> nodes... (18)
> [junit4] 2> 448333 T2591 oascc.ZkStateReader$3.process Updating live
> nodes... (18)
> [junit4] 2> 448338 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 448336 T2585 oascc.ZkStateReader$3.process Updating live
> nodes... (18)
> [junit4] 2> 448335 T2593 oascc.ZkStateReader$3.process Updating live
> nodes... (18)
> [junit4] 2> 448335 T2599 oascc.ZkStateReader$3.process Updating live
> nodes... (18)
> [junit4] 2> 448335 T2581 oascc.ZkStateReader$3.process Updating live
> nodes... (18)
> [junit4] 2> 448335 T2605 oascc.ZkStateReader$3.process Updating live
> nodes... (18)
> [junit4] 2> 448335 T2597 oascc.ZkStateReader$3.process Updating live
> nodes... (18)
> [junit4] 2> 448341 T2522
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
> [junit4] 2> 448346 T2522 oascc.SolrZkClient.makePath makePath:
> /live_nodes/node18
> [junit4] 2> 448350 T2573 oascc.ZkStateReader$3.process Updating live
> nodes... (19)
> [junit4] 2> 448350 T2577 oascc.ZkStateReader$3.process Updating live
> nodes... (19)
> [junit4] 2> 448351 T2607 oascc.ZkStateReader$3.process Updating live
> nodes... (19)
> [junit4] 2> 448352 T2587 oascc.ZkStateReader$3.process Updating live
> nodes... (19)
> [junit4] 2> 448352 T2593 oascc.ZkStateReader$3.process Updating live
> nodes... (19)
> [junit4] 2> 448351 T2601 oascc.ZkStateReader$3.process Updating live
> nodes... (19)
> [junit4] 2> 448353 T2579 oascc.ZkStateReader$3.process Updating live
> nodes... (19)
> [junit4] 2> 448353 T2581 oascc.ZkStateReader$3.process Updating live
> nodes... (19)
> [junit4] 2> 448351 T2595 oascc.ZkStateReader$3.process Updating live
> nodes... (19)
> [junit4] 2> 448353 T2605 oascc.ZkStateReader$3.process Updating live
> nodes... (19)
> [junit4] 2> 448354 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 448353 T2609 oascc.ZkStateReader$3.process Updating live
> nodes... (19)
> [junit4] 2> 448353 T2603 oascc.ZkStateReader$3.process Updating live
> nodes... (19)
> [junit4] 2> 448353 T2597 oascc.ZkStateReader$3.process Updating live
> nodes... (19)
> [junit4] 2> 448353 T2589 oascc.ZkStateReader$3.process Updating live
> nodes... (19)
> [junit4] 2> 448353 T2567 oascc.ZkStateReader$3.process Updating live
> nodes... (19)
> [junit4] 2> 448353 T2591 oascc.ZkStateReader$3.process Updating live
> nodes... (19)
> [junit4] 2> 448353 T2575 oascc.ZkStateReader$3.process Updating live
> nodes... (19)
> [junit4] 2> 448352 T2599 oascc.ZkStateReader$3.process Updating live
> nodes... (19)
> [junit4] 2> 448352 T2583 oascc.ZkStateReader$3.process Updating live
> nodes... (19)
> [junit4] 2> 448352 T2585 oascc.ZkStateReader$3.process Updating live
> nodes... (19)
> [junit4] 2> 448356 T2611 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@1b5c498
> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 448359 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 448362 T2522
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
> [junit4] 2> 448366 T2522 oascc.SolrZkClient.makePath makePath:
> /live_nodes/node19
> [junit4] 2> 448371 T2587 oascc.ZkStateReader$3.process Updating live
> nodes... (20)
> [junit4] 2> 448371 T2591 oascc.ZkStateReader$3.process Updating live
> nodes... (20)
> [junit4] 2> 448372 T2579 oascc.ZkStateReader$3.process Updating live
> nodes... (20)
> [junit4] 2> 448372 T2575 oascc.ZkStateReader$3.process Updating live
> nodes... (20)
> [junit4] 2> 448372 T2577 oascc.ZkStateReader$3.process Updating live
> nodes... (20)
> [junit4] 2> 448373 T2581 oascc.ZkStateReader$3.process Updating live
> nodes... (20)
> [junit4] 2> 448372 T2589 oascc.ZkStateReader$3.process Updating live
> nodes... (20)
> [junit4] 2> 448372 T2573 oascc.ZkStateReader$3.process Updating live
> nodes... (20)
> [junit4] 2> 448372 T2595 oascc.ZkStateReader$3.process Updating live
> nodes... (20)
> [junit4] 2> 448374 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 448373 T2609 oascc.ZkStateReader$3.process Updating live
> nodes... (20)
> [junit4] 2> 448373 T2611 oascc.ZkStateReader$3.process Updating live
> nodes... (20)
> [junit4] 2> 448373 T2603 oascc.ZkStateReader$3.process Updating live
> nodes... (20)
> [junit4] 2> 448373 T2585 oascc.ZkStateReader$3.process Updating live
> nodes... (20)
> [junit4] 2> 448373 T2593 oascc.ZkStateReader$3.process Updating live
> nodes... (20)
> [junit4] 2> 448373 T2567 oascc.ZkStateReader$3.process Updating live
> nodes... (20)
> [junit4] 2> 448373 T2607 oascc.ZkStateReader$3.process Updating live
> nodes... (20)
> [junit4] 2> 448372 T2599 oascc.ZkStateReader$3.process Updating live
> nodes... (20)
> [junit4] 2> 448372 T2597 oascc.ZkStateReader$3.process Updating live
> nodes... (20)
> [junit4] 2> 448372 T2601 oascc.ZkStateReader$3.process Updating live
> nodes... (20)
> [junit4] 2> 448372 T2583 oascc.ZkStateReader$3.process Updating live
> nodes... (20)
> [junit4] 2> 448372 T2605 oascc.ZkStateReader$3.process Updating live
> nodes... (20)
> [junit4] 2> 448376 T2613 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@18ed79a
> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 448380 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 448382 T2522
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
> [junit4] 2> 448385 T2522 oascc.SolrZkClient.makePath makePath:
> /live_nodes/node20
> [junit4] 2> 448390 T2577 oascc.ZkStateReader$3.process Updating live
> nodes... (21)
> [junit4] 2> 448390 T2579 oascc.ZkStateReader$3.process Updating live
> nodes... (21)
> [junit4] 2> 448391 T2589 oascc.ZkStateReader$3.process Updating live
> nodes... (21)
> [junit4] 2> 448391 T2601 oascc.ZkStateReader$3.process Updating live
> nodes... (21)
> [junit4] 2> 448391 T2599 oascc.ZkStateReader$3.process Updating live
> nodes... (21)
> [junit4] 2> 448391 T2573 oascc.ZkStateReader$3.process Updating live
> nodes... (21)
> [junit4] 2> 448392 T2581 oascc.ZkStateReader$3.process Updating live
> nodes... (21)
> [junit4] 2> 448391 T2605 oascc.ZkStateReader$3.process Updating live
> nodes... (21)
> [junit4] 2> 448390 T2591 oascc.ZkStateReader$3.process Updating live
> nodes... (21)
> [junit4] 2> 448390 T2595 oascc.ZkStateReader$3.process Updating live
> nodes... (21)
> [junit4] 2> 448393 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 448392 T2603 oascc.ZkStateReader$3.process Updating live
> nodes... (21)
> [junit4] 2> 448392 T2609 oascc.ZkStateReader$3.process Updating live
> nodes... (21)
> [junit4] 2> 448392 T2611 oascc.ZkStateReader$3.process Updating live
> nodes... (21)
> [junit4] 2> 448392 T2593 oascc.ZkStateReader$3.process Updating live
> nodes... (21)
> [junit4] 2> 448392 T2585 oascc.ZkStateReader$3.process Updating live
> nodes... (21)
> [junit4] 2> 448392 T2567 oascc.ZkStateReader$3.process Updating live
> nodes... (21)
> [junit4] 2> 448392 T2575 oascc.ZkStateReader$3.process Updating live
> nodes... (21)
> [junit4] 2> 448391 T2613 oascc.ZkStateReader$3.process Updating live
> nodes... (21)
> [junit4] 2> 448391 T2583 oascc.ZkStateReader$3.process Updating live
> nodes... (21)
> [junit4] 2> 448391 T2607 oascc.ZkStateReader$3.process Updating live
> nodes... (21)
> [junit4] 2> 448391 T2597 oascc.ZkStateReader$3.process Updating live
> nodes... (21)
> [junit4] 2> 448391 T2587 oascc.ZkStateReader$3.process Updating live
> nodes... (21)
> [junit4] 2> 448395 T2615 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@5f6bba
> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 448399 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 448401 T2522
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
> [junit4] 2> 448404 T2522 oascc.SolrZkClient.makePath makePath:
> /live_nodes/node21
> [junit4] 2> 448409 T2587 oascc.ZkStateReader$3.process Updating live
> nodes... (22)
> [junit4] 2> 448410 T2573 oascc.ZkStateReader$3.process Updating live
> nodes... (22)
> [junit4] 2> 448410 T2595 oascc.ZkStateReader$3.process Updating live
> nodes... (22)
> [junit4] 2> 448410 T2577 oascc.ZkStateReader$3.process Updating live
> nodes... (22)
> [junit4] 2> 448409 T2591 oascc.ZkStateReader$3.process Updating live
> nodes... (22)
> [junit4] 2> 448412 T2607 oascc.ZkStateReader$3.process Updating live
> nodes... (22)
> [junit4] 2> 448412 T2593 oascc.ZkStateReader$3.process Updating live
> nodes... (22)
> [junit4] 2> 448412 T2599 oascc.ZkStateReader$3.process Updating live
> nodes... (22)
> [junit4] 2> 448413 T2603 oascc.ZkStateReader$3.process Updating live
> nodes... (22)
> [junit4] 2> 448413 T2609 oascc.ZkStateReader$3.process Updating live
> nodes... (22)
> [junit4] 2> 448412 T2567 oascc.ZkStateReader$3.process Updating live
> nodes... (22)
> [junit4] 2> 448412 T2613 oascc.ZkStateReader$3.process Updating live
> nodes... (22)
> [junit4] 2> 448412 T2575 oascc.ZkStateReader$3.process Updating live
> nodes... (22)
> [junit4] 2> 448411 T2583 oascc.ZkStateReader$3.process Updating live
> nodes... (22)
> [junit4] 2> 448411 T2601 oascc.ZkStateReader$3.process Updating live
> nodes... (22)
> [junit4] 2> 448411 T2597 oascc.ZkStateReader$3.process Updating live
> nodes... (22)
> [junit4] 2> 448411 T2579 oascc.ZkStateReader$3.process Updating live
> nodes... (22)
> [junit4] 2> 448411 T2605 oascc.ZkStateReader$3.process Updating live
> nodes... (22)
> [junit4] 2> 448411 T2589 oascc.ZkStateReader$3.process Updating live
> nodes... (22)
> [junit4] 2> 448416 T2617 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@13fabc5
> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 448414 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 448413 T2615 oascc.ZkStateReader$3.process Updating live
> nodes... (22)
> [junit4] 2> 448413 T2611 oascc.ZkStateReader$3.process Updating live
> nodes... (22)
> [junit4] 2> 448412 T2581 oascc.ZkStateReader$3.process Updating live
> nodes... (22)
> [junit4] 2> 448412 T2585 oascc.ZkStateReader$3.process Updating live
> nodes... (22)
> [junit4] 2> 448418 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 448421 T2522
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
> [junit4] 2> 448426 T2522 oascc.SolrZkClient.makePath makePath:
> /live_nodes/node22
> [junit4] 2> 448431 T2587 oascc.ZkStateReader$3.process Updating live
> nodes... (23)
> [junit4] 2> 448432 T2577 oascc.ZkStateReader$3.process Updating live
> nodes... (23)
> [junit4] 2> 448432 T2579 oascc.ZkStateReader$3.process Updating live
> nodes... (23)
> [junit4] 2> 448432 T2605 oascc.ZkStateReader$3.process Updating live
> nodes... (23)
> [junit4] 2> 448433 T2611 oascc.ZkStateReader$3.process Updating live
> nodes... (23)
> [junit4] 2> 448432 T2595 oascc.ZkStateReader$3.process Updating live
> nodes... (23)
> [junit4] 2> 448432 T2591 oascc.ZkStateReader$3.process Updating live
> nodes... (23)
> [junit4] 2> 448434 T2609 oascc.ZkStateReader$3.process Updating live
> nodes... (23)
> [junit4] 2> 448432 T2573 oascc.ZkStateReader$3.process Updating live
> nodes... (23)
> [junit4] 2> 448435 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 448434 T2603 oascc.ZkStateReader$3.process Updating live
> nodes... (23)
> [junit4] 2> 448434 T2617 oascc.ZkStateReader$3.process Updating live
> nodes... (23)
> [junit4] 2> 448434 T2581 oascc.ZkStateReader$3.process Updating live
> nodes... (23)
> [junit4] 2> 448437 T2619 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@71217e
> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
> WatchedEvent state:SyncConnected type:None path:null
>
> [...truncated too long message...]
>
> queue/qn-
> [junit4] 2> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
> [junit4] 2> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> [junit4] 2> at
> org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
> [junit4] 2> at
> org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:243)
> [junit4] 2> at
> org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:240)
> [junit4] 2> at
> org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
> [junit4] 2> at
> org.apache.solr.common.cloud.SolrZkClient.create(SolrZkClient.java:240)
> [junit4] 2> at
> org.apache.solr.cloud.DistributedQueue.createData(DistributedQueue.java:346)
> [junit4] 2> at
> org.apache.solr.cloud.DistributedQueue.offer(DistributedQueue.java:330)
> [junit4] 2> at
> org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:198)
> [junit4] 2> at
> org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:163)
> [junit4] 2> at
> org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:125)
> [junit4] 2> at
> org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:314)
> [junit4] 2> at
> org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:221)
> [junit4] 2> at
> org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:452)
> [junit4] 2> at
> org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:216)
> [junit4] 2> at
> org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:163)
> [junit4] 2> at
> org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:125)
> [junit4] 2> at
> org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
> [junit4] 2> at
> org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:358)
> [junit4] 2> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
> [junit4] 2> at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
> [junit4] 2>
> [junit4] 2> 750036 T3044 oascc.ZkStateReader$3.process WARN ZooKeeper
> watch triggered, but Solr cannot talk to ZK
> [junit4] 2> 750037 T3044 oasc.LeaderElector$ElectionWatcher.process
> WARN org.apache.zookeeper.KeeperException$SessionExpiredException:
> KeeperErrorCode = Session expired for /overseer_elect/election
> [junit4] 2> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
> [junit4] 2> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> [junit4] 2> at
> org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
> [junit4] 2> at
> org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
> [junit4] 2> at
> org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
> [junit4] 2> at
> org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
> [junit4] 2> at
> org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
> [junit4] 2> at
> org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
> [junit4] 2> at
> org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
> [junit4] 2> at
> org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:358)
> [junit4] 2> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
> [junit4] 2> at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
> [junit4] 2>
> [junit4] 2> 750038 T3044 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
> [junit4] 2> 750038 T3044 oascc.ZkStateReader$2.process WARN ZooKeeper
> watch triggered, but Solr cannot talk to ZK
> [junit4] 2> 750121 T3116 oascc.ZkStateReader$3.process WARN ZooKeeper
> watch triggered, but Solr cannot talk to ZK
> [junit4] 2> 750206 T2787 oascc.ZkStateReader$3.process WARN ZooKeeper
> watch triggered, but Solr cannot talk to ZK
> [junit4] 2> 750206 T2787
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
> [junit4] 2> 750207 T2787
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
> [junit4] 2> 750207 T2787
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
> [junit4] 2> 750207 T2787
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
> [junit4] 2> 750206 T3156
> oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not
> read the data org.apache.zookeeper.KeeperException$SessionExpiredException:
> KeeperErrorCode = Session expired for /overseer_elect/leader
> [junit4] 2> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
> [junit4] 2> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> [junit4] 2> at
> org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
> [junit4] 2> at
> org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
> [junit4] 2> at
> org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
> [junit4] 2> at
> org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
> [junit4] 2> at
> org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
> [junit4] 2> at
> org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:316)
> [junit4] 2> at
> org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
> [junit4] 2> at
> org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:288)
> [junit4] 2>
> [junit4] 2> NOTE: leaving temporary files on disk at:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-A819205877C572F6-001
> [junit4] 2> NOTE: test params are: codec=Lucene41,
> sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=ar_JO,
> timezone=Africa/Ndjamena
> [junit4] 2> NOTE: Linux 3.8.0-42-generic i386/Oracle Corporation
> 1.8.0_05 (32-bit)/cpus=8,threads=1,free=118175464,total=317980672
> [junit4] 2> NOTE: All tests run in this JVM: [TestRecovery,
> OpenCloseCoreStressTest, HardAutoCommitTest, DeleteReplicaTest,
> IndexSchemaRuntimeFieldTest, TestXIncludeConfig, TestFreeTextSuggestions,
> CollectionsAPIDistributedZkTest, TestFieldTypeResource, BadCopyFieldTest,
> CopyFieldTest, TestCSVLoader, UniqFieldsUpdateProcessorFactoryTest,
> TestIndexingPerformance, TestDocumentBuilder, AutoCommitTest,
> AtomicUpdatesTest, TestQueryTypes, TestCollapseQParserPlugin,
> BasicDistributedZk2Test, TestWriterPerf, TestCoreContainer,
> TestComplexPhraseQParserPlugin, TestDistribDocBasedVersion,
> TestFastLRUCache, TestDynamicFieldResource, DOMUtilTest,
> QueryResultKeyTest, DistribDocExpirationUpdateProcessorTest,
> UpdateRequestProcessorFactoryTest, TimeZoneUtilsTest,
> SchemaVersionSpecificBehaviorTest, PreAnalyzedUpdateProcessorTest,
> SOLR749Test, TestNonDefinedSimilarityFactory, TestFileDictionaryLookup,
> MultiThreadedOCPTest, TestRTGBase, QueryParsingTest, OutputWriterTest,
> LoggingHandlerTest, TestInfoStreamLogging, TestPseudoReturnFields,
> TestFastWriter, TestReplicationHandlerBackup, AssignTest,
> TermVectorComponentTest, SolrRequestParserTest, TestPerFieldSimilarity,
> TestAtomicUpdateErrorCases, TestManagedResource,
> TermVectorComponentDistributedTest, TestReRankQParserPlugin,
> TestSerializedLuceneMatchVersion, TestChildDocTransformer,
> TestNoOpRegenerator, TestGroupingSearch, TestFieldTypeCollectionResource,
> TestBinaryField, StatsComponentTest, CursorMarkTest, BasicZkTest,
> SuggesterFSTTest, BJQParserTest, CurrencyFieldXmlFileTest,
> DistributedQueryElevationComponentTest, TestDistributedMissingSort,
> SearchHandlerTest, TestLFUCache, TestSolrJ, FileBasedSpellCheckerTest,
> TestRealTimeGet, RequestHandlersTest, AnalysisAfterCoreReloadTest,
> SolrInfoMBeanTest, TestDistributedGrouping, TestExtendedDismaxParser,
> URLClassifyProcessorTest, SuggestComponentTest, ChaosMonkeySafeLeaderTest]
> [junit4] Completed on J1 in 74.71s, 1 test, 1 failure <<< FAILURES!
>
> [...truncated 59 lines...]
> BUILD FAILED
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:467: The
> following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:447: The
> following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:45: The
> following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The
> following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The
> following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:496:
> The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1299:
> The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:923:
> There were test failures: 405 suites, 1656 tests, 3 suite-level errors, 1
> error, 1 failure, 954 ignored (13 assumptions)
>
> Total time: 65 minutes 33 seconds
> Build step 'Invoke Ant' marked build as failure
> [description-setter] Description set: Java: 32bit/jdk1.8.0_05 -client
> -XX:+UseParallelGC
> Archiving artifacts
> Recording test results
> Email was triggered for: Failure - Any
> Sending email for trigger: Failure - Any
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
--
Regards,
Shalin Shekhar Mangar.
[JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.8.0_05) - Build # 10616
- Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/10616/
Java: 32bit/jdk1.8.0_05 -client -XX:+UseParallelGC
5 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.OverseerTest
Error Message:
ERROR: SolrZkClient opens=165 closes=164
Stack Trace:
java.lang.AssertionError: ERROR: SolrZkClient opens=165 closes=164
at __randomizedtesting.SeedInfo.seed([A819205877C572F6]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.SolrTestCaseJ4.endTrackingZkClients(SolrTestCaseJ4.java:436)
at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:182)
at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:790)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at java.lang.Thread.run(Thread.java:745)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.OverseerTest
Error Message:
2 threads leaked from SUITE scope at org.apache.solr.cloud.OverseerTest: 1) Thread[id=2969, name=TEST-OverseerTest.testShardLeaderChange-seed#[A819205877C572F6]-SendThread(127.0.0.1:33880), state=TIMED_WAITING, group=TGRP-OverseerTest] at java.lang.Thread.sleep(Native Method) at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003) 2) Thread[id=2970, name=TEST-OverseerTest.testShardLeaderChange-seed#[A819205877C572F6]-EventThread, state=WAITING, group=TGRP-OverseerTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.OverseerTest:
1) Thread[id=2969, name=TEST-OverseerTest.testShardLeaderChange-seed#[A819205877C572F6]-SendThread(127.0.0.1:33880), state=TIMED_WAITING, group=TGRP-OverseerTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
2) Thread[id=2970, name=TEST-OverseerTest.testShardLeaderChange-seed#[A819205877C572F6]-EventThread, state=WAITING, group=TGRP-OverseerTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
at __randomizedtesting.SeedInfo.seed([A819205877C572F6]:0)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.OverseerTest
Error Message:
There are still zombie threads that couldn't be terminated: 1) Thread[id=2969, name=TEST-OverseerTest.testShardLeaderChange-seed#[A819205877C572F6]-SendThread(127.0.0.1:33880), state=TIMED_WAITING, group=TGRP-OverseerTest] at java.lang.Thread.sleep(Native Method) at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
1) Thread[id=2969, name=TEST-OverseerTest.testShardLeaderChange-seed#[A819205877C572F6]-SendThread(127.0.0.1:33880), state=TIMED_WAITING, group=TGRP-OverseerTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
at __randomizedtesting.SeedInfo.seed([A819205877C572F6]:0)
REGRESSION: org.apache.solr.cloud.OverseerTest.testShardLeaderChange
Error Message:
KeeperErrorCode = NodeExists for /live_nodes/node1
Stack Trace:
org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /live_nodes/node1
at __randomizedtesting.SeedInfo.seed([A819205877C572F6:764AA7AF6D5D8707]:0)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
at org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:432)
at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:429)
at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:386)
at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:373)
at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:350)
at org.apache.solr.cloud.OverseerTest$MockZKController.<init>(OverseerTest.java:85)
at org.apache.solr.cloud.OverseerTest.testShardLeaderChange(OverseerTest.java:746)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at java.lang.Thread.run(Thread.java:745)
FAILED: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch
Error Message:
shard2 is not consistent. Got 153 from http://127.0.0.1:51995/collection1lastClient and got 132 from http://127.0.0.1:36739/collection1
Stack Trace:
java.lang.AssertionError: shard2 is not consistent. Got 153 from http://127.0.0.1:51995/collection1lastClient and got 132 from http://127.0.0.1:36739/collection1
at __randomizedtesting.SeedInfo.seed([A819205877C572F6:29FFAE40009A12CA]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1139)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1118)
at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:150)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at java.lang.Thread.run(Thread.java:745)
Build Log:
[...truncated 11133 lines...]
[junit4] Suite: org.apache.solr.cloud.OverseerTest
[junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.OverseerTest-A819205877C572F6-001/init-core-data-001
[junit4] 2> 443102 T2522 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
[junit4] 2> 443103 T2522 oas.SolrTestCaseJ4.initCore ####initCore
[junit4] 2> 443103 T2522 oas.SolrTestCaseJ4.initCore ####initCore end
[junit4] 2> 443107 T2522 oas.SolrTestCaseJ4.setUp ###Starting testStateChange
[junit4] 2> 443107 T2522 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 443108 T2523 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 443208 T2522 oasc.ZkTestServer.run start zk server on port:45124
[junit4] 2> 443209 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 443282 T2529 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ed72a5 name:ZooKeeperConnection Watcher:127.0.0.1:45124/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 443283 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 443284 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 443287 T2531 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6366c4 name:ZooKeeperConnection Watcher:127.0.0.1:45124 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 443287 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 443290 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 443292 T2533 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@60240b name:ZooKeeperConnection Watcher:127.0.0.1:45124 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 443292 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 443293 T2522 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 443298 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 443301 T2522 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 443304 T2522 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 443306 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 443311 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 443312 T2535 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@68e786 name:ZooKeeperConnection Watcher:127.0.0.1:45124/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 443313 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 443314 T2522 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 443318 T2522 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 443324 T2522 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:45124_solr
[junit4] 2> 443324 T2522 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 443328 T2522 oasc.Overseer.start Overseer (id=92053395863699459-127.0.0.1:45124_solr-n_0000000000) starting
[junit4] 2> 443331 T2522 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 443336 T2522 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 443340 T2522 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 443345 T2522 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 443350 T2522 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 443357 T2522 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 443365 T2537 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 443366 T2536 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 443368 T2522 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 443369 T2536 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 443370 T2536 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1/solr",
[junit4] 2> "node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "core":"core1",
[junit4] 2> "roles":"",
[junit4] 2> "state":"recovering"}
[junit4] 2> 443370 T2536 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 443374 T2535 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 443375 T2529 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 443420 T2522 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 443422 T2535 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 443424 T2536 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 443425 T2536 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1/solr",
[junit4] 2> "node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "core":"core1",
[junit4] 2> "roles":"",
[junit4] 2> "state":"active"}
[junit4] 2> 443426 T2536 oasc.Overseer$ClusterStateUpdater.updateState node=core_node1 is already registered
[junit4] 2> 443426 T2536 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
[junit4] 2> 443431 T2535 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 443534 T2529 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 443575 T2522 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:45124 45124
[junit4] 2> 443988 T2523 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:45124 45124
[junit4] 2> 443991 T2522 oas.SolrTestCaseJ4.tearDown ###Ending testStateChange
[junit4] 2> 443992 T2536 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:45124_solr
[junit4] 2> 443994 T2538 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:316)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:288)
[junit4] 2>
[junit4] 2> 444000 T2522 oas.SolrTestCaseJ4.setUp ###Starting testShardAssignment
[junit4] 2> 444001 T2522 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 444002 T2539 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 444101 T2522 oasc.ZkTestServer.run start zk server on port:57742
[junit4] 2> 444103 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 444107 T2545 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@9fc604 name:ZooKeeperConnection Watcher:127.0.0.1:57742 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 444108 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 444111 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 444112 T2547 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@104f2e0 name:ZooKeeperConnection Watcher:127.0.0.1:57742 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 444113 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 444114 T2522 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 444119 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 444121 T2549 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@896251 name:ZooKeeperConnection Watcher:127.0.0.1:57742/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 444121 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 444122 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 444125 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 444127 T2551 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@10c3e42 name:ZooKeeperConnection Watcher:127.0.0.1:57742/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 444127 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 444128 T2522 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 444132 T2522 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 444137 T2522 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:57742_solr
[junit4] 2> 444138 T2522 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 444141 T2522 oasc.Overseer.start Overseer (id=92053395922223107-127.0.0.1:57742_solr-n_0000000000) starting
[junit4] 2> 444144 T2522 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 444149 T2522 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 444153 T2522 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 444157 T2522 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 444162 T2522 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 444166 T2522 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 444173 T2553 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 444175 T2522 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 444175 T2552 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 444177 T2522 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 444179 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 444183 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 444185 T2555 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1196b81 name:ZooKeeperConnection Watcher:127.0.0.1:57742/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 444185 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 444187 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 444190 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1
[junit4] 2> 444194 T2549 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4] 2> 444194 T2555 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4] 2> 444197 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 444199 T2552 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 444199 T2552 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 444200 T2552 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
[junit4] 2> 444200 T2552 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 444204 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 444205 T2555 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 444205 T2549 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 444697 T2522 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 444712 T2522 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4] 2> 444719 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 444721 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 444721 T2552 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 444727 T2552 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core2",
[junit4] 2> "core_node_name":"node2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 444728 T2552 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 444728 T2552 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
[junit4] 2> 444731 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 444834 T2555 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 444834 T2549 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 445222 T2522 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4] 2> 445233 T2522 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
[junit4] 2> 445240 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 445240 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 445241 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 445242 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 445242 T2552 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 445248 T2552 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core3",
[junit4] 2> "core_node_name":"node3",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 445248 T2552 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 445248 T2552 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 445251 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 445354 T2549 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 445354 T2555 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 445742 T2522 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 445753 T2522 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 445764 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 445765 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 445765 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 445766 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 445767 T2552 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 445771 T2552 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core4",
[junit4] 2> "core_node_name":"node4",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 445771 T2552 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 445771 T2552 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 445773 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 445876 T2555 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 445876 T2549 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 446267 T2522 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 446276 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 446276 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 446276 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 446278 T2552 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 446279 T2552 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core5",
[junit4] 2> "core_node_name":"node5",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 446280 T2552 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 446280 T2552 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
[junit4] 2> 446283 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 446386 T2555 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 446386 T2549 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 446776 T2522 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4] 2> 446787 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 446788 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 446788 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 446790 T2552 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 446791 T2552 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core6",
[junit4] 2> "core_node_name":"node6",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 446791 T2552 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 446791 T2552 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 446795 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 446899 T2549 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 446899 T2555 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 447288 T2522 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 447298 T2555 oascc.ZkStateReader$3.process Updating live nodes... (0)
[junit4] 2> 447301 T2522 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:57742 57742
[junit4] 2> 447869 T2539 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:57742 57742
[junit4] 2> 447871 T2522 oas.SolrTestCaseJ4.tearDown ###Ending testShardAssignment
[junit4] 2> 447872 T2552 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:57742_solr
[junit4] 2> 447875 T2556 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:316)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:288)
[junit4] 2>
[junit4] 2> 447881 T2522 oas.SolrTestCaseJ4.setUp ###Starting testShardAssignmentBigger
[junit4] 2> 447882 T2522 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 447883 T2557 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 447983 T2522 oasc.ZkTestServer.run start zk server on port:50146
[junit4] 2> 447984 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 447988 T2563 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@daa195 name:ZooKeeperConnection Watcher:127.0.0.1:50146 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 447989 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 447993 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 447994 T2565 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f8aac name:ZooKeeperConnection Watcher:127.0.0.1:50146 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 447995 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 447995 T2522 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 448000 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 448002 T2567 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@141291c name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 448002 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 448003 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 448006 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 448008 T2569 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@49f99d name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 448008 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 448009 T2522 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 448012 T2522 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 448018 T2522 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:50146_solr
[junit4] 2> 448018 T2522 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 448021 T2522 oasc.Overseer.start Overseer (id=92053396176633859-127.0.0.1:50146_solr-n_0000000000) starting
[junit4] 2> 448024 T2522 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 448029 T2522 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 448034 T2522 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 448038 T2522 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 448043 T2522 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 448048 T2522 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 448055 T2571 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 448056 T2522 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 448056 T2570 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 448058 T2522 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 448061 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 448065 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 448066 T2573 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1242e08 name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 448067 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 448069 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 448072 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node0
[junit4] 2> 448076 T2573 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4] 2> 448076 T2567 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4] 2> 448077 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 448078 T2575 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1fc812a name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 448078 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 448080 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 448083 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
[junit4] 2> 448087 T2573 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 448087 T2567 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 448087 T2575 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 448088 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 448089 T2577 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1714ec0 name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 448090 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 448091 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 448095 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node2
[junit4] 2> 448099 T2573 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 448099 T2577 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 448099 T2575 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 448100 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 448099 T2567 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 448102 T2579 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@fbc5e2 name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 448102 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 448104 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 448107 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node3
[junit4] 2> 448112 T2577 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 448112 T2575 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 448112 T2567 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 448112 T2573 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 448113 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 448113 T2579 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 448115 T2581 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@75e1db name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 448115 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 448117 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 448121 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node4
[junit4] 2> 448126 T2577 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4] 2> 448126 T2575 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4] 2> 448126 T2573 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4] 2> 448127 T2581 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4] 2> 448126 T2567 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4] 2> 448126 T2579 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4] 2> 448127 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 448129 T2583 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@11dc02e name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 448130 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 448132 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 448136 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node5
[junit4] 2> 448139 T2575 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4] 2> 448140 T2577 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4] 2> 448140 T2581 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4] 2> 448140 T2573 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4] 2> 448140 T2579 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4] 2> 448141 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 448141 T2583 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4] 2> 448140 T2567 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4] 2> 448143 T2585 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6d9f03 name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 448143 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 448145 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 448148 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node6
[junit4] 2> 448153 T2575 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4] 2> 448153 T2577 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4] 2> 448153 T2573 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4] 2> 448153 T2585 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4] 2> 448153 T2583 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4] 2> 448153 T2579 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4] 2> 448153 T2581 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4] 2> 448153 T2567 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4] 2> 448154 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 448156 T2587 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@958679 name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 448157 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 448159 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 448162 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node7
[junit4] 2> 448167 T2567 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4] 2> 448167 T2577 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4] 2> 448168 T2585 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4] 2> 448168 T2583 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4] 2> 448168 T2581 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4] 2> 448168 T2579 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4] 2> 448168 T2573 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4] 2> 448167 T2587 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4] 2> 448167 T2575 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4] 2> 448169 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 448171 T2589 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@169d949 name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 448171 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 448174 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 448177 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node8
[junit4] 2> 448182 T2589 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4] 2> 448183 T2585 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4] 2> 448183 T2583 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4] 2> 448183 T2587 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4] 2> 448183 T2573 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4] 2> 448183 T2579 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4] 2> 448183 T2581 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4] 2> 448183 T2575 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4] 2> 448183 T2577 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4] 2> 448182 T2567 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4] 2> 448185 T2591 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2e78fa name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 448184 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 448188 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 448190 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 448192 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node9
[junit4] 2> 448195 T2589 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 448195 T2577 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 448195 T2567 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 448196 T2573 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 448195 T2575 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 448195 T2587 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 448195 T2579 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 448195 T2583 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 448196 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 448198 T2593 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@156c3b name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 448196 T2591 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 448196 T2585 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 448196 T2581 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 448198 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 448200 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 448203 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node10
[junit4] 2> 448207 T2575 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 448207 T2583 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 448208 T2579 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 448207 T2577 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 448207 T2573 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 448207 T2589 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 448207 T2587 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 448209 T2585 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 448208 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 448208 T2581 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 448208 T2593 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 448208 T2591 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 448208 T2567 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 448211 T2595 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f1978e name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 448212 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 448213 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 448216 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node11
[junit4] 2> 448220 T2587 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 448220 T2567 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 448221 T2583 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 448220 T2589 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 448222 T2595 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 448220 T2575 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 448222 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 448222 T2593 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 448222 T2581 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 448224 T2597 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@175be88 name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 448221 T2591 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 448221 T2573 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 448221 T2577 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 448221 T2585 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 448221 T2579 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 448224 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 448228 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 448232 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node12
[junit4] 2> 448236 T2573 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 448236 T2577 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 448237 T2595 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 448237 T2593 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 448237 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 448238 T2581 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 448238 T2597 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 448237 T2583 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 448239 T2599 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@dde0f4 name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 448237 T2589 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 448237 T2575 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 448237 T2587 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 448237 T2585 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 448236 T2567 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 448236 T2579 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 448236 T2591 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 448240 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 448244 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 448248 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node13
[junit4] 2> 448254 T2587 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 448255 T2577 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 448255 T2579 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 448255 T2595 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 448255 T2573 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 448256 T2597 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 448256 T2599 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 448256 T2593 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 448256 T2585 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 448256 T2581 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 448256 T2583 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 448256 T2567 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 448256 T2575 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 448256 T2589 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 448256 T2591 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 448259 T2601 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@19e3760 name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 448257 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 448262 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 448264 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 448268 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node14
[junit4] 2> 448273 T2573 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 448273 T2587 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 448273 T2579 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 448273 T2591 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 448273 T2577 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 448273 T2595 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 448275 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 448275 T2599 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 448275 T2583 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 448275 T2585 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 448275 T2597 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 448275 T2601 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 448275 T2567 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 448275 T2575 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 448275 T2581 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 448274 T2593 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 448273 T2589 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 448277 T2603 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1818cfa name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 448281 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 448283 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 448286 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node15
[junit4] 2> 448291 T2577 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 448292 T2573 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 448291 T2587 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 448293 T2579 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 448293 T2575 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 448293 T2601 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 448294 T2599 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 448293 T2591 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 448293 T2597 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 448293 T2583 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 448293 T2567 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 448293 T2589 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 448295 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 448294 T2603 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 448294 T2581 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 448294 T2585 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 448294 T2595 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 448294 T2593 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 448297 T2605 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@154e83c name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 448299 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 448300 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 448304 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node16
[junit4] 2> 448313 T2587 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 448313 T2589 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 448314 T2595 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 448315 T2575 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 448314 T2585 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 448316 T2591 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 448313 T2573 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 448313 T2577 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 448313 T2593 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 448317 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 448316 T2599 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 448316 T2581 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 448316 T2605 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 448316 T2603 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 448316 T2583 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 448316 T2567 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 448315 T2601 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 448315 T2597 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 448315 T2579 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 448319 T2607 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d963a2 name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 448323 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 448324 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 448328 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node17
[junit4] 2> 448333 T2573 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 448333 T2577 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 448334 T2601 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 448334 T2607 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 448335 T2567 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 448335 T2583 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 448335 T2603 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 448334 T2589 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 448336 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 448334 T2595 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 448334 T2579 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 448334 T2575 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 448338 T2609 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@826a2e name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 448333 T2587 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 448333 T2591 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 448338 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 448336 T2585 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 448335 T2593 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 448335 T2599 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 448335 T2581 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 448335 T2605 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 448335 T2597 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 448341 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 448346 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node18
[junit4] 2> 448350 T2573 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 448350 T2577 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 448351 T2607 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 448352 T2587 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 448352 T2593 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 448351 T2601 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 448353 T2579 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 448353 T2581 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 448351 T2595 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 448353 T2605 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 448354 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 448353 T2609 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 448353 T2603 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 448353 T2597 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 448353 T2589 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 448353 T2567 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 448353 T2591 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 448353 T2575 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 448352 T2599 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 448352 T2583 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 448352 T2585 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 448356 T2611 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b5c498 name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 448359 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 448362 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 448366 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node19
[junit4] 2> 448371 T2587 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 448371 T2591 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 448372 T2579 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 448372 T2575 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 448372 T2577 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 448373 T2581 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 448372 T2589 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 448372 T2573 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 448372 T2595 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 448374 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 448373 T2609 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 448373 T2611 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 448373 T2603 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 448373 T2585 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 448373 T2593 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 448373 T2567 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 448373 T2607 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 448372 T2599 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 448372 T2597 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 448372 T2601 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 448372 T2583 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 448372 T2605 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 448376 T2613 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@18ed79a name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 448380 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 448382 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 448385 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node20
[junit4] 2> 448390 T2577 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 448390 T2579 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 448391 T2589 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 448391 T2601 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 448391 T2599 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 448391 T2573 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 448392 T2581 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 448391 T2605 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 448390 T2591 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 448390 T2595 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 448393 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 448392 T2603 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 448392 T2609 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 448392 T2611 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 448392 T2593 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 448392 T2585 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 448392 T2567 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 448392 T2575 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 448391 T2613 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 448391 T2583 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 448391 T2607 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 448391 T2597 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 448391 T2587 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 448395 T2615 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5f6bba name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 448399 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 448401 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 448404 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node21
[junit4] 2> 448409 T2587 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 448410 T2573 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 448410 T2595 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 448410 T2577 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 448409 T2591 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 448412 T2607 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 448412 T2593 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 448412 T2599 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 448413 T2603 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 448413 T2609 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 448412 T2567 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 448412 T2613 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 448412 T2575 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 448411 T2583 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 448411 T2601 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 448411 T2597 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 448411 T2579 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 448411 T2605 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 448411 T2589 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 448416 T2617 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@13fabc5 name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 448414 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 448413 T2615 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 448413 T2611 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 448412 T2581 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 448412 T2585 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 448418 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 448421 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 448426 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node22
[junit4] 2> 448431 T2587 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 448432 T2577 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 448432 T2579 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 448432 T2605 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 448433 T2611 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 448432 T2595 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 448432 T2591 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 448434 T2609 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 448432 T2573 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 448435 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 448434 T2603 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 448434 T2617 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 448434 T2581 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 448437 T2619 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@71217e name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null
[...truncated too long message...]
queue/qn-
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:243)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:240)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.create(SolrZkClient.java:240)
[junit4] 2> at org.apache.solr.cloud.DistributedQueue.createData(DistributedQueue.java:346)
[junit4] 2> at org.apache.solr.cloud.DistributedQueue.offer(DistributedQueue.java:330)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:198)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:163)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:125)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:314)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:221)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:452)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:216)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:163)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:125)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:358)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
[junit4] 2>
[junit4] 2> 750036 T3044 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 750037 T3044 oasc.LeaderElector$ElectionWatcher.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:358)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
[junit4] 2>
[junit4] 2> 750038 T3044 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4] 2> 750038 T3044 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 750121 T3116 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 750206 T2787 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 750206 T2787 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 750207 T2787 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 750207 T2787 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 750207 T2787 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 750206 T3156 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:316)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:288)
[junit4] 2>
[junit4] 2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-A819205877C572F6-001
[junit4] 2> NOTE: test params are: codec=Lucene41, sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=ar_JO, timezone=Africa/Ndjamena
[junit4] 2> NOTE: Linux 3.8.0-42-generic i386/Oracle Corporation 1.8.0_05 (32-bit)/cpus=8,threads=1,free=118175464,total=317980672
[junit4] 2> NOTE: All tests run in this JVM: [TestRecovery, OpenCloseCoreStressTest, HardAutoCommitTest, DeleteReplicaTest, IndexSchemaRuntimeFieldTest, TestXIncludeConfig, TestFreeTextSuggestions, CollectionsAPIDistributedZkTest, TestFieldTypeResource, BadCopyFieldTest, CopyFieldTest, TestCSVLoader, UniqFieldsUpdateProcessorFactoryTest, TestIndexingPerformance, TestDocumentBuilder, AutoCommitTest, AtomicUpdatesTest, TestQueryTypes, TestCollapseQParserPlugin, BasicDistributedZk2Test, TestWriterPerf, TestCoreContainer, TestComplexPhraseQParserPlugin, TestDistribDocBasedVersion, TestFastLRUCache, TestDynamicFieldResource, DOMUtilTest, QueryResultKeyTest, DistribDocExpirationUpdateProcessorTest, UpdateRequestProcessorFactoryTest, TimeZoneUtilsTest, SchemaVersionSpecificBehaviorTest, PreAnalyzedUpdateProcessorTest, SOLR749Test, TestNonDefinedSimilarityFactory, TestFileDictionaryLookup, MultiThreadedOCPTest, TestRTGBase, QueryParsingTest, OutputWriterTest, LoggingHandlerTest, TestInfoStreamLogging, TestPseudoReturnFields, TestFastWriter, TestReplicationHandlerBackup, AssignTest, TermVectorComponentTest, SolrRequestParserTest, TestPerFieldSimilarity, TestAtomicUpdateErrorCases, TestManagedResource, TermVectorComponentDistributedTest, TestReRankQParserPlugin, TestSerializedLuceneMatchVersion, TestChildDocTransformer, TestNoOpRegenerator, TestGroupingSearch, TestFieldTypeCollectionResource, TestBinaryField, StatsComponentTest, CursorMarkTest, BasicZkTest, SuggesterFSTTest, BJQParserTest, CurrencyFieldXmlFileTest, DistributedQueryElevationComponentTest, TestDistributedMissingSort, SearchHandlerTest, TestLFUCache, TestSolrJ, FileBasedSpellCheckerTest, TestRealTimeGet, RequestHandlersTest, AnalysisAfterCoreReloadTest, SolrInfoMBeanTest, TestDistributedGrouping, TestExtendedDismaxParser, URLClassifyProcessorTest, SuggestComponentTest, ChaosMonkeySafeLeaderTest]
[junit4] Completed on J1 in 74.71s, 1 test, 1 failure <<< FAILURES!
[...truncated 59 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1299: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:923: There were test failures: 405 suites, 1656 tests, 3 suite-level errors, 1 error, 1 failure, 954 ignored (13 assumptions)
Total time: 65 minutes 33 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 32bit/jdk1.8.0_05 -client -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any