You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/07/06 07:16:38 UTC
[JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.8.0_05) - Build # 10616
- Still Failing!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/10616/
Java: 32bit/jdk1.8.0_05 -client -XX:+UseParallelGC
5 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.OverseerTest
Error Message:
ERROR: SolrZkClient opens=165 closes=164
Stack Trace:
java.lang.AssertionError: ERROR: SolrZkClient opens=165 closes=164
at __randomizedtesting.SeedInfo.seed([A819205877C572F6]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.SolrTestCaseJ4.endTrackingZkClients(SolrTestCaseJ4.java:436)
at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:182)
at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:790)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at java.lang.Thread.run(Thread.java:745)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.OverseerTest
Error Message:
2 threads leaked from SUITE scope at org.apache.solr.cloud.OverseerTest: 1) Thread[id=2969, name=TEST-OverseerTest.testShardLeaderChange-seed#[A819205877C572F6]-SendThread(127.0.0.1:33880), state=TIMED_WAITING, group=TGRP-OverseerTest] at java.lang.Thread.sleep(Native Method) at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003) 2) Thread[id=2970, name=TEST-OverseerTest.testShardLeaderChange-seed#[A819205877C572F6]-EventThread, state=WAITING, group=TGRP-OverseerTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.OverseerTest:
1) Thread[id=2969, name=TEST-OverseerTest.testShardLeaderChange-seed#[A819205877C572F6]-SendThread(127.0.0.1:33880), state=TIMED_WAITING, group=TGRP-OverseerTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
2) Thread[id=2970, name=TEST-OverseerTest.testShardLeaderChange-seed#[A819205877C572F6]-EventThread, state=WAITING, group=TGRP-OverseerTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
at __randomizedtesting.SeedInfo.seed([A819205877C572F6]:0)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.OverseerTest
Error Message:
There are still zombie threads that couldn't be terminated: 1) Thread[id=2969, name=TEST-OverseerTest.testShardLeaderChange-seed#[A819205877C572F6]-SendThread(127.0.0.1:33880), state=TIMED_WAITING, group=TGRP-OverseerTest] at java.lang.Thread.sleep(Native Method) at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
1) Thread[id=2969, name=TEST-OverseerTest.testShardLeaderChange-seed#[A819205877C572F6]-SendThread(127.0.0.1:33880), state=TIMED_WAITING, group=TGRP-OverseerTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
at __randomizedtesting.SeedInfo.seed([A819205877C572F6]:0)
REGRESSION: org.apache.solr.cloud.OverseerTest.testShardLeaderChange
Error Message:
KeeperErrorCode = NodeExists for /live_nodes/node1
Stack Trace:
org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /live_nodes/node1
at __randomizedtesting.SeedInfo.seed([A819205877C572F6:764AA7AF6D5D8707]:0)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
at org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:432)
at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:429)
at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:386)
at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:373)
at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:350)
at org.apache.solr.cloud.OverseerTest$MockZKController.<init>(OverseerTest.java:85)
at org.apache.solr.cloud.OverseerTest.testShardLeaderChange(OverseerTest.java:746)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at java.lang.Thread.run(Thread.java:745)
FAILED: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch
Error Message:
shard2 is not consistent. Got 153 from http://127.0.0.1:51995/collection1lastClient and got 132 from http://127.0.0.1:36739/collection1
Stack Trace:
java.lang.AssertionError: shard2 is not consistent. Got 153 from http://127.0.0.1:51995/collection1lastClient and got 132 from http://127.0.0.1:36739/collection1
at __randomizedtesting.SeedInfo.seed([A819205877C572F6:29FFAE40009A12CA]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1139)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1118)
at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:150)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at java.lang.Thread.run(Thread.java:745)
Build Log:
[...truncated 11133 lines...]
[junit4] Suite: org.apache.solr.cloud.OverseerTest
[junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.OverseerTest-A819205877C572F6-001/init-core-data-001
[junit4] 2> 443102 T2522 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
[junit4] 2> 443103 T2522 oas.SolrTestCaseJ4.initCore ####initCore
[junit4] 2> 443103 T2522 oas.SolrTestCaseJ4.initCore ####initCore end
[junit4] 2> 443107 T2522 oas.SolrTestCaseJ4.setUp ###Starting testStateChange
[junit4] 2> 443107 T2522 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 443108 T2523 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 443208 T2522 oasc.ZkTestServer.run start zk server on port:45124
[junit4] 2> 443209 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 443282 T2529 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ed72a5 name:ZooKeeperConnection Watcher:127.0.0.1:45124/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 443283 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 443284 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 443287 T2531 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6366c4 name:ZooKeeperConnection Watcher:127.0.0.1:45124 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 443287 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 443290 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 443292 T2533 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@60240b name:ZooKeeperConnection Watcher:127.0.0.1:45124 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 443292 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 443293 T2522 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 443298 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 443301 T2522 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 443304 T2522 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 443306 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 443311 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 443312 T2535 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@68e786 name:ZooKeeperConnection Watcher:127.0.0.1:45124/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 443313 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 443314 T2522 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 443318 T2522 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 443324 T2522 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:45124_solr
[junit4] 2> 443324 T2522 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 443328 T2522 oasc.Overseer.start Overseer (id=92053395863699459-127.0.0.1:45124_solr-n_0000000000) starting
[junit4] 2> 443331 T2522 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 443336 T2522 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 443340 T2522 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 443345 T2522 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 443350 T2522 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 443357 T2522 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 443365 T2537 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 443366 T2536 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 443368 T2522 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 443369 T2536 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 443370 T2536 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1/solr",
[junit4] 2> "node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "core":"core1",
[junit4] 2> "roles":"",
[junit4] 2> "state":"recovering"}
[junit4] 2> 443370 T2536 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 443374 T2535 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 443375 T2529 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 443420 T2522 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 443422 T2535 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 443424 T2536 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 443425 T2536 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1/solr",
[junit4] 2> "node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "core":"core1",
[junit4] 2> "roles":"",
[junit4] 2> "state":"active"}
[junit4] 2> 443426 T2536 oasc.Overseer$ClusterStateUpdater.updateState node=core_node1 is already registered
[junit4] 2> 443426 T2536 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
[junit4] 2> 443431 T2535 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 443534 T2529 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 443575 T2522 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:45124 45124
[junit4] 2> 443988 T2523 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:45124 45124
[junit4] 2> 443991 T2522 oas.SolrTestCaseJ4.tearDown ###Ending testStateChange
[junit4] 2> 443992 T2536 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:45124_solr
[junit4] 2> 443994 T2538 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:316)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:288)
[junit4] 2>
[junit4] 2> 444000 T2522 oas.SolrTestCaseJ4.setUp ###Starting testShardAssignment
[junit4] 2> 444001 T2522 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 444002 T2539 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 444101 T2522 oasc.ZkTestServer.run start zk server on port:57742
[junit4] 2> 444103 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 444107 T2545 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@9fc604 name:ZooKeeperConnection Watcher:127.0.0.1:57742 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 444108 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 444111 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 444112 T2547 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@104f2e0 name:ZooKeeperConnection Watcher:127.0.0.1:57742 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 444113 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 444114 T2522 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 444119 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 444121 T2549 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@896251 name:ZooKeeperConnection Watcher:127.0.0.1:57742/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 444121 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 444122 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 444125 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 444127 T2551 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@10c3e42 name:ZooKeeperConnection Watcher:127.0.0.1:57742/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 444127 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 444128 T2522 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 444132 T2522 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 444137 T2522 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:57742_solr
[junit4] 2> 444138 T2522 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 444141 T2522 oasc.Overseer.start Overseer (id=92053395922223107-127.0.0.1:57742_solr-n_0000000000) starting
[junit4] 2> 444144 T2522 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 444149 T2522 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 444153 T2522 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 444157 T2522 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 444162 T2522 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 444166 T2522 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 444173 T2553 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 444175 T2522 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 444175 T2552 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 444177 T2522 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 444179 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 444183 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 444185 T2555 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1196b81 name:ZooKeeperConnection Watcher:127.0.0.1:57742/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 444185 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 444187 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 444190 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1
[junit4] 2> 444194 T2549 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4] 2> 444194 T2555 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4] 2> 444197 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 444199 T2552 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 444199 T2552 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 444200 T2552 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
[junit4] 2> 444200 T2552 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 444204 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 444205 T2555 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 444205 T2549 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 444697 T2522 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 444712 T2522 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4] 2> 444719 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 444721 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 444721 T2552 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 444727 T2552 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core2",
[junit4] 2> "core_node_name":"node2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 444728 T2552 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 444728 T2552 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
[junit4] 2> 444731 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 444834 T2555 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 444834 T2549 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 445222 T2522 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4] 2> 445233 T2522 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
[junit4] 2> 445240 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 445240 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 445241 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 445242 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 445242 T2552 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 445248 T2552 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core3",
[junit4] 2> "core_node_name":"node3",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 445248 T2552 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 445248 T2552 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 445251 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 445354 T2549 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 445354 T2555 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 445742 T2522 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 445753 T2522 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 445764 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 445765 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 445765 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 445766 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 445767 T2552 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 445771 T2552 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core4",
[junit4] 2> "core_node_name":"node4",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 445771 T2552 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 445771 T2552 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 445773 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 445876 T2555 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 445876 T2549 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 446267 T2522 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 446276 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 446276 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 446276 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 446278 T2552 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 446279 T2552 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core5",
[junit4] 2> "core_node_name":"node5",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 446280 T2552 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 446280 T2552 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
[junit4] 2> 446283 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 446386 T2555 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 446386 T2549 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 446776 T2522 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4] 2> 446787 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 446788 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 446788 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 446790 T2552 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 446791 T2552 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core6",
[junit4] 2> "core_node_name":"node6",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 446791 T2552 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 446791 T2552 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 446795 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 446899 T2549 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 446899 T2555 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 447288 T2522 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 447298 T2555 oascc.ZkStateReader$3.process Updating live nodes... (0)
[junit4] 2> 447301 T2522 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:57742 57742
[junit4] 2> 447869 T2539 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:57742 57742
[junit4] 2> 447871 T2522 oas.SolrTestCaseJ4.tearDown ###Ending testShardAssignment
[junit4] 2> 447872 T2552 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:57742_solr
[junit4] 2> 447875 T2556 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:316)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:288)
[junit4] 2>
[junit4] 2> 447881 T2522 oas.SolrTestCaseJ4.setUp ###Starting testShardAssignmentBigger
[junit4] 2> 447882 T2522 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 447883 T2557 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 447983 T2522 oasc.ZkTestServer.run start zk server on port:50146
[junit4] 2> 447984 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 447988 T2563 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@daa195 name:ZooKeeperConnection Watcher:127.0.0.1:50146 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 447989 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 447993 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 447994 T2565 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f8aac name:ZooKeeperConnection Watcher:127.0.0.1:50146 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 447995 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 447995 T2522 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 448000 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 448002 T2567 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@141291c name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 448002 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 448003 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 448006 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 448008 T2569 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@49f99d name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 448008 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 448009 T2522 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 448012 T2522 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 448018 T2522 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:50146_solr
[junit4] 2> 448018 T2522 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 448021 T2522 oasc.Overseer.start Overseer (id=92053396176633859-127.0.0.1:50146_solr-n_0000000000) starting
[junit4] 2> 448024 T2522 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 448029 T2522 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 448034 T2522 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 448038 T2522 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 448043 T2522 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 448048 T2522 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 448055 T2571 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 448056 T2522 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 448056 T2570 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 448058 T2522 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 448061 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 448065 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 448066 T2573 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1242e08 name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 448067 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 448069 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 448072 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node0
[junit4] 2> 448076 T2573 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4] 2> 448076 T2567 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4] 2> 448077 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 448078 T2575 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1fc812a name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 448078 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 448080 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 448083 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
[junit4] 2> 448087 T2573 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 448087 T2567 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 448087 T2575 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 448088 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 448089 T2577 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1714ec0 name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 448090 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 448091 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 448095 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node2
[junit4] 2> 448099 T2573 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 448099 T2577 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 448099 T2575 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 448100 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 448099 T2567 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 448102 T2579 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@fbc5e2 name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 448102 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 448104 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 448107 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node3
[junit4] 2> 448112 T2577 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 448112 T2575 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 448112 T2567 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 448112 T2573 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 448113 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 448113 T2579 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 448115 T2581 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@75e1db name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 448115 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 448117 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 448121 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node4
[junit4] 2> 448126 T2577 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4] 2> 448126 T2575 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4] 2> 448126 T2573 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4] 2> 448127 T2581 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4] 2> 448126 T2567 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4] 2> 448126 T2579 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4] 2> 448127 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 448129 T2583 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@11dc02e name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 448130 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 448132 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 448136 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node5
[junit4] 2> 448139 T2575 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4] 2> 448140 T2577 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4] 2> 448140 T2581 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4] 2> 448140 T2573 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4] 2> 448140 T2579 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4] 2> 448141 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 448141 T2583 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4] 2> 448140 T2567 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4] 2> 448143 T2585 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6d9f03 name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 448143 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 448145 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 448148 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node6
[junit4] 2> 448153 T2575 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4] 2> 448153 T2577 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4] 2> 448153 T2573 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4] 2> 448153 T2585 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4] 2> 448153 T2583 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4] 2> 448153 T2579 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4] 2> 448153 T2581 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4] 2> 448153 T2567 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4] 2> 448154 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 448156 T2587 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@958679 name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 448157 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 448159 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 448162 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node7
[junit4] 2> 448167 T2567 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4] 2> 448167 T2577 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4] 2> 448168 T2585 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4] 2> 448168 T2583 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4] 2> 448168 T2581 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4] 2> 448168 T2579 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4] 2> 448168 T2573 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4] 2> 448167 T2587 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4] 2> 448167 T2575 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4] 2> 448169 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 448171 T2589 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@169d949 name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 448171 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 448174 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 448177 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node8
[junit4] 2> 448182 T2589 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4] 2> 448183 T2585 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4] 2> 448183 T2583 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4] 2> 448183 T2587 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4] 2> 448183 T2573 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4] 2> 448183 T2579 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4] 2> 448183 T2581 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4] 2> 448183 T2575 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4] 2> 448183 T2577 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4] 2> 448182 T2567 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4] 2> 448185 T2591 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2e78fa name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 448184 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 448188 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 448190 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 448192 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node9
[junit4] 2> 448195 T2589 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 448195 T2577 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 448195 T2567 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 448196 T2573 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 448195 T2575 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 448195 T2587 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 448195 T2579 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 448195 T2583 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 448196 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 448198 T2593 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@156c3b name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 448196 T2591 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 448196 T2585 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 448196 T2581 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 448198 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 448200 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 448203 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node10
[junit4] 2> 448207 T2575 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 448207 T2583 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 448208 T2579 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 448207 T2577 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 448207 T2573 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 448207 T2589 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 448207 T2587 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 448209 T2585 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 448208 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 448208 T2581 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 448208 T2593 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 448208 T2591 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 448208 T2567 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 448211 T2595 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f1978e name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 448212 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 448213 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 448216 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node11
[junit4] 2> 448220 T2587 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 448220 T2567 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 448221 T2583 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 448220 T2589 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 448222 T2595 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 448220 T2575 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 448222 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 448222 T2593 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 448222 T2581 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 448224 T2597 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@175be88 name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 448221 T2591 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 448221 T2573 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 448221 T2577 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 448221 T2585 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 448221 T2579 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 448224 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 448228 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 448232 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node12
[junit4] 2> 448236 T2573 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 448236 T2577 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 448237 T2595 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 448237 T2593 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 448237 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 448238 T2581 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 448238 T2597 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 448237 T2583 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 448239 T2599 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@dde0f4 name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 448237 T2589 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 448237 T2575 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 448237 T2587 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 448237 T2585 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 448236 T2567 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 448236 T2579 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 448236 T2591 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 448240 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 448244 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 448248 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node13
[junit4] 2> 448254 T2587 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 448255 T2577 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 448255 T2579 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 448255 T2595 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 448255 T2573 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 448256 T2597 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 448256 T2599 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 448256 T2593 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 448256 T2585 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 448256 T2581 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 448256 T2583 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 448256 T2567 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 448256 T2575 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 448256 T2589 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 448256 T2591 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 448259 T2601 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@19e3760 name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 448257 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 448262 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 448264 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 448268 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node14
[junit4] 2> 448273 T2573 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 448273 T2587 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 448273 T2579 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 448273 T2591 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 448273 T2577 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 448273 T2595 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 448275 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 448275 T2599 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 448275 T2583 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 448275 T2585 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 448275 T2597 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 448275 T2601 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 448275 T2567 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 448275 T2575 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 448275 T2581 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 448274 T2593 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 448273 T2589 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 448277 T2603 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1818cfa name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 448281 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 448283 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 448286 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node15
[junit4] 2> 448291 T2577 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 448292 T2573 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 448291 T2587 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 448293 T2579 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 448293 T2575 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 448293 T2601 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 448294 T2599 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 448293 T2591 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 448293 T2597 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 448293 T2583 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 448293 T2567 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 448293 T2589 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 448295 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 448294 T2603 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 448294 T2581 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 448294 T2585 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 448294 T2595 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 448294 T2593 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 448297 T2605 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@154e83c name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 448299 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 448300 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 448304 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node16
[junit4] 2> 448313 T2587 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 448313 T2589 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 448314 T2595 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 448315 T2575 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 448314 T2585 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 448316 T2591 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 448313 T2573 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 448313 T2577 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 448313 T2593 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 448317 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 448316 T2599 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 448316 T2581 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 448316 T2605 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 448316 T2603 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 448316 T2583 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 448316 T2567 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 448315 T2601 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 448315 T2597 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 448315 T2579 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 448319 T2607 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d963a2 name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 448323 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 448324 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 448328 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node17
[junit4] 2> 448333 T2573 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 448333 T2577 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 448334 T2601 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 448334 T2607 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 448335 T2567 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 448335 T2583 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 448335 T2603 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 448334 T2589 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 448336 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 448334 T2595 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 448334 T2579 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 448334 T2575 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 448338 T2609 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@826a2e name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 448333 T2587 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 448333 T2591 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 448338 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 448336 T2585 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 448335 T2593 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 448335 T2599 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 448335 T2581 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 448335 T2605 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 448335 T2597 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 448341 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 448346 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node18
[junit4] 2> 448350 T2573 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 448350 T2577 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 448351 T2607 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 448352 T2587 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 448352 T2593 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 448351 T2601 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 448353 T2579 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 448353 T2581 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 448351 T2595 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 448353 T2605 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 448354 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 448353 T2609 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 448353 T2603 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 448353 T2597 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 448353 T2589 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 448353 T2567 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 448353 T2591 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 448353 T2575 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 448352 T2599 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 448352 T2583 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 448352 T2585 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 448356 T2611 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b5c498 name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 448359 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 448362 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 448366 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node19
[junit4] 2> 448371 T2587 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 448371 T2591 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 448372 T2579 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 448372 T2575 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 448372 T2577 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 448373 T2581 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 448372 T2589 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 448372 T2573 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 448372 T2595 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 448374 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 448373 T2609 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 448373 T2611 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 448373 T2603 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 448373 T2585 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 448373 T2593 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 448373 T2567 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 448373 T2607 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 448372 T2599 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 448372 T2597 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 448372 T2601 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 448372 T2583 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 448372 T2605 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 448376 T2613 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@18ed79a name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 448380 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 448382 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 448385 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node20
[junit4] 2> 448390 T2577 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 448390 T2579 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 448391 T2589 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 448391 T2601 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 448391 T2599 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 448391 T2573 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 448392 T2581 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 448391 T2605 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 448390 T2591 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 448390 T2595 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 448393 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 448392 T2603 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 448392 T2609 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 448392 T2611 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 448392 T2593 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 448392 T2585 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 448392 T2567 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 448392 T2575 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 448391 T2613 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 448391 T2583 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 448391 T2607 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 448391 T2597 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 448391 T2587 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 448395 T2615 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5f6bba name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 448399 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 448401 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 448404 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node21
[junit4] 2> 448409 T2587 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 448410 T2573 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 448410 T2595 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 448410 T2577 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 448409 T2591 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 448412 T2607 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 448412 T2593 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 448412 T2599 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 448413 T2603 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 448413 T2609 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 448412 T2567 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 448412 T2613 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 448412 T2575 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 448411 T2583 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 448411 T2601 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 448411 T2597 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 448411 T2579 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 448411 T2605 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 448411 T2589 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 448416 T2617 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@13fabc5 name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 448414 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 448413 T2615 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 448413 T2611 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 448412 T2581 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 448412 T2585 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 448418 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 448421 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 448426 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node22
[junit4] 2> 448431 T2587 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 448432 T2577 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 448432 T2579 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 448432 T2605 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 448433 T2611 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 448432 T2595 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 448432 T2591 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 448434 T2609 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 448432 T2573 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 448435 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 448434 T2603 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 448434 T2617 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 448434 T2581 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 448437 T2619 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@71217e name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null
[...truncated too long message...]
queue/qn-
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:243)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:240)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.create(SolrZkClient.java:240)
[junit4] 2> at org.apache.solr.cloud.DistributedQueue.createData(DistributedQueue.java:346)
[junit4] 2> at org.apache.solr.cloud.DistributedQueue.offer(DistributedQueue.java:330)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:198)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:163)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:125)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:314)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:221)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:452)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:216)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:163)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:125)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:358)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
[junit4] 2>
[junit4] 2> 750036 T3044 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 750037 T3044 oasc.LeaderElector$ElectionWatcher.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:358)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
[junit4] 2>
[junit4] 2> 750038 T3044 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4] 2> 750038 T3044 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 750121 T3116 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 750206 T2787 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 750206 T2787 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 750207 T2787 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 750207 T2787 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 750207 T2787 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 750206 T3156 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:316)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:288)
[junit4] 2>
[junit4] 2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-A819205877C572F6-001
[junit4] 2> NOTE: test params are: codec=Lucene41, sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=ar_JO, timezone=Africa/Ndjamena
[junit4] 2> NOTE: Linux 3.8.0-42-generic i386/Oracle Corporation 1.8.0_05 (32-bit)/cpus=8,threads=1,free=118175464,total=317980672
[junit4] 2> NOTE: All tests run in this JVM: [TestRecovery, OpenCloseCoreStressTest, HardAutoCommitTest, DeleteReplicaTest, IndexSchemaRuntimeFieldTest, TestXIncludeConfig, TestFreeTextSuggestions, CollectionsAPIDistributedZkTest, TestFieldTypeResource, BadCopyFieldTest, CopyFieldTest, TestCSVLoader, UniqFieldsUpdateProcessorFactoryTest, TestIndexingPerformance, TestDocumentBuilder, AutoCommitTest, AtomicUpdatesTest, TestQueryTypes, TestCollapseQParserPlugin, BasicDistributedZk2Test, TestWriterPerf, TestCoreContainer, TestComplexPhraseQParserPlugin, TestDistribDocBasedVersion, TestFastLRUCache, TestDynamicFieldResource, DOMUtilTest, QueryResultKeyTest, DistribDocExpirationUpdateProcessorTest, UpdateRequestProcessorFactoryTest, TimeZoneUtilsTest, SchemaVersionSpecificBehaviorTest, PreAnalyzedUpdateProcessorTest, SOLR749Test, TestNonDefinedSimilarityFactory, TestFileDictionaryLookup, MultiThreadedOCPTest, TestRTGBase, QueryParsingTest, OutputWriterTest, LoggingHandlerTest, TestInfoStreamLogging, TestPseudoReturnFields, TestFastWriter, TestReplicationHandlerBackup, AssignTest, TermVectorComponentTest, SolrRequestParserTest, TestPerFieldSimilarity, TestAtomicUpdateErrorCases, TestManagedResource, TermVectorComponentDistributedTest, TestReRankQParserPlugin, TestSerializedLuceneMatchVersion, TestChildDocTransformer, TestNoOpRegenerator, TestGroupingSearch, TestFieldTypeCollectionResource, TestBinaryField, StatsComponentTest, CursorMarkTest, BasicZkTest, SuggesterFSTTest, BJQParserTest, CurrencyFieldXmlFileTest, DistributedQueryElevationComponentTest, TestDistributedMissingSort, SearchHandlerTest, TestLFUCache, TestSolrJ, FileBasedSpellCheckerTest, TestRealTimeGet, RequestHandlersTest, AnalysisAfterCoreReloadTest, SolrInfoMBeanTest, TestDistributedGrouping, TestExtendedDismaxParser, URLClassifyProcessorTest, SuggestComponentTest, ChaosMonkeySafeLeaderTest]
[junit4] Completed on J1 in 74.71s, 1 test, 1 failure <<< FAILURES!
[...truncated 59 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1299: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:923: There were test failures: 405 suites, 1656 tests, 3 suite-level errors, 1 error, 1 failure, 954 ignored (13 assumptions)
Total time: 65 minutes 33 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 32bit/jdk1.8.0_05 -client -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.7.0_60) - Build # 10617
- Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/10617/
Java: 32bit/jdk1.7.0_60 -client -XX:+UseG1GC
1 tests failed.
REGRESSION: org.apache.lucene.analysis.core.TestRandomChains.testRandomChains
Error Message:
startOffset must be non-negative, and endOffset must be >= startOffset, startOffset=9,endOffset=5
Stack Trace:
java.lang.IllegalArgumentException: startOffset must be non-negative, and endOffset must be >= startOffset, startOffset=9,endOffset=5
at __randomizedtesting.SeedInfo.seed([8C2FE2CDE4898454:B1CECBACA39B9994]:0)
at org.apache.lucene.analysis.tokenattributes.PackedTokenAttributeImpl.setOffset(PackedTokenAttributeImpl.java:107)
at org.apache.lucene.analysis.shingle.ShingleFilter.incrementToken(ShingleFilter.java:345)
at org.apache.lucene.analysis.ValidatingTokenFilter.incrementToken(ValidatingTokenFilter.java:68)
at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkAnalysisConsistency(BaseTokenStreamTestCase.java:703)
at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:614)
at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:513)
at org.apache.lucene.analysis.core.TestRandomChains.testRandomChains(TestRandomChains.java:927)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at java.lang.Thread.run(Thread.java:745)
Build Log:
[...truncated 5756 lines...]
[junit4] Suite: org.apache.lucene.analysis.core.TestRandomChains
[junit4] 2> TEST FAIL: useCharFilter=false text='&;</p \ud83c\udc3f \u039a\uda51\udef5 \ua4a0\ua4ce\ua492\ua4b3\ua494'
[junit4] 2> Exception from random analyzer:
[junit4] 2> charfilters=
[junit4] 2> tokenizer=
[junit4] 2> org.apache.lucene.analysis.path.ReversePathHierarchyTokenizer(org.apache.lucene.analysis.core.TestRandomChains$CheckThatYouDidntReadAnythingReaderWrapper@c83859)
[junit4] 2> filters=
[junit4] 2> org.apache.lucene.analysis.id.IndonesianStemFilter(ValidatingTokenFilter@18a6330 term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,keyword=false, true)
[junit4] 2> org.apache.lucene.analysis.miscellaneous.WordDelimiterFilter(LUCENE_4_10, ValidatingTokenFilter@1cffabc term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,keyword=false, 7, [regebj, y, zfeqchmqil, ujunkv])
[junit4] 2> org.apache.lucene.analysis.miscellaneous.HyphenatedWordsFilter(ValidatingTokenFilter@f013b0 term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,keyword=false)
[junit4] 2> org.apache.lucene.analysis.shingle.ShingleFilter(ValidatingTokenFilter@3eaecd term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,keyword=false, <EMAIL>)
[junit4] 2> offsetsAreCorrect=false
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestRandomChains -Dtests.method=testRandomChains -Dtests.seed=8C2FE2CDE4898454 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=mk_MK -Dtests.timezone=Europe/Kiev -Dtests.file.encoding=ISO-8859-1
[junit4] ERROR 32.3s J1 | TestRandomChains.testRandomChains <<<
[junit4] > Throwable #1: java.lang.IllegalArgumentException: startOffset must be non-negative, and endOffset must be >= startOffset, startOffset=9,endOffset=5
[junit4] > at __randomizedtesting.SeedInfo.seed([8C2FE2CDE4898454:B1CECBACA39B9994]:0)
[junit4] > at org.apache.lucene.analysis.tokenattributes.PackedTokenAttributeImpl.setOffset(PackedTokenAttributeImpl.java:107)
[junit4] > at org.apache.lucene.analysis.shingle.ShingleFilter.incrementToken(ShingleFilter.java:345)
[junit4] > at org.apache.lucene.analysis.ValidatingTokenFilter.incrementToken(ValidatingTokenFilter.java:68)
[junit4] > at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkAnalysisConsistency(BaseTokenStreamTestCase.java:703)
[junit4] > at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:614)
[junit4] > at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:513)
[junit4] > at org.apache.lucene.analysis.core.TestRandomChains.testRandomChains(TestRandomChains.java:927)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] 2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/build/analysis/common/test/J1/./lucene.analysis.core.TestRandomChains-8C2FE2CDE4898454-001
[junit4] 2> NOTE: test params are: codec=CheapBastard, sim=DefaultSimilarity, locale=mk_MK, timezone=Europe/Kiev
[junit4] 2> NOTE: Linux 3.8.0-42-generic i386/Oracle Corporation 1.7.0_60 (32-bit)/cpus=8,threads=1,free=80436608,total=310378496
[junit4] 2> NOTE: All tests run in this JVM: [TestSolrSynonymParser, TestSwedishLightStemFilterFactory, TestHindiFilters, TestSnowball, TestCharArrayIterator, TestPersianCharFilter, TestTwoSuffixes, TestWordlistLoader, TestSpanishLightStemFilterFactory, TestTurkishLowerCaseFilter, TestClassicAnalyzer, TestReverseStringFilter, TestPortugueseStemFilter, TestBulgarianStemmer, TestKeepFilterFactory, TestCharacterUtils, TestNorwegianMinimalStemFilter, TestPerFieldAnalyzerWrapper, TestPrefixAwareTokenFilter, TestCircumfix, TestFinnishLightStemFilterFactory, TestMorphData, TestCodepointCountFilter, ShingleFilterTest, TestRollingCharBuffer, TestGreekStemmer, TestPatternReplaceFilter, TestKeywordMarkerFilter, TestTurkishAnalyzer, QueryAutoStopWordAnalyzerTest, TestPortugueseLightStemFilterFactory, TestCollationKeyFilterFactory, TestFinnishAnalyzer, TestRussianAnalyzer, TestPortugueseStemFilterFactory, TestFilesystemResourceLoader, TestHunspellStemFilterFactory, TestFrenchAnalyzer, TestSynonymMap, TestWordnetSynonymParser, TestScandinavianNormalizationFilterFactory, TestStemmerOverrideFilterFactory, TestCharTokenizers, TestArmenianAnalyzer, TestLimitTokenCountFilter, DateRecognizerSinkTokenizerTest, TestCommonGramsFilterFactory, TestEnglishMinimalStemFilterFactory, TestCJKTokenizer, TestCJKBigramFilterFactory, EdgeNGramTokenizerTest, TestDanishAnalyzer, TestHindiAnalyzer, TestSpanishLightStemFilter, TestGreekStemFilterFactory, TestApostropheFilterFactory, TestSnowballPorterFilterFactory, TestCapitalizationFilter, TestIrishLowerCaseFilter, TestHungarianLightStemFilterFactory, TestPersianAnalyzer, TestCollationKeyAnalyzer, TestSoraniNormalizationFilter, TestStandardFactories, TestCzechStemmer, TestTypeTokenFilter, TestKeywordMarkerFilterFactory, TestFactories, TestRandomChains]
[junit4] Completed on J1 in 56.67s, 2 tests, 1 error <<< FAILURES!
[...truncated 421 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/build.xml:544: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:2019: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/analysis/build.xml:106: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/analysis/build.xml:38: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/module-build.xml:60: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1299: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:923: There were test failures: 271 suites, 1456 tests, 1 error, 1 ignored
Total time: 32 minutes 37 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 32bit/jdk1.7.0_60 -client -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
Re: [JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.8.0_05) - Build #
10616 - Still Failing!
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
Thanks Mark. This is the same issue as SOLR-5596. I suggest that we disable
fsync for our tests. Let's discuss on the jira issue.
https://issues.apache.org/jira/browse/SOLR-5596?focusedCommentId=14053776&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14053776
On Mon, Jul 7, 2014 at 9:15 PM, Mark Miller <ma...@gmail.com> wrote:
> On July 7, 2014 at 11:39:31 AM, Shalin Shekhar Mangar (
> shalinmangar@gmail.com) wrote:
>
> I was looking into the OverseerTest failure and the following caught my
> attention.
>
> This error can be due to a slow machine but it also happens on fast
> machines if you try to do a lot of writes very fast on ZooKeeper which is
> what the testShardLeaderChange does. Perhaps we should add a small wait
> between operations?
>
>
> It’s probably worth a try. I tried once to work around it via ZooKeeper
> configuration options, thought there was a promising path or two after some
> googling, but I don’t think I ended up getting anywhere.
>
> - Mark
>
>
>
> [junit4] 2> 472241 T2893 oazsp.FileTxnLog.commit WARN fsync-ing the write ahead log in SyncThread:0 took 11588ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
>
>
>
> On Sun, Jul 6, 2014 at 10:46 AM, Policeman Jenkins Server <
> jenkins@thetaphi.de> wrote:
>
>> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/10616/
>> Java: 32bit/jdk1.8.0_05 -client -XX:+UseParallelGC
>>
>> 5 tests failed.
>> FAILED: junit.framework.TestSuite.org.apache.solr.cloud.OverseerTest
>>
>> Error Message:
>> ERROR: SolrZkClient opens=165 closes=164
>>
>> Stack Trace:
>> java.lang.AssertionError: ERROR: SolrZkClient opens=165 closes=164
>> at __randomizedtesting.SeedInfo.seed([A819205877C572F6]:0)
>> at org.junit.Assert.fail(Assert.java:93)
>> at
>> org.apache.solr.SolrTestCaseJ4.endTrackingZkClients(SolrTestCaseJ4.java:436)
>> at
>> org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:182)
>> at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:483)
>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:790)
>> at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at
>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
>> at
>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>> at
>> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>> at
>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>> at
>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>> at
>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>> at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at
>> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
>> at
>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>> at
>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>> at
>> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>> at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>> at java.lang.Thread.run(Thread.java:745)
>>
>>
>> FAILED: junit.framework.TestSuite.org.apache.solr.cloud.OverseerTest
>>
>> Error Message:
>> 2 threads leaked from SUITE scope at org.apache.solr.cloud.OverseerTest:
>> 1) Thread[id=2969,
>> name=TEST-OverseerTest.testShardLeaderChange-seed#[A819205877C572F6]-SendThread(
>> 127.0.0.1:33880), state=TIMED_WAITING, group=TGRP-OverseerTest]
>> at java.lang.Thread.sleep(Native Method) at
>> org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003) 2)
>> Thread[id=2970,
>> name=TEST-OverseerTest.testShardLeaderChange-seed#[A819205877C572F6]-EventThread,
>> state=WAITING, group=TGRP-OverseerTest] at
>> sun.misc.Unsafe.park(Native Method) at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> at
>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
>>
>> Stack Trace:
>> com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from
>> SUITE scope at org.apache.solr.cloud.OverseerTest:
>> 1) Thread[id=2969,
>> name=TEST-OverseerTest.testShardLeaderChange-seed#[A819205877C572F6]-SendThread(
>> 127.0.0.1:33880), state=TIMED_WAITING, group=TGRP-OverseerTest]
>> at java.lang.Thread.sleep(Native Method)
>> at
>> org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
>> 2) Thread[id=2970,
>> name=TEST-OverseerTest.testShardLeaderChange-seed#[A819205877C572F6]-EventThread,
>> state=WAITING, group=TGRP-OverseerTest]
>> at sun.misc.Unsafe.park(Native Method)
>> at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> at
>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
>> at __randomizedtesting.SeedInfo.seed([A819205877C572F6]:0)
>>
>>
>> FAILED: junit.framework.TestSuite.org.apache.solr.cloud.OverseerTest
>>
>> Error Message:
>> There are still zombie threads that couldn't be terminated: 1)
>> Thread[id=2969,
>> name=TEST-OverseerTest.testShardLeaderChange-seed#[A819205877C572F6]-SendThread(
>> 127.0.0.1:33880), state=TIMED_WAITING, group=TGRP-OverseerTest]
>> at java.lang.Thread.sleep(Native Method) at
>> org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
>>
>> Stack Trace:
>> com.carrotsearch.randomizedtesting.ThreadLeakError: There are still
>> zombie threads that couldn't be terminated:
>> 1) Thread[id=2969,
>> name=TEST-OverseerTest.testShardLeaderChange-seed#[A819205877C572F6]-SendThread(
>> 127.0.0.1:33880), state=TIMED_WAITING, group=TGRP-OverseerTest]
>> at java.lang.Thread.sleep(Native Method)
>> at
>> org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
>> at __randomizedtesting.SeedInfo.seed([A819205877C572F6]:0)
>>
>>
>> REGRESSION: org.apache.solr.cloud.OverseerTest.testShardLeaderChange
>>
>> Error Message:
>> KeeperErrorCode = NodeExists for /live_nodes/node1
>>
>> Stack Trace:
>> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode
>> = NodeExists for /live_nodes/node1
>> at
>> __randomizedtesting.SeedInfo.seed([A819205877C572F6:764AA7AF6D5D8707]:0)
>> at
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
>> at
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
>> at
>> org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:432)
>> at
>> org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
>> at
>> org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:429)
>> at
>> org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:386)
>> at
>> org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:373)
>> at
>> org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:350)
>> at
>> org.apache.solr.cloud.OverseerTest$MockZKController.<init>(OverseerTest.java:85)
>> at
>> org.apache.solr.cloud.OverseerTest.testShardLeaderChange(OverseerTest.java:746)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:483)
>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
>> at
>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
>> at
>> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>> at
>> org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
>> at
>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>> at
>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>> at
>> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>> at
>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>> at
>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>> at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>> at
>> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
>> at
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
>> at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at
>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
>> at
>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>> at
>> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>> at
>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>> at
>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>> at
>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>> at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at
>> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
>> at
>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>> at
>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>> at
>> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>> at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>> at java.lang.Thread.run(Thread.java:745)
>>
>>
>> FAILED: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch
>>
>> Error Message:
>> shard2 is not consistent. Got 153 from
>> http://127.0.0.1:51995/collection1lastClient and got 132 from
>> http://127.0.0.1:36739/collection1
>>
>> Stack Trace:
>> java.lang.AssertionError: shard2 is not consistent. Got 153 from
>> http://127.0.0.1:51995/collection1lastClient and got 132 from
>> http://127.0.0.1:36739/collection1
>> at
>> __randomizedtesting.SeedInfo.seed([A819205877C572F6:29FFAE40009A12CA]:0)
>> at org.junit.Assert.fail(Assert.java:93)
>> at
>> org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1139)
>> at
>> org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1118)
>> at
>> org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:150)
>> at
>> org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:483)
>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
>> at
>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
>> at
>> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>> at
>> org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
>> at
>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>> at
>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>> at
>> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>> at
>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>> at
>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>> at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>> at
>> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
>> at
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
>> at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at
>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
>> at
>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>> at
>> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>> at
>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>> at
>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>> at
>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>> at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at
>> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
>> at
>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>> at
>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>> at
>> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>> at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>> at java.lang.Thread.run(Thread.java:745)
>>
>>
>>
>>
>> Build Log:
>> [...truncated 11133 lines...]
>> [junit4] Suite: org.apache.solr.cloud.OverseerTest
>> [junit4] 2> Creating dataDir:
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.OverseerTest-A819205877C572F6-001/init-core-data-001
>> [junit4] 2> 443102 T2522 oas.SolrTestCaseJ4.buildSSLConfig
>> Randomized ssl (false) and clientAuth (false)
>> [junit4] 2> 443103 T2522 oas.SolrTestCaseJ4.initCore ####initCore
>> [junit4] 2> 443103 T2522 oas.SolrTestCaseJ4.initCore ####initCore end
>> [junit4] 2> 443107 T2522 oas.SolrTestCaseJ4.setUp ###Starting
>> testStateChange
>> [junit4] 2> 443107 T2522 oasc.ZkTestServer.run STARTING ZK TEST
>> SERVER
>> [junit4] 1> client port:0.0.0.0/0.0.0.0:0
>> [junit4] 2> 443108 T2523
>> oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
>> [junit4] 2> 443208 T2522 oasc.ZkTestServer.run start zk server on
>> port:45124
>> [junit4] 2> 443209 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 443282 T2529 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@ed72a5
>> name:ZooKeeperConnection Watcher:127.0.0.1:45124/solr got event
>> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 443283 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 443284 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 443287 T2531 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@6366c4
>> name:ZooKeeperConnection Watcher:127.0.0.1:45124 got event WatchedEvent
>> state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 443287 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 443290 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 443292 T2533 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@60240b
>> name:ZooKeeperConnection Watcher:127.0.0.1:45124 got event WatchedEvent
>> state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 443292 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 443293 T2522 oascc.SolrZkClient.makePath makePath: /solr
>> [junit4] 2> 443298 T2522 oascc.SolrZkClient.makePath makePath:
>> /live_nodes
>> [junit4] 2> 443301 T2522 oascc.SolrZkClient.makePath makePath:
>> /clusterstate.json
>> [junit4] 2> 443304 T2522 oascc.SolrZkClient.makePath makePath:
>> /aliases.json
>> [junit4] 2> 443306 T2522
>> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
>> state from ZooKeeper...
>> [junit4] 2> 443311 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 443312 T2535 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@68e786
>> name:ZooKeeperConnection Watcher:127.0.0.1:45124/solr got event
>> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 443313 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 443314 T2522 oascc.SolrZkClient.makePath makePath:
>> /overseer_elect
>> [junit4] 2> 443318 T2522 oascc.SolrZkClient.makePath makePath:
>> /overseer_elect/election
>> [junit4] 2> 443324 T2522
>> oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader
>> 127.0.0.1:45124_solr
>> [junit4] 2> 443324 T2522 oascc.SolrZkClient.makePath makePath:
>> /overseer_elect/leader
>> [junit4] 2> 443328 T2522 oasc.Overseer.start Overseer
>> (id=92053395863699459-127.0.0.1:45124_solr-n_0000000000) starting
>> [junit4] 2> 443331 T2522 oascc.SolrZkClient.makePath makePath:
>> /overseer/queue
>> [junit4] 2> 443336 T2522 oascc.SolrZkClient.makePath makePath:
>> /overseer/queue-work
>> [junit4] 2> 443340 T2522 oascc.SolrZkClient.makePath makePath:
>> /overseer/collection-map-failure
>> [junit4] 2> 443345 T2522 oascc.SolrZkClient.makePath makePath:
>> /overseer/collection-map-running
>> [junit4] 2> 443350 T2522 oascc.SolrZkClient.makePath makePath:
>> /overseer/collection-map-completed
>> [junit4] 2> 443357 T2522 oascc.SolrZkClient.makePath makePath:
>> /overseer/collection-queue-work
>> [junit4] 2> 443365 T2537 oasc.OverseerCollectionProcessor.run
>> Process current queue of collection creations
>> [junit4] 2> 443366 T2536 oasc.Overseer$ClusterStateUpdater.run
>> Starting to work on the main queue
>> [junit4] 2> 443368 T2522 oascc.ZkStateReader.updateClusterState
>> Updating cloud state from ZooKeeper...
>> [junit4] 2> 443369 T2536 oascc.ZkStateReader.updateClusterState
>> Updating cloud state from ZooKeeper...
>> [junit4] 2> 443370 T2536
>> oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null
>> message={
>> [junit4] 2> "operation":"state",
>> [junit4] 2> "base_url":"http://127.0.0.1/solr",
>> [junit4] 2> "node_name":"node1",
>> [junit4] 2> "collection":"collection1",
>> [junit4] 2> "core":"core1",
>> [junit4] 2> "roles":"",
>> [junit4] 2> "state":"recovering"}
>> [junit4] 2> 443370 T2536
>> oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard
>> shard=shard1
>> [junit4] 2> 443374 T2535
>> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
>> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>> [junit4] 2> 443375 T2529 oascc.ZkStateReader$2.process A cluster
>> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
>> [junit4] 2> 443420 T2522 oascc.ZkStateReader.updateClusterState
>> Updating cloud state from ZooKeeper...
>> [junit4] 2> 443422 T2535
>> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
>> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>> [junit4] 2> 443424 T2536 oascc.ZkStateReader.updateClusterState
>> Updating cloud state from ZooKeeper...
>> [junit4] 2> 443425 T2536
>> oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null
>> message={
>> [junit4] 2> "operation":"state",
>> [junit4] 2> "base_url":"http://127.0.0.1/solr",
>> [junit4] 2> "node_name":"node1",
>> [junit4] 2> "collection":"collection1",
>> [junit4] 2> "core":"core1",
>> [junit4] 2> "roles":"",
>> [junit4] 2> "state":"active"}
>> [junit4] 2> 443426 T2536
>> oasc.Overseer$ClusterStateUpdater.updateState node=core_node1 is already
>> registered
>> [junit4] 2> 443426 T2536
>> oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already
>> registered
>> [junit4] 2> 443431 T2535
>> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
>> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>> [junit4] 2> 443534 T2529 oascc.ZkStateReader$2.process A cluster
>> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
>> [junit4] 2> 443575 T2522 oasc.ZkTestServer.send4LetterWord
>> connecting to 127.0.0.1:45124 45124
>> [junit4] 2> 443988 T2523 oasc.ZkTestServer.send4LetterWord
>> connecting to 127.0.0.1:45124 45124
>> [junit4] 2> 443991 T2522 oas.SolrTestCaseJ4.tearDown ###Ending
>> testStateChange
>> [junit4] 2> 443992 T2536 oasc.Overseer$ClusterStateUpdater.run
>> Overseer Loop exiting : 127.0.0.1:45124_solr
>> [junit4] 2> 443994 T2538
>> oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not
>> read the data org.apache.zookeeper.KeeperException$SessionExpiredException:
>> KeeperErrorCode = Session expired for /overseer_elect/leader
>> [junit4] 2> at
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
>> [junit4] 2> at
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>> [junit4] 2> at
>> org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
>> [junit4] 2> at
>> org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
>> [junit4] 2> at
>> org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
>> [junit4] 2> at
>> org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
>> [junit4] 2> at
>> org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
>> [junit4] 2> at
>> org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:316)
>> [junit4] 2> at
>> org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
>> [junit4] 2> at
>> org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:288)
>> [junit4] 2>
>> [junit4] 2> 444000 T2522 oas.SolrTestCaseJ4.setUp ###Starting
>> testShardAssignment
>> [junit4] 2> 444001 T2522 oasc.ZkTestServer.run STARTING ZK TEST
>> SERVER
>> [junit4] 1> client port:0.0.0.0/0.0.0.0:0
>> [junit4] 2> 444002 T2539
>> oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
>> [junit4] 2> 444101 T2522 oasc.ZkTestServer.run start zk server on
>> port:57742
>> [junit4] 2> 444103 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 444107 T2545 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@9fc604
>> name:ZooKeeperConnection Watcher:127.0.0.1:57742 got event WatchedEvent
>> state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 444108 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 444111 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 444112 T2547 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@104f2e0
>> name:ZooKeeperConnection Watcher:127.0.0.1:57742 got event WatchedEvent
>> state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 444113 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 444114 T2522 oascc.SolrZkClient.makePath makePath: /solr
>> [junit4] 2> 444119 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 444121 T2549 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@896251
>> name:ZooKeeperConnection Watcher:127.0.0.1:57742/solr got event
>> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 444121 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 444122 T2522 oascc.SolrZkClient.makePath makePath:
>> /live_nodes
>> [junit4] 2> 444125 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 444127 T2551 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@10c3e42
>> name:ZooKeeperConnection Watcher:127.0.0.1:57742/solr got event
>> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 444127 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 444128 T2522 oascc.SolrZkClient.makePath makePath:
>> /overseer_elect
>> [junit4] 2> 444132 T2522 oascc.SolrZkClient.makePath makePath:
>> /overseer_elect/election
>> [junit4] 2> 444137 T2522
>> oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader
>> 127.0.0.1:57742_solr
>> [junit4] 2> 444138 T2522 oascc.SolrZkClient.makePath makePath:
>> /overseer_elect/leader
>> [junit4] 2> 444141 T2522 oasc.Overseer.start Overseer
>> (id=92053395922223107-127.0.0.1:57742_solr-n_0000000000) starting
>> [junit4] 2> 444144 T2522 oascc.SolrZkClient.makePath makePath:
>> /overseer/queue
>> [junit4] 2> 444149 T2522 oascc.SolrZkClient.makePath makePath:
>> /overseer/queue-work
>> [junit4] 2> 444153 T2522 oascc.SolrZkClient.makePath makePath:
>> /overseer/collection-map-failure
>> [junit4] 2> 444157 T2522 oascc.SolrZkClient.makePath makePath:
>> /overseer/collection-map-running
>> [junit4] 2> 444162 T2522 oascc.SolrZkClient.makePath makePath:
>> /overseer/collection-map-completed
>> [junit4] 2> 444166 T2522 oascc.SolrZkClient.makePath makePath:
>> /overseer/collection-queue-work
>> [junit4] 2> 444173 T2553 oasc.OverseerCollectionProcessor.run
>> Process current queue of collection creations
>> [junit4] 2> 444175 T2522 oascc.SolrZkClient.makePath makePath:
>> /clusterstate.json
>> [junit4] 2> 444175 T2552 oasc.Overseer$ClusterStateUpdater.run
>> Starting to work on the main queue
>> [junit4] 2> 444177 T2522 oascc.SolrZkClient.makePath makePath:
>> /aliases.json
>> [junit4] 2> 444179 T2522
>> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
>> state from ZooKeeper...
>> [junit4] 2> 444183 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 444185 T2555 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@1196b81
>> name:ZooKeeperConnection Watcher:127.0.0.1:57742/solr got event
>> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 444185 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 444187 T2522
>> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
>> state from ZooKeeper...
>> [junit4] 2> 444190 T2522 oascc.SolrZkClient.makePath makePath:
>> /live_nodes/127.0.0.1
>> [junit4] 2> 444194 T2549 oascc.ZkStateReader$3.process Updating live
>> nodes... (1)
>> [junit4] 2> 444194 T2555 oascc.ZkStateReader$3.process Updating live
>> nodes... (1)
>> [junit4] 2> 444197 T2551
>> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
>> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>> [junit4] 2> 444199 T2552 oascc.ZkStateReader.updateClusterState
>> Updating cloud state from ZooKeeper...
>> [junit4] 2> 444199 T2552
>> oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3
>> message={
>> [junit4] 2> "operation":"state",
>> [junit4] 2> "state":"active",
>> [junit4] 2> "node_name":"127.0.0.1",
>> [junit4] 2> "core":"core1",
>> [junit4] 2> "core_node_name":"node1",
>> [junit4] 2> "collection":"collection1",
>> [junit4] 2> "numShards":"3",
>> [junit4] 2> "base_url":"http://127.0.0.1/solr/"}
>> [junit4] 2> 444200 T2552
>> oasc.Overseer$ClusterStateUpdater.createCollection Create collection
>> collection1 with shards [shard1, shard2, shard3]
>> [junit4] 2> 444200 T2552
>> oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard
>> shard=shard2
>> [junit4] 2> 444204 T2551
>> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
>> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>> [junit4] 2> 444205 T2555 oascc.ZkStateReader$2.process A cluster
>> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>> [junit4] 2> 444205 T2549 oascc.ZkStateReader$2.process A cluster
>> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>> [junit4] 2> 444697 T2522 oascc.SolrZkClient.makePath makePath:
>> /collections/collection1/leader_elect/shard2/election
>> [junit4] 2> 444712 T2522 oascc.SolrZkClient.makePath makePath:
>> /collections/collection1/leaders/shard2
>> [junit4] 2> 444719 T2551
>> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
>> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>> [junit4] 2> 444721 T2551
>> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
>> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>> [junit4] 2> 444721 T2552 oascc.ZkStateReader.updateClusterState
>> Updating cloud state from ZooKeeper...
>> [junit4] 2> 444727 T2552
>> oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3
>> message={
>> [junit4] 2> "operation":"state",
>> [junit4] 2> "state":"active",
>> [junit4] 2> "node_name":"127.0.0.1",
>> [junit4] 2> "core":"core2",
>> [junit4] 2> "core_node_name":"node2",
>> [junit4] 2> "collection":"collection1",
>> [junit4] 2> "numShards":"3",
>> [junit4] 2> "base_url":"http://127.0.0.1/solr/"}
>> [junit4] 2> 444728 T2552
>> oasc.Overseer$ClusterStateUpdater.updateState Collection already exists
>> with numShards=3
>> [junit4] 2> 444728 T2552
>> oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard
>> shard=shard3
>> [junit4] 2> 444731 T2551
>> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
>> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>> [junit4] 2> 444834 T2555 oascc.ZkStateReader$2.process A cluster
>> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>> [junit4] 2> 444834 T2549 oascc.ZkStateReader$2.process A cluster
>> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>> [junit4] 2> 445222 T2522 oascc.SolrZkClient.makePath makePath:
>> /collections/collection1/leader_elect/shard3/election
>> [junit4] 2> 445233 T2522 oascc.SolrZkClient.makePath makePath:
>> /collections/collection1/leaders/shard3
>> [junit4] 2> 445240 T2551
>> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
>> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>> [junit4] 2> 445240 T2551
>> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
>> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>> [junit4] 2> 445241 T2551
>> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
>> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>> [junit4] 2> 445242 T2551
>> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
>> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>> [junit4] 2> 445242 T2552 oascc.ZkStateReader.updateClusterState
>> Updating cloud state from ZooKeeper...
>> [junit4] 2> 445248 T2552
>> oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3
>> message={
>> [junit4] 2> "operation":"state",
>> [junit4] 2> "state":"active",
>> [junit4] 2> "node_name":"127.0.0.1",
>> [junit4] 2> "core":"core3",
>> [junit4] 2> "core_node_name":"node3",
>> [junit4] 2> "collection":"collection1",
>> [junit4] 2> "numShards":"3",
>> [junit4] 2> "base_url":"http://127.0.0.1/solr/"}
>> [junit4] 2> 445248 T2552
>> oasc.Overseer$ClusterStateUpdater.updateState Collection already exists
>> with numShards=3
>> [junit4] 2> 445248 T2552
>> oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard
>> shard=shard1
>> [junit4] 2> 445251 T2551
>> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
>> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>> [junit4] 2> 445354 T2549 oascc.ZkStateReader$2.process A cluster
>> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>> [junit4] 2> 445354 T2555 oascc.ZkStateReader$2.process A cluster
>> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>> [junit4] 2> 445742 T2522 oascc.SolrZkClient.makePath makePath:
>> /collections/collection1/leader_elect/shard1/election
>> [junit4] 2> 445753 T2522 oascc.SolrZkClient.makePath makePath:
>> /collections/collection1/leaders/shard1
>> [junit4] 2> 445764 T2551
>> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
>> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>> [junit4] 2> 445765 T2551
>> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
>> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>> [junit4] 2> 445765 T2551
>> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
>> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>> [junit4] 2> 445766 T2551
>> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
>> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>> [junit4] 2> 445767 T2552 oascc.ZkStateReader.updateClusterState
>> Updating cloud state from ZooKeeper...
>> [junit4] 2> 445771 T2552
>> oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3
>> message={
>> [junit4] 2> "operation":"state",
>> [junit4] 2> "state":"active",
>> [junit4] 2> "node_name":"127.0.0.1",
>> [junit4] 2> "core":"core4",
>> [junit4] 2> "core_node_name":"node4",
>> [junit4] 2> "collection":"collection1",
>> [junit4] 2> "numShards":"3",
>> [junit4] 2> "base_url":"http://127.0.0.1/solr/"}
>> [junit4] 2> 445771 T2552
>> oasc.Overseer$ClusterStateUpdater.updateState Collection already exists
>> with numShards=3
>> [junit4] 2> 445771 T2552
>> oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard
>> shard=shard2
>> [junit4] 2> 445773 T2551
>> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
>> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>> [junit4] 2> 445876 T2555 oascc.ZkStateReader$2.process A cluster
>> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>> [junit4] 2> 445876 T2549 oascc.ZkStateReader$2.process A cluster
>> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>> [junit4] 2> 446267 T2522 oascc.SolrZkClient.makePath makePath:
>> /collections/collection1/leader_elect/shard2/election
>> [junit4] 2> 446276 T2551
>> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
>> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>> [junit4] 2> 446276 T2551
>> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
>> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>> [junit4] 2> 446276 T2551
>> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
>> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>> [junit4] 2> 446278 T2552 oascc.ZkStateReader.updateClusterState
>> Updating cloud state from ZooKeeper...
>> [junit4] 2> 446279 T2552
>> oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3
>> message={
>> [junit4] 2> "operation":"state",
>> [junit4] 2> "state":"active",
>> [junit4] 2> "node_name":"127.0.0.1",
>> [junit4] 2> "core":"core5",
>> [junit4] 2> "core_node_name":"node5",
>> [junit4] 2> "collection":"collection1",
>> [junit4] 2> "numShards":"3",
>> [junit4] 2> "base_url":"http://127.0.0.1/solr/"}
>> [junit4] 2> 446280 T2552
>> oasc.Overseer$ClusterStateUpdater.updateState Collection already exists
>> with numShards=3
>> [junit4] 2> 446280 T2552
>> oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard
>> shard=shard3
>> [junit4] 2> 446283 T2551
>> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
>> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>> [junit4] 2> 446386 T2555 oascc.ZkStateReader$2.process A cluster
>> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>> [junit4] 2> 446386 T2549 oascc.ZkStateReader$2.process A cluster
>> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>> [junit4] 2> 446776 T2522 oascc.SolrZkClient.makePath makePath:
>> /collections/collection1/leader_elect/shard3/election
>> [junit4] 2> 446787 T2551
>> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
>> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>> [junit4] 2> 446788 T2551
>> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
>> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>> [junit4] 2> 446788 T2551
>> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
>> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>> [junit4] 2> 446790 T2552 oascc.ZkStateReader.updateClusterState
>> Updating cloud state from ZooKeeper...
>> [junit4] 2> 446791 T2552
>> oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3
>> message={
>> [junit4] 2> "operation":"state",
>> [junit4] 2> "state":"active",
>> [junit4] 2> "node_name":"127.0.0.1",
>> [junit4] 2> "core":"core6",
>> [junit4] 2> "core_node_name":"node6",
>> [junit4] 2> "collection":"collection1",
>> [junit4] 2> "numShards":"3",
>> [junit4] 2> "base_url":"http://127.0.0.1/solr/"}
>> [junit4] 2> 446791 T2552
>> oasc.Overseer$ClusterStateUpdater.updateState Collection already exists
>> with numShards=3
>> [junit4] 2> 446791 T2552
>> oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard
>> shard=shard1
>> [junit4] 2> 446795 T2551
>> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
>> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>> [junit4] 2> 446899 T2549 oascc.ZkStateReader$2.process A cluster
>> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>> [junit4] 2> 446899 T2555 oascc.ZkStateReader$2.process A cluster
>> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>> [junit4] 2> 447288 T2522 oascc.SolrZkClient.makePath makePath:
>> /collections/collection1/leader_elect/shard1/election
>> [junit4] 2> 447298 T2555 oascc.ZkStateReader$3.process Updating live
>> nodes... (0)
>> [junit4] 2> 447301 T2522 oasc.ZkTestServer.send4LetterWord
>> connecting to 127.0.0.1:57742 57742
>> [junit4] 2> 447869 T2539 oasc.ZkTestServer.send4LetterWord
>> connecting to 127.0.0.1:57742 57742
>> [junit4] 2> 447871 T2522 oas.SolrTestCaseJ4.tearDown ###Ending
>> testShardAssignment
>> [junit4] 2> 447872 T2552 oasc.Overseer$ClusterStateUpdater.run
>> Overseer Loop exiting : 127.0.0.1:57742_solr
>> [junit4] 2> 447875 T2556
>> oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not
>> read the data org.apache.zookeeper.KeeperException$SessionExpiredException:
>> KeeperErrorCode = Session expired for /overseer_elect/leader
>> [junit4] 2> at
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
>> [junit4] 2> at
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>> [junit4] 2> at
>> org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
>> [junit4] 2> at
>> org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
>> [junit4] 2> at
>> org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
>> [junit4] 2> at
>> org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
>> [junit4] 2> at
>> org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
>> [junit4] 2> at
>> org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:316)
>> [junit4] 2> at
>> org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
>> [junit4] 2> at
>> org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:288)
>> [junit4] 2>
>> [junit4] 2> 447881 T2522 oas.SolrTestCaseJ4.setUp ###Starting
>> testShardAssignmentBigger
>> [junit4] 2> 447882 T2522 oasc.ZkTestServer.run STARTING ZK TEST
>> SERVER
>> [junit4] 1> client port:0.0.0.0/0.0.0.0:0
>> [junit4] 2> 447883 T2557
>> oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
>> [junit4] 2> 447983 T2522 oasc.ZkTestServer.run start zk server on
>> port:50146
>> [junit4] 2> 447984 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 447988 T2563 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@daa195
>> name:ZooKeeperConnection Watcher:127.0.0.1:50146 got event WatchedEvent
>> state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 447989 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 447993 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 447994 T2565 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@f8aac
>> name:ZooKeeperConnection Watcher:127.0.0.1:50146 got event WatchedEvent
>> state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 447995 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 447995 T2522 oascc.SolrZkClient.makePath makePath: /solr
>> [junit4] 2> 448000 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 448002 T2567 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@141291c
>> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
>> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 448002 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 448003 T2522 oascc.SolrZkClient.makePath makePath:
>> /live_nodes
>> [junit4] 2> 448006 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 448008 T2569 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@49f99d
>> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
>> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 448008 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 448009 T2522 oascc.SolrZkClient.makePath makePath:
>> /overseer_elect
>> [junit4] 2> 448012 T2522 oascc.SolrZkClient.makePath makePath:
>> /overseer_elect/election
>> [junit4] 2> 448018 T2522
>> oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader
>> 127.0.0.1:50146_solr
>> [junit4] 2> 448018 T2522 oascc.SolrZkClient.makePath makePath:
>> /overseer_elect/leader
>> [junit4] 2> 448021 T2522 oasc.Overseer.start Overseer
>> (id=92053396176633859-127.0.0.1:50146_solr-n_0000000000) starting
>> [junit4] 2> 448024 T2522 oascc.SolrZkClient.makePath makePath:
>> /overseer/queue
>> [junit4] 2> 448029 T2522 oascc.SolrZkClient.makePath makePath:
>> /overseer/queue-work
>> [junit4] 2> 448034 T2522 oascc.SolrZkClient.makePath makePath:
>> /overseer/collection-map-failure
>> [junit4] 2> 448038 T2522 oascc.SolrZkClient.makePath makePath:
>> /overseer/collection-map-running
>> [junit4] 2> 448043 T2522 oascc.SolrZkClient.makePath makePath:
>> /overseer/collection-map-completed
>> [junit4] 2> 448048 T2522 oascc.SolrZkClient.makePath makePath:
>> /overseer/collection-queue-work
>> [junit4] 2> 448055 T2571 oasc.OverseerCollectionProcessor.run
>> Process current queue of collection creations
>> [junit4] 2> 448056 T2522 oascc.SolrZkClient.makePath makePath:
>> /clusterstate.json
>> [junit4] 2> 448056 T2570 oasc.Overseer$ClusterStateUpdater.run
>> Starting to work on the main queue
>> [junit4] 2> 448058 T2522 oascc.SolrZkClient.makePath makePath:
>> /aliases.json
>> [junit4] 2> 448061 T2522
>> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
>> state from ZooKeeper...
>> [junit4] 2> 448065 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 448066 T2573 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@1242e08
>> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
>> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 448067 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 448069 T2522
>> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
>> state from ZooKeeper...
>> [junit4] 2> 448072 T2522 oascc.SolrZkClient.makePath makePath:
>> /live_nodes/node0
>> [junit4] 2> 448076 T2573 oascc.ZkStateReader$3.process Updating live
>> nodes... (1)
>> [junit4] 2> 448076 T2567 oascc.ZkStateReader$3.process Updating live
>> nodes... (1)
>> [junit4] 2> 448077 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 448078 T2575 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@1fc812a
>> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
>> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 448078 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 448080 T2522
>> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
>> state from ZooKeeper...
>> [junit4] 2> 448083 T2522 oascc.SolrZkClient.makePath makePath:
>> /live_nodes/node1
>> [junit4] 2> 448087 T2573 oascc.ZkStateReader$3.process Updating live
>> nodes... (2)
>> [junit4] 2> 448087 T2567 oascc.ZkStateReader$3.process Updating live
>> nodes... (2)
>> [junit4] 2> 448087 T2575 oascc.ZkStateReader$3.process Updating live
>> nodes... (2)
>> [junit4] 2> 448088 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 448089 T2577 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@1714ec0
>> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
>> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 448090 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 448091 T2522
>> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
>> state from ZooKeeper...
>> [junit4] 2> 448095 T2522 oascc.SolrZkClient.makePath makePath:
>> /live_nodes/node2
>> [junit4] 2> 448099 T2573 oascc.ZkStateReader$3.process Updating live
>> nodes... (3)
>> [junit4] 2> 448099 T2577 oascc.ZkStateReader$3.process Updating live
>> nodes... (3)
>> [junit4] 2> 448099 T2575 oascc.ZkStateReader$3.process Updating live
>> nodes... (3)
>> [junit4] 2> 448100 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 448099 T2567 oascc.ZkStateReader$3.process Updating live
>> nodes... (3)
>> [junit4] 2> 448102 T2579 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@fbc5e2
>> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
>> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 448102 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 448104 T2522
>> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
>> state from ZooKeeper...
>> [junit4] 2> 448107 T2522 oascc.SolrZkClient.makePath makePath:
>> /live_nodes/node3
>> [junit4] 2> 448112 T2577 oascc.ZkStateReader$3.process Updating live
>> nodes... (4)
>> [junit4] 2> 448112 T2575 oascc.ZkStateReader$3.process Updating live
>> nodes... (4)
>> [junit4] 2> 448112 T2567 oascc.ZkStateReader$3.process Updating live
>> nodes... (4)
>> [junit4] 2> 448112 T2573 oascc.ZkStateReader$3.process Updating live
>> nodes... (4)
>> [junit4] 2> 448113 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 448113 T2579 oascc.ZkStateReader$3.process Updating live
>> nodes... (4)
>> [junit4] 2> 448115 T2581 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@75e1db
>> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
>> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 448115 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 448117 T2522
>> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
>> state from ZooKeeper...
>> [junit4] 2> 448121 T2522 oascc.SolrZkClient.makePath makePath:
>> /live_nodes/node4
>> [junit4] 2> 448126 T2577 oascc.ZkStateReader$3.process Updating live
>> nodes... (5)
>> [junit4] 2> 448126 T2575 oascc.ZkStateReader$3.process Updating live
>> nodes... (5)
>> [junit4] 2> 448126 T2573 oascc.ZkStateReader$3.process Updating live
>> nodes... (5)
>> [junit4] 2> 448127 T2581 oascc.ZkStateReader$3.process Updating live
>> nodes... (5)
>> [junit4] 2> 448126 T2567 oascc.ZkStateReader$3.process Updating live
>> nodes... (5)
>> [junit4] 2> 448126 T2579 oascc.ZkStateReader$3.process Updating live
>> nodes... (5)
>> [junit4] 2> 448127 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 448129 T2583 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@11dc02e
>> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
>> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 448130 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 448132 T2522
>> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
>> state from ZooKeeper...
>> [junit4] 2> 448136 T2522 oascc.SolrZkClient.makePath makePath:
>> /live_nodes/node5
>> [junit4] 2> 448139 T2575 oascc.ZkStateReader$3.process Updating live
>> nodes... (6)
>> [junit4] 2> 448140 T2577 oascc.ZkStateReader$3.process Updating live
>> nodes... (6)
>> [junit4] 2> 448140 T2581 oascc.ZkStateReader$3.process Updating live
>> nodes... (6)
>> [junit4] 2> 448140 T2573 oascc.ZkStateReader$3.process Updating live
>> nodes... (6)
>> [junit4] 2> 448140 T2579 oascc.ZkStateReader$3.process Updating live
>> nodes... (6)
>> [junit4] 2> 448141 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 448141 T2583 oascc.ZkStateReader$3.process Updating live
>> nodes... (6)
>> [junit4] 2> 448140 T2567 oascc.ZkStateReader$3.process Updating live
>> nodes... (6)
>> [junit4] 2> 448143 T2585 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@6d9f03
>> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
>> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 448143 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 448145 T2522
>> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
>> state from ZooKeeper...
>> [junit4] 2> 448148 T2522 oascc.SolrZkClient.makePath makePath:
>> /live_nodes/node6
>> [junit4] 2> 448153 T2575 oascc.ZkStateReader$3.process Updating live
>> nodes... (7)
>> [junit4] 2> 448153 T2577 oascc.ZkStateReader$3.process Updating live
>> nodes... (7)
>> [junit4] 2> 448153 T2573 oascc.ZkStateReader$3.process Updating live
>> nodes... (7)
>> [junit4] 2> 448153 T2585 oascc.ZkStateReader$3.process Updating live
>> nodes... (7)
>> [junit4] 2> 448153 T2583 oascc.ZkStateReader$3.process Updating live
>> nodes... (7)
>> [junit4] 2> 448153 T2579 oascc.ZkStateReader$3.process Updating live
>> nodes... (7)
>> [junit4] 2> 448153 T2581 oascc.ZkStateReader$3.process Updating live
>> nodes... (7)
>> [junit4] 2> 448153 T2567 oascc.ZkStateReader$3.process Updating live
>> nodes... (7)
>> [junit4] 2> 448154 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 448156 T2587 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@958679
>> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
>> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 448157 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 448159 T2522
>> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
>> state from ZooKeeper...
>> [junit4] 2> 448162 T2522 oascc.SolrZkClient.makePath makePath:
>> /live_nodes/node7
>> [junit4] 2> 448167 T2567 oascc.ZkStateReader$3.process Updating live
>> nodes... (8)
>> [junit4] 2> 448167 T2577 oascc.ZkStateReader$3.process Updating live
>> nodes... (8)
>> [junit4] 2> 448168 T2585 oascc.ZkStateReader$3.process Updating live
>> nodes... (8)
>> [junit4] 2> 448168 T2583 oascc.ZkStateReader$3.process Updating live
>> nodes... (8)
>> [junit4] 2> 448168 T2581 oascc.ZkStateReader$3.process Updating live
>> nodes... (8)
>> [junit4] 2> 448168 T2579 oascc.ZkStateReader$3.process Updating live
>> nodes... (8)
>> [junit4] 2> 448168 T2573 oascc.ZkStateReader$3.process Updating live
>> nodes... (8)
>> [junit4] 2> 448167 T2587 oascc.ZkStateReader$3.process Updating live
>> nodes... (8)
>> [junit4] 2> 448167 T2575 oascc.ZkStateReader$3.process Updating live
>> nodes... (8)
>> [junit4] 2> 448169 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 448171 T2589 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@169d949
>> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
>> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 448171 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 448174 T2522
>> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
>> state from ZooKeeper...
>> [junit4] 2> 448177 T2522 oascc.SolrZkClient.makePath makePath:
>> /live_nodes/node8
>> [junit4] 2> 448182 T2589 oascc.ZkStateReader$3.process Updating live
>> nodes... (9)
>> [junit4] 2> 448183 T2585 oascc.ZkStateReader$3.process Updating live
>> nodes... (9)
>> [junit4] 2> 448183 T2583 oascc.ZkStateReader$3.process Updating live
>> nodes... (9)
>> [junit4] 2> 448183 T2587 oascc.ZkStateReader$3.process Updating live
>> nodes... (9)
>> [junit4] 2> 448183 T2573 oascc.ZkStateReader$3.process Updating live
>> nodes... (9)
>> [junit4] 2> 448183 T2579 oascc.ZkStateReader$3.process Updating live
>> nodes... (9)
>> [junit4] 2> 448183 T2581 oascc.ZkStateReader$3.process Updating live
>> nodes... (9)
>> [junit4] 2> 448183 T2575 oascc.ZkStateReader$3.process Updating live
>> nodes... (9)
>> [junit4] 2> 448183 T2577 oascc.ZkStateReader$3.process Updating live
>> nodes... (9)
>> [junit4] 2> 448182 T2567 oascc.ZkStateReader$3.process Updating live
>> nodes... (9)
>> [junit4] 2> 448185 T2591 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@2e78fa
>> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
>> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 448184 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 448188 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 448190 T2522
>> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
>> state from ZooKeeper...
>> [junit4] 2> 448192 T2522 oascc.SolrZkClient.makePath makePath:
>> /live_nodes/node9
>> [junit4] 2> 448195 T2589 oascc.ZkStateReader$3.process Updating live
>> nodes... (10)
>> [junit4] 2> 448195 T2577 oascc.ZkStateReader$3.process Updating live
>> nodes... (10)
>> [junit4] 2> 448195 T2567 oascc.ZkStateReader$3.process Updating live
>> nodes... (10)
>> [junit4] 2> 448196 T2573 oascc.ZkStateReader$3.process Updating live
>> nodes... (10)
>> [junit4] 2> 448195 T2575 oascc.ZkStateReader$3.process Updating live
>> nodes... (10)
>> [junit4] 2> 448195 T2587 oascc.ZkStateReader$3.process Updating live
>> nodes... (10)
>> [junit4] 2> 448195 T2579 oascc.ZkStateReader$3.process Updating live
>> nodes... (10)
>> [junit4] 2> 448195 T2583 oascc.ZkStateReader$3.process Updating live
>> nodes... (10)
>> [junit4] 2> 448196 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 448198 T2593 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@156c3b
>> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
>> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 448196 T2591 oascc.ZkStateReader$3.process Updating live
>> nodes... (10)
>> [junit4] 2> 448196 T2585 oascc.ZkStateReader$3.process Updating live
>> nodes... (10)
>> [junit4] 2> 448196 T2581 oascc.ZkStateReader$3.process Updating live
>> nodes... (10)
>> [junit4] 2> 448198 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 448200 T2522
>> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
>> state from ZooKeeper...
>> [junit4] 2> 448203 T2522 oascc.SolrZkClient.makePath makePath:
>> /live_nodes/node10
>> [junit4] 2> 448207 T2575 oascc.ZkStateReader$3.process Updating live
>> nodes... (11)
>> [junit4] 2> 448207 T2583 oascc.ZkStateReader$3.process Updating live
>> nodes... (11)
>> [junit4] 2> 448208 T2579 oascc.ZkStateReader$3.process Updating live
>> nodes... (11)
>> [junit4] 2> 448207 T2577 oascc.ZkStateReader$3.process Updating live
>> nodes... (11)
>> [junit4] 2> 448207 T2573 oascc.ZkStateReader$3.process Updating live
>> nodes... (11)
>> [junit4] 2> 448207 T2589 oascc.ZkStateReader$3.process Updating live
>> nodes... (11)
>> [junit4] 2> 448207 T2587 oascc.ZkStateReader$3.process Updating live
>> nodes... (11)
>> [junit4] 2> 448209 T2585 oascc.ZkStateReader$3.process Updating live
>> nodes... (11)
>> [junit4] 2> 448208 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 448208 T2581 oascc.ZkStateReader$3.process Updating live
>> nodes... (11)
>> [junit4] 2> 448208 T2593 oascc.ZkStateReader$3.process Updating live
>> nodes... (11)
>> [junit4] 2> 448208 T2591 oascc.ZkStateReader$3.process Updating live
>> nodes... (11)
>> [junit4] 2> 448208 T2567 oascc.ZkStateReader$3.process Updating live
>> nodes... (11)
>> [junit4] 2> 448211 T2595 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@f1978e
>> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
>> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 448212 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 448213 T2522
>> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
>> state from ZooKeeper...
>> [junit4] 2> 448216 T2522 oascc.SolrZkClient.makePath makePath:
>> /live_nodes/node11
>> [junit4] 2> 448220 T2587 oascc.ZkStateReader$3.process Updating live
>> nodes... (12)
>> [junit4] 2> 448220 T2567 oascc.ZkStateReader$3.process Updating live
>> nodes... (12)
>> [junit4] 2> 448221 T2583 oascc.ZkStateReader$3.process Updating live
>> nodes... (12)
>> [junit4] 2> 448220 T2589 oascc.ZkStateReader$3.process Updating live
>> nodes... (12)
>> [junit4] 2> 448222 T2595 oascc.ZkStateReader$3.process Updating live
>> nodes... (12)
>> [junit4] 2> 448220 T2575 oascc.ZkStateReader$3.process Updating live
>> nodes... (12)
>> [junit4] 2> 448222 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 448222 T2593 oascc.ZkStateReader$3.process Updating live
>> nodes... (12)
>> [junit4] 2> 448222 T2581 oascc.ZkStateReader$3.process Updating live
>> nodes... (12)
>> [junit4] 2> 448224 T2597 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@175be88
>> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
>> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 448221 T2591 oascc.ZkStateReader$3.process Updating live
>> nodes... (12)
>> [junit4] 2> 448221 T2573 oascc.ZkStateReader$3.process Updating live
>> nodes... (12)
>> [junit4] 2> 448221 T2577 oascc.ZkStateReader$3.process Updating live
>> nodes... (12)
>> [junit4] 2> 448221 T2585 oascc.ZkStateReader$3.process Updating live
>> nodes... (12)
>> [junit4] 2> 448221 T2579 oascc.ZkStateReader$3.process Updating live
>> nodes... (12)
>> [junit4] 2> 448224 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 448228 T2522
>> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
>> state from ZooKeeper...
>> [junit4] 2> 448232 T2522 oascc.SolrZkClient.makePath makePath:
>> /live_nodes/node12
>> [junit4] 2> 448236 T2573 oascc.ZkStateReader$3.process Updating live
>> nodes... (13)
>> [junit4] 2> 448236 T2577 oascc.ZkStateReader$3.process Updating live
>> nodes... (13)
>> [junit4] 2> 448237 T2595 oascc.ZkStateReader$3.process Updating live
>> nodes... (13)
>> [junit4] 2> 448237 T2593 oascc.ZkStateReader$3.process Updating live
>> nodes... (13)
>> [junit4] 2> 448237 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 448238 T2581 oascc.ZkStateReader$3.process Updating live
>> nodes... (13)
>> [junit4] 2> 448238 T2597 oascc.ZkStateReader$3.process Updating live
>> nodes... (13)
>> [junit4] 2> 448237 T2583 oascc.ZkStateReader$3.process Updating live
>> nodes... (13)
>> [junit4] 2> 448239 T2599 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@dde0f4
>> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
>> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 448237 T2589 oascc.ZkStateReader$3.process Updating live
>> nodes... (13)
>> [junit4] 2> 448237 T2575 oascc.ZkStateReader$3.process Updating live
>> nodes... (13)
>> [junit4] 2> 448237 T2587 oascc.ZkStateReader$3.process Updating live
>> nodes... (13)
>> [junit4] 2> 448237 T2585 oascc.ZkStateReader$3.process Updating live
>> nodes... (13)
>> [junit4] 2> 448236 T2567 oascc.ZkStateReader$3.process Updating live
>> nodes... (13)
>> [junit4] 2> 448236 T2579 oascc.ZkStateReader$3.process Updating live
>> nodes... (13)
>> [junit4] 2> 448236 T2591 oascc.ZkStateReader$3.process Updating live
>> nodes... (13)
>> [junit4] 2> 448240 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 448244 T2522
>> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
>> state from ZooKeeper...
>> [junit4] 2> 448248 T2522 oascc.SolrZkClient.makePath makePath:
>> /live_nodes/node13
>> [junit4] 2> 448254 T2587 oascc.ZkStateReader$3.process Updating live
>> nodes... (14)
>> [junit4] 2> 448255 T2577 oascc.ZkStateReader$3.process Updating live
>> nodes... (14)
>> [junit4] 2> 448255 T2579 oascc.ZkStateReader$3.process Updating live
>> nodes... (14)
>> [junit4] 2> 448255 T2595 oascc.ZkStateReader$3.process Updating live
>> nodes... (14)
>> [junit4] 2> 448255 T2573 oascc.ZkStateReader$3.process Updating live
>> nodes... (14)
>> [junit4] 2> 448256 T2597 oascc.ZkStateReader$3.process Updating live
>> nodes... (14)
>> [junit4] 2> 448256 T2599 oascc.ZkStateReader$3.process Updating live
>> nodes... (14)
>> [junit4] 2> 448256 T2593 oascc.ZkStateReader$3.process Updating live
>> nodes... (14)
>> [junit4] 2> 448256 T2585 oascc.ZkStateReader$3.process Updating live
>> nodes... (14)
>> [junit4] 2> 448256 T2581 oascc.ZkStateReader$3.process Updating live
>> nodes... (14)
>> [junit4] 2> 448256 T2583 oascc.ZkStateReader$3.process Updating live
>> nodes... (14)
>> [junit4] 2> 448256 T2567 oascc.ZkStateReader$3.process Updating live
>> nodes... (14)
>> [junit4] 2> 448256 T2575 oascc.ZkStateReader$3.process Updating live
>> nodes... (14)
>> [junit4] 2> 448256 T2589 oascc.ZkStateReader$3.process Updating live
>> nodes... (14)
>> [junit4] 2> 448256 T2591 oascc.ZkStateReader$3.process Updating live
>> nodes... (14)
>> [junit4] 2> 448259 T2601 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@19e3760
>> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
>> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 448257 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 448262 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 448264 T2522
>> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
>> state from ZooKeeper...
>> [junit4] 2> 448268 T2522 oascc.SolrZkClient.makePath makePath:
>> /live_nodes/node14
>> [junit4] 2> 448273 T2573 oascc.ZkStateReader$3.process Updating live
>> nodes... (15)
>> [junit4] 2> 448273 T2587 oascc.ZkStateReader$3.process Updating live
>> nodes... (15)
>> [junit4] 2> 448273 T2579 oascc.ZkStateReader$3.process Updating live
>> nodes... (15)
>> [junit4] 2> 448273 T2591 oascc.ZkStateReader$3.process Updating live
>> nodes... (15)
>> [junit4] 2> 448273 T2577 oascc.ZkStateReader$3.process Updating live
>> nodes... (15)
>> [junit4] 2> 448273 T2595 oascc.ZkStateReader$3.process Updating live
>> nodes... (15)
>> [junit4] 2> 448275 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 448275 T2599 oascc.ZkStateReader$3.process Updating live
>> nodes... (15)
>> [junit4] 2> 448275 T2583 oascc.ZkStateReader$3.process Updating live
>> nodes... (15)
>> [junit4] 2> 448275 T2585 oascc.ZkStateReader$3.process Updating live
>> nodes... (15)
>> [junit4] 2> 448275 T2597 oascc.ZkStateReader$3.process Updating live
>> nodes... (15)
>> [junit4] 2> 448275 T2601 oascc.ZkStateReader$3.process Updating live
>> nodes... (15)
>> [junit4] 2> 448275 T2567 oascc.ZkStateReader$3.process Updating live
>> nodes... (15)
>> [junit4] 2> 448275 T2575 oascc.ZkStateReader$3.process Updating live
>> nodes... (15)
>> [junit4] 2> 448275 T2581 oascc.ZkStateReader$3.process Updating live
>> nodes... (15)
>> [junit4] 2> 448274 T2593 oascc.ZkStateReader$3.process Updating live
>> nodes... (15)
>> [junit4] 2> 448273 T2589 oascc.ZkStateReader$3.process Updating live
>> nodes... (15)
>> [junit4] 2> 448277 T2603 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@1818cfa
>> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
>> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 448281 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 448283 T2522
>> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
>> state from ZooKeeper...
>> [junit4] 2> 448286 T2522 oascc.SolrZkClient.makePath makePath:
>> /live_nodes/node15
>> [junit4] 2> 448291 T2577 oascc.ZkStateReader$3.process Updating live
>> nodes... (16)
>> [junit4] 2> 448292 T2573 oascc.ZkStateReader$3.process Updating live
>> nodes... (16)
>> [junit4] 2> 448291 T2587 oascc.ZkStateReader$3.process Updating live
>> nodes... (16)
>> [junit4] 2> 448293 T2579 oascc.ZkStateReader$3.process Updating live
>> nodes... (16)
>> [junit4] 2> 448293 T2575 oascc.ZkStateReader$3.process Updating live
>> nodes... (16)
>> [junit4] 2> 448293 T2601 oascc.ZkStateReader$3.process Updating live
>> nodes... (16)
>> [junit4] 2> 448294 T2599 oascc.ZkStateReader$3.process Updating live
>> nodes... (16)
>> [junit4] 2> 448293 T2591 oascc.ZkStateReader$3.process Updating live
>> nodes... (16)
>> [junit4] 2> 448293 T2597 oascc.ZkStateReader$3.process Updating live
>> nodes... (16)
>> [junit4] 2> 448293 T2583 oascc.ZkStateReader$3.process Updating live
>> nodes... (16)
>> [junit4] 2> 448293 T2567 oascc.ZkStateReader$3.process Updating live
>> nodes... (16)
>> [junit4] 2> 448293 T2589 oascc.ZkStateReader$3.process Updating live
>> nodes... (16)
>> [junit4] 2> 448295 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 448294 T2603 oascc.ZkStateReader$3.process Updating live
>> nodes... (16)
>> [junit4] 2> 448294 T2581 oascc.ZkStateReader$3.process Updating live
>> nodes... (16)
>> [junit4] 2> 448294 T2585 oascc.ZkStateReader$3.process Updating live
>> nodes... (16)
>> [junit4] 2> 448294 T2595 oascc.ZkStateReader$3.process Updating live
>> nodes... (16)
>> [junit4] 2> 448294 T2593 oascc.ZkStateReader$3.process Updating live
>> nodes... (16)
>> [junit4] 2> 448297 T2605 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@154e83c
>> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
>> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 448299 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 448300 T2522
>> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
>> state from ZooKeeper...
>> [junit4] 2> 448304 T2522 oascc.SolrZkClient.makePath makePath:
>> /live_nodes/node16
>> [junit4] 2> 448313 T2587 oascc.ZkStateReader$3.process Updating live
>> nodes... (17)
>> [junit4] 2> 448313 T2589 oascc.ZkStateReader$3.process Updating live
>> nodes... (17)
>> [junit4] 2> 448314 T2595 oascc.ZkStateReader$3.process Updating live
>> nodes... (17)
>> [junit4] 2> 448315 T2575 oascc.ZkStateReader$3.process Updating live
>> nodes... (17)
>> [junit4] 2> 448314 T2585 oascc.ZkStateReader$3.process Updating live
>> nodes... (17)
>> [junit4] 2> 448316 T2591 oascc.ZkStateReader$3.process Updating live
>> nodes... (17)
>> [junit4] 2> 448313 T2573 oascc.ZkStateReader$3.process Updating live
>> nodes... (17)
>> [junit4] 2> 448313 T2577 oascc.ZkStateReader$3.process Updating live
>> nodes... (17)
>> [junit4] 2> 448313 T2593 oascc.ZkStateReader$3.process Updating live
>> nodes... (17)
>> [junit4] 2> 448317 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 448316 T2599 oascc.ZkStateReader$3.process Updating live
>> nodes... (17)
>> [junit4] 2> 448316 T2581 oascc.ZkStateReader$3.process Updating live
>> nodes... (17)
>> [junit4] 2> 448316 T2605 oascc.ZkStateReader$3.process Updating live
>> nodes... (17)
>> [junit4] 2> 448316 T2603 oascc.ZkStateReader$3.process Updating live
>> nodes... (17)
>> [junit4] 2> 448316 T2583 oascc.ZkStateReader$3.process Updating live
>> nodes... (17)
>> [junit4] 2> 448316 T2567 oascc.ZkStateReader$3.process Updating live
>> nodes... (17)
>> [junit4] 2> 448315 T2601 oascc.ZkStateReader$3.process Updating live
>> nodes... (17)
>> [junit4] 2> 448315 T2597 oascc.ZkStateReader$3.process Updating live
>> nodes... (17)
>> [junit4] 2> 448315 T2579 oascc.ZkStateReader$3.process Updating live
>> nodes... (17)
>> [junit4] 2> 448319 T2607 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@d963a2
>> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
>> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 448323 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 448324 T2522
>> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
>> state from ZooKeeper...
>> [junit4] 2> 448328 T2522 oascc.SolrZkClient.makePath makePath:
>> /live_nodes/node17
>> [junit4] 2> 448333 T2573 oascc.ZkStateReader$3.process Updating live
>> nodes... (18)
>> [junit4] 2> 448333 T2577 oascc.ZkStateReader$3.process Updating live
>> nodes... (18)
>> [junit4] 2> 448334 T2601 oascc.ZkStateReader$3.process Updating live
>> nodes... (18)
>> [junit4] 2> 448334 T2607 oascc.ZkStateReader$3.process Updating live
>> nodes... (18)
>> [junit4] 2> 448335 T2567 oascc.ZkStateReader$3.process Updating live
>> nodes... (18)
>> [junit4] 2> 448335 T2583 oascc.ZkStateReader$3.process Updating live
>> nodes... (18)
>> [junit4] 2> 448335 T2603 oascc.ZkStateReader$3.process Updating live
>> nodes... (18)
>> [junit4] 2> 448334 T2589 oascc.ZkStateReader$3.process Updating live
>> nodes... (18)
>> [junit4] 2> 448336 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 448334 T2595 oascc.ZkStateReader$3.process Updating live
>> nodes... (18)
>> [junit4] 2> 448334 T2579 oascc.ZkStateReader$3.process Updating live
>> nodes... (18)
>> [junit4] 2> 448334 T2575 oascc.ZkStateReader$3.process Updating live
>> nodes... (18)
>> [junit4] 2> 448338 T2609 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@826a2e
>> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
>> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 448333 T2587 oascc.ZkStateReader$3.process Updating live
>> nodes... (18)
>> [junit4] 2> 448333 T2591 oascc.ZkStateReader$3.process Updating live
>> nodes... (18)
>> [junit4] 2> 448338 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 448336 T2585 oascc.ZkStateReader$3.process Updating live
>> nodes... (18)
>> [junit4] 2> 448335 T2593 oascc.ZkStateReader$3.process Updating live
>> nodes... (18)
>> [junit4] 2> 448335 T2599 oascc.ZkStateReader$3.process Updating live
>> nodes... (18)
>> [junit4] 2> 448335 T2581 oascc.ZkStateReader$3.process Updating live
>> nodes... (18)
>> [junit4] 2> 448335 T2605 oascc.ZkStateReader$3.process Updating live
>> nodes... (18)
>> [junit4] 2> 448335 T2597 oascc.ZkStateReader$3.process Updating live
>> nodes... (18)
>> [junit4] 2> 448341 T2522
>> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
>> state from ZooKeeper...
>> [junit4] 2> 448346 T2522 oascc.SolrZkClient.makePath makePath:
>> /live_nodes/node18
>> [junit4] 2> 448350 T2573 oascc.ZkStateReader$3.process Updating live
>> nodes... (19)
>> [junit4] 2> 448350 T2577 oascc.ZkStateReader$3.process Updating live
>> nodes... (19)
>> [junit4] 2> 448351 T2607 oascc.ZkStateReader$3.process Updating live
>> nodes... (19)
>> [junit4] 2> 448352 T2587 oascc.ZkStateReader$3.process Updating live
>> nodes... (19)
>> [junit4] 2> 448352 T2593 oascc.ZkStateReader$3.process Updating live
>> nodes... (19)
>> [junit4] 2> 448351 T2601 oascc.ZkStateReader$3.process Updating live
>> nodes... (19)
>> [junit4] 2> 448353 T2579 oascc.ZkStateReader$3.process Updating live
>> nodes... (19)
>> [junit4] 2> 448353 T2581 oascc.ZkStateReader$3.process Updating live
>> nodes... (19)
>> [junit4] 2> 448351 T2595 oascc.ZkStateReader$3.process Updating live
>> nodes... (19)
>> [junit4] 2> 448353 T2605 oascc.ZkStateReader$3.process Updating live
>> nodes... (19)
>> [junit4] 2> 448354 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 448353 T2609 oascc.ZkStateReader$3.process Updating live
>> nodes... (19)
>> [junit4] 2> 448353 T2603 oascc.ZkStateReader$3.process Updating live
>> nodes... (19)
>> [junit4] 2> 448353 T2597 oascc.ZkStateReader$3.process Updating live
>> nodes... (19)
>> [junit4] 2> 448353 T2589 oascc.ZkStateReader$3.process Updating live
>> nodes... (19)
>> [junit4] 2> 448353 T2567 oascc.ZkStateReader$3.process Updating live
>> nodes... (19)
>> [junit4] 2> 448353 T2591 oascc.ZkStateReader$3.process Updating live
>> nodes... (19)
>> [junit4] 2> 448353 T2575 oascc.ZkStateReader$3.process Updating live
>> nodes... (19)
>> [junit4] 2> 448352 T2599 oascc.ZkStateReader$3.process Updating live
>> nodes... (19)
>> [junit4] 2> 448352 T2583 oascc.ZkStateReader$3.process Updating live
>> nodes... (19)
>> [junit4] 2> 448352 T2585 oascc.ZkStateReader$3.process Updating live
>> nodes... (19)
>> [junit4] 2> 448356 T2611 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@1b5c498
>> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
>> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 448359 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 448362 T2522
>> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
>> state from ZooKeeper...
>> [junit4] 2> 448366 T2522 oascc.SolrZkClient.makePath makePath:
>> /live_nodes/node19
>> [junit4] 2> 448371 T2587 oascc.ZkStateReader$3.process Updating live
>> nodes... (20)
>> [junit4] 2> 448371 T2591 oascc.ZkStateReader$3.process Updating live
>> nodes... (20)
>> [junit4] 2> 448372 T2579 oascc.ZkStateReader$3.process Updating live
>> nodes... (20)
>> [junit4] 2> 448372 T2575 oascc.ZkStateReader$3.process Updating live
>> nodes... (20)
>> [junit4] 2> 448372 T2577 oascc.ZkStateReader$3.process Updating live
>> nodes... (20)
>> [junit4] 2> 448373 T2581 oascc.ZkStateReader$3.process Updating live
>> nodes... (20)
>> [junit4] 2> 448372 T2589 oascc.ZkStateReader$3.process Updating live
>> nodes... (20)
>> [junit4] 2> 448372 T2573 oascc.ZkStateReader$3.process Updating live
>> nodes... (20)
>> [junit4] 2> 448372 T2595 oascc.ZkStateReader$3.process Updating live
>> nodes... (20)
>> [junit4] 2> 448374 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 448373 T2609 oascc.ZkStateReader$3.process Updating live
>> nodes... (20)
>> [junit4] 2> 448373 T2611 oascc.ZkStateReader$3.process Updating live
>> nodes... (20)
>> [junit4] 2> 448373 T2603 oascc.ZkStateReader$3.process Updating live
>> nodes... (20)
>> [junit4] 2> 448373 T2585 oascc.ZkStateReader$3.process Updating live
>> nodes... (20)
>> [junit4] 2> 448373 T2593 oascc.ZkStateReader$3.process Updating live
>> nodes... (20)
>> [junit4] 2> 448373 T2567 oascc.ZkStateReader$3.process Updating live
>> nodes... (20)
>> [junit4] 2> 448373 T2607 oascc.ZkStateReader$3.process Updating live
>> nodes... (20)
>> [junit4] 2> 448372 T2599 oascc.ZkStateReader$3.process Updating live
>> nodes... (20)
>> [junit4] 2> 448372 T2597 oascc.ZkStateReader$3.process Updating live
>> nodes... (20)
>> [junit4] 2> 448372 T2601 oascc.ZkStateReader$3.process Updating live
>> nodes... (20)
>> [junit4] 2> 448372 T2583 oascc.ZkStateReader$3.process Updating live
>> nodes... (20)
>> [junit4] 2> 448372 T2605 oascc.ZkStateReader$3.process Updating live
>> nodes... (20)
>> [junit4] 2> 448376 T2613 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@18ed79a
>> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
>> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 448380 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 448382 T2522
>> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
>> state from ZooKeeper...
>> [junit4] 2> 448385 T2522 oascc.SolrZkClient.makePath makePath:
>> /live_nodes/node20
>> [junit4] 2> 448390 T2577 oascc.ZkStateReader$3.process Updating live
>> nodes... (21)
>> [junit4] 2> 448390 T2579 oascc.ZkStateReader$3.process Updating live
>> nodes... (21)
>> [junit4] 2> 448391 T2589 oascc.ZkStateReader$3.process Updating live
>> nodes... (21)
>> [junit4] 2> 448391 T2601 oascc.ZkStateReader$3.process Updating live
>> nodes... (21)
>> [junit4] 2> 448391 T2599 oascc.ZkStateReader$3.process Updating live
>> nodes... (21)
>> [junit4] 2> 448391 T2573 oascc.ZkStateReader$3.process Updating live
>> nodes... (21)
>> [junit4] 2> 448392 T2581 oascc.ZkStateReader$3.process Updating live
>> nodes... (21)
>> [junit4] 2> 448391 T2605 oascc.ZkStateReader$3.process Updating live
>> nodes... (21)
>> [junit4] 2> 448390 T2591 oascc.ZkStateReader$3.process Updating live
>> nodes... (21)
>> [junit4] 2> 448390 T2595 oascc.ZkStateReader$3.process Updating live
>> nodes... (21)
>> [junit4] 2> 448393 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 448392 T2603 oascc.ZkStateReader$3.process Updating live
>> nodes... (21)
>> [junit4] 2> 448392 T2609 oascc.ZkStateReader$3.process Updating live
>> nodes... (21)
>> [junit4] 2> 448392 T2611 oascc.ZkStateReader$3.process Updating live
>> nodes... (21)
>> [junit4] 2> 448392 T2593 oascc.ZkStateReader$3.process Updating live
>> nodes... (21)
>> [junit4] 2> 448392 T2585 oascc.ZkStateReader$3.process Updating live
>> nodes... (21)
>> [junit4] 2> 448392 T2567 oascc.ZkStateReader$3.process Updating live
>> nodes... (21)
>> [junit4] 2> 448392 T2575 oascc.ZkStateReader$3.process Updating live
>> nodes... (21)
>> [junit4] 2> 448391 T2613 oascc.ZkStateReader$3.process Updating live
>> nodes... (21)
>> [junit4] 2> 448391 T2583 oascc.ZkStateReader$3.process Updating live
>> nodes... (21)
>> [junit4] 2> 448391 T2607 oascc.ZkStateReader$3.process Updating live
>> nodes... (21)
>> [junit4] 2> 448391 T2597 oascc.ZkStateReader$3.process Updating live
>> nodes... (21)
>> [junit4] 2> 448391 T2587 oascc.ZkStateReader$3.process Updating live
>> nodes... (21)
>> [junit4] 2> 448395 T2615 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@5f6bba
>> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
>> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 448399 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 448401 T2522
>> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
>> state from ZooKeeper...
>> [junit4] 2> 448404 T2522 oascc.SolrZkClient.makePath makePath:
>> /live_nodes/node21
>> [junit4] 2> 448409 T2587 oascc.ZkStateReader$3.process Updating live
>> nodes... (22)
>> [junit4] 2> 448410 T2573 oascc.ZkStateReader$3.process Updating live
>> nodes... (22)
>> [junit4] 2> 448410 T2595 oascc.ZkStateReader$3.process Updating live
>> nodes... (22)
>> [junit4] 2> 448410 T2577 oascc.ZkStateReader$3.process Updating live
>> nodes... (22)
>> [junit4] 2> 448409 T2591 oascc.ZkStateReader$3.process Updating live
>> nodes... (22)
>> [junit4] 2> 448412 T2607 oascc.ZkStateReader$3.process Updating live
>> nodes... (22)
>> [junit4] 2> 448412 T2593 oascc.ZkStateReader$3.process Updating live
>> nodes... (22)
>> [junit4] 2> 448412 T2599 oascc.ZkStateReader$3.process Updating live
>> nodes... (22)
>> [junit4] 2> 448413 T2603 oascc.ZkStateReader$3.process Updating live
>> nodes... (22)
>> [junit4] 2> 448413 T2609 oascc.ZkStateReader$3.process Updating live
>> nodes... (22)
>> [junit4] 2> 448412 T2567 oascc.ZkStateReader$3.process Updating live
>> nodes... (22)
>> [junit4] 2> 448412 T2613 oascc.ZkStateReader$3.process Updating live
>> nodes... (22)
>> [junit4] 2> 448412 T2575 oascc.ZkStateReader$3.process Updating live
>> nodes... (22)
>> [junit4] 2> 448411 T2583 oascc.ZkStateReader$3.process Updating live
>> nodes... (22)
>> [junit4] 2> 448411 T2601 oascc.ZkStateReader$3.process Updating live
>> nodes... (22)
>> [junit4] 2> 448411 T2597 oascc.ZkStateReader$3.process Updating live
>> nodes... (22)
>> [junit4] 2> 448411 T2579 oascc.ZkStateReader$3.process Updating live
>> nodes... (22)
>> [junit4] 2> 448411 T2605 oascc.ZkStateReader$3.process Updating live
>> nodes... (22)
>> [junit4] 2> 448411 T2589 oascc.ZkStateReader$3.process Updating live
>> nodes... (22)
>> [junit4] 2> 448416 T2617 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@13fabc5
>> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
>> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 448414 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 448413 T2615 oascc.ZkStateReader$3.process Updating live
>> nodes... (22)
>> [junit4] 2> 448413 T2611 oascc.ZkStateReader$3.process Updating live
>> nodes... (22)
>> [junit4] 2> 448412 T2581 oascc.ZkStateReader$3.process Updating live
>> nodes... (22)
>> [junit4] 2> 448412 T2585 oascc.ZkStateReader$3.process Updating live
>> nodes... (22)
>> [junit4] 2> 448418 T2522 oascc.ConnectionManager.waitForConnected
>> Client is connected to ZooKeeper
>> [junit4] 2> 448421 T2522
>> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
>> state from ZooKeeper...
>> [junit4] 2> 448426 T2522 oascc.SolrZkClient.makePath makePath:
>> /live_nodes/node22
>> [junit4] 2> 448431 T2587 oascc.ZkStateReader$3.process Updating live
>> nodes... (23)
>> [junit4] 2> 448432 T2577 oascc.ZkStateReader$3.process Updating live
>> nodes... (23)
>> [junit4] 2> 448432 T2579 oascc.ZkStateReader$3.process Updating live
>> nodes... (23)
>> [junit4] 2> 448432 T2605 oascc.ZkStateReader$3.process Updating live
>> nodes... (23)
>> [junit4] 2> 448433 T2611 oascc.ZkStateReader$3.process Updating live
>> nodes... (23)
>> [junit4] 2> 448432 T2595 oascc.ZkStateReader$3.process Updating live
>> nodes... (23)
>> [junit4] 2> 448432 T2591 oascc.ZkStateReader$3.process Updating live
>> nodes... (23)
>> [junit4] 2> 448434 T2609 oascc.ZkStateReader$3.process Updating live
>> nodes... (23)
>> [junit4] 2> 448432 T2573 oascc.ZkStateReader$3.process Updating live
>> nodes... (23)
>> [junit4] 2> 448435 T2522 oascc.ConnectionManager.waitForConnected
>> Waiting for client to connect to ZooKeeper
>> [junit4] 2> 448434 T2603 oascc.ZkStateReader$3.process Updating live
>> nodes... (23)
>> [junit4] 2> 448434 T2617 oascc.ZkStateReader$3.process Updating live
>> nodes... (23)
>> [junit4] 2> 448434 T2581 oascc.ZkStateReader$3.process Updating live
>> nodes... (23)
>> [junit4] 2> 448437 T2619 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@71217e
>> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
>> WatchedEvent state:SyncConnected type:None path:null
>>
>> [...truncated too long message...]
>>
>> queue/qn-
>> [junit4] 2> at
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
>> [junit4] 2> at
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>> [junit4] 2> at
>> org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
>> [junit4] 2> at
>> org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:243)
>> [junit4] 2> at
>> org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:240)
>> [junit4] 2> at
>> org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
>> [junit4] 2> at
>> org.apache.solr.common.cloud.SolrZkClient.create(SolrZkClient.java:240)
>> [junit4] 2> at
>> org.apache.solr.cloud.DistributedQueue.createData(DistributedQueue.java:346)
>> [junit4] 2> at
>> org.apache.solr.cloud.DistributedQueue.offer(DistributedQueue.java:330)
>> [junit4] 2> at
>> org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:198)
>> [junit4] 2> at
>> org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:163)
>> [junit4] 2> at
>> org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:125)
>> [junit4] 2> at
>> org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:314)
>> [junit4] 2> at
>> org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:221)
>> [junit4] 2> at
>> org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:452)
>> [junit4] 2> at
>> org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:216)
>> [junit4] 2> at
>> org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:163)
>> [junit4] 2> at
>> org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:125)
>> [junit4] 2> at
>> org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
>> [junit4] 2> at
>> org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:358)
>> [junit4] 2> at
>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
>> [junit4] 2> at
>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
>> [junit4] 2>
>> [junit4] 2> 750036 T3044 oascc.ZkStateReader$3.process WARN
>> ZooKeeper watch triggered, but Solr cannot talk to ZK
>> [junit4] 2> 750037 T3044 oasc.LeaderElector$ElectionWatcher.process
>> WARN org.apache.zookeeper.KeeperException$SessionExpiredException:
>> KeeperErrorCode = Session expired for /overseer_elect/election
>> [junit4] 2> at
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
>> [junit4] 2> at
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>> [junit4] 2> at
>> org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
>> [junit4] 2> at
>> org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
>> [junit4] 2> at
>> org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
>> [junit4] 2> at
>> org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
>> [junit4] 2> at
>> org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
>> [junit4] 2> at
>> org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
>> [junit4] 2> at
>> org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
>> [junit4] 2> at
>> org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:358)
>> [junit4] 2> at
>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
>> [junit4] 2> at
>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
>> [junit4] 2>
>> [junit4] 2> 750038 T3044 oascc.ZkStateReader$2.process A cluster
>> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
>> [junit4] 2> 750038 T3044 oascc.ZkStateReader$2.process WARN
>> ZooKeeper watch triggered, but Solr cannot talk to ZK
>> [junit4] 2> 750121 T3116 oascc.ZkStateReader$3.process WARN
>> ZooKeeper watch triggered, but Solr cannot talk to ZK
>> [junit4] 2> 750206 T2787 oascc.ZkStateReader$3.process WARN
>> ZooKeeper watch triggered, but Solr cannot talk to ZK
>> [junit4] 2> 750206 T2787
>> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
>> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>> [junit4] 2> 750207 T2787
>> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
>> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>> [junit4] 2> 750207 T2787
>> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
>> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>> [junit4] 2> 750207 T2787
>> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
>> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>> [junit4] 2> 750206 T3156
>> oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not
>> read the data org.apache.zookeeper.KeeperException$SessionExpiredException:
>> KeeperErrorCode = Session expired for /overseer_elect/leader
>> [junit4] 2> at
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
>> [junit4] 2> at
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>> [junit4] 2> at
>> org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
>> [junit4] 2> at
>> org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
>> [junit4] 2> at
>> org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
>> [junit4] 2> at
>> org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
>> [junit4] 2> at
>> org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
>> [junit4] 2> at
>> org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:316)
>> [junit4] 2> at
>> org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
>> [junit4] 2> at
>> org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:288)
>> [junit4] 2>
>> [junit4] 2> NOTE: leaving temporary files on disk at:
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-A819205877C572F6-001
>> [junit4] 2> NOTE: test params are: codec=Lucene41,
>> sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=ar_JO,
>> timezone=Africa/Ndjamena
>> [junit4] 2> NOTE: Linux 3.8.0-42-generic i386/Oracle Corporation
>> 1.8.0_05 (32-bit)/cpus=8,threads=1,free=118175464,total=317980672
>> [junit4] 2> NOTE: All tests run in this JVM: [TestRecovery,
>> OpenCloseCoreStressTest, HardAutoCommitTest, DeleteReplicaTest,
>> IndexSchemaRuntimeFieldTest, TestXIncludeConfig, TestFreeTextSuggestions,
>> CollectionsAPIDistributedZkTest, TestFieldTypeResource, BadCopyFieldTest,
>> CopyFieldTest, TestCSVLoader, UniqFieldsUpdateProcessorFactoryTest,
>> TestIndexingPerformance, TestDocumentBuilder, AutoCommitTest,
>> AtomicUpdatesTest, TestQueryTypes, TestCollapseQParserPlugin,
>> BasicDistributedZk2Test, TestWriterPerf, TestCoreContainer,
>> TestComplexPhraseQParserPlugin, TestDistribDocBasedVersion,
>> TestFastLRUCache, TestDynamicFieldResource, DOMUtilTest,
>> QueryResultKeyTest, DistribDocExpirationUpdateProcessorTest,
>> UpdateRequestProcessorFactoryTest, TimeZoneUtilsTest,
>> SchemaVersionSpecificBehaviorTest, PreAnalyzedUpdateProcessorTest,
>> SOLR749Test, TestNonDefinedSimilarityFactory, TestFileDictionaryLookup,
>> MultiThreadedOCPTest, TestRTGBase, QueryParsingTest, OutputWriterTest,
>> LoggingHandlerTest, TestInfoStreamLogging, TestPseudoReturnFields,
>> TestFastWriter, TestReplicationHandlerBackup, AssignTest,
>> TermVectorComponentTest, SolrRequestParserTest, TestPerFieldSimilarity,
>> TestAtomicUpdateErrorCases, TestManagedResource,
>> TermVectorComponentDistributedTest, TestReRankQParserPlugin,
>> TestSerializedLuceneMatchVersion, TestChildDocTransformer,
>> TestNoOpRegenerator, TestGroupingSearch, TestFieldTypeCollectionResource,
>> TestBinaryField, StatsComponentTest, CursorMarkTest, BasicZkTest,
>> SuggesterFSTTest, BJQParserTest, CurrencyFieldXmlFileTest,
>> DistributedQueryElevationComponentTest, TestDistributedMissingSort,
>> SearchHandlerTest, TestLFUCache, TestSolrJ, FileBasedSpellCheckerTest,
>> TestRealTimeGet, RequestHandlersTest, AnalysisAfterCoreReloadTest,
>> SolrInfoMBeanTest, TestDistributedGrouping, TestExtendedDismaxParser,
>> URLClassifyProcessorTest, SuggestComponentTest, ChaosMonkeySafeLeaderTest]
>> [junit4] Completed on J1 in 74.71s, 1 test, 1 failure <<< FAILURES!
>>
>> [...truncated 59 lines...]
>> BUILD FAILED
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:467: The
>> following error occurred while executing this line:
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:447: The
>> following error occurred while executing this line:
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:45: The
>> following error occurred while executing this line:
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37:
>> The following error occurred while executing this line:
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The
>> following error occurred while executing this line:
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:496:
>> The following error occurred while executing this line:
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1299:
>> The following error occurred while executing this line:
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:923:
>> There were test failures: 405 suites, 1656 tests, 3 suite-level errors, 1
>> error, 1 failure, 954 ignored (13 assumptions)
>>
>> Total time: 65 minutes 33 seconds
>> Build step 'Invoke Ant' marked build as failure
>> [description-setter] Description set: Java: 32bit/jdk1.8.0_05 -client
>> -XX:+UseParallelGC
>> Archiving artifacts
>> Recording test results
>> Email was triggered for: Failure - Any
>> Sending email for trigger: Failure - Any
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
>
>
--
Regards,
Shalin Shekhar Mangar.
Re: [JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.8.0_05) - Build
# 10616 - Still Failing!
Posted by Mark Miller <ma...@gmail.com>.
On July 7, 2014 at 11:39:31 AM, Shalin Shekhar Mangar (shalinmangar@gmail.com) wrote:
I was looking into the OverseerTest failure and the following caught my attention.
This error can be due to a slow machine but it also happens on fast machines if you try to do a lot of writes very fast on ZooKeeper which is what the testShardLeaderChange does. Perhaps we should add a small wait between operations?
It’s probably worth a try. I tried once to work around it via ZooKeeper configuration options, thought there was a promising path or two after some googling, but I don’t think I ended up getting anywhere.
- Mark
[junit4] 2> 472241 T2893 oazsp.FileTxnLog.commit WARN fsync-ing the write ahead log in SyncThread:0 took 11588ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
On Sun, Jul 6, 2014 at 10:46 AM, Policeman Jenkins Server <je...@thetaphi.de> wrote:
Build:Â http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/10616/
Java: 32bit/jdk1.8.0_05 -client -XX:+UseParallelGC
5 tests failed.
FAILED: Â junit.framework.TestSuite.org.apache.solr.cloud.OverseerTest
Error Message:
ERROR: SolrZkClient opens=165 closes=164
Stack Trace:
java.lang.AssertionError: ERROR: SolrZkClient opens=165 closes=164
    at __randomizedtesting.SeedInfo.seed([A819205877C572F6]:0)
    at org.junit.Assert.fail(Assert.java:93)
    at org.apache.solr.SolrTestCaseJ4.endTrackingZkClients(SolrTestCaseJ4.java:436)
    at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:182)
    at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:483)
    at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:790)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
    at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
    at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
    at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
    at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
    at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
    at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
    at java.lang.Thread.run(Thread.java:745)
FAILED: Â junit.framework.TestSuite.org.apache.solr.cloud.OverseerTest
Error Message:
2 threads leaked from SUITE scope at org.apache.solr.cloud.OverseerTest: Â Â 1) Thread[id=2969, name=TEST-OverseerTest.testShardLeaderChange-seed#[A819205877C572F6]-SendThread(127.0.0.1:33880), state=TIMED_WAITING, group=TGRP-OverseerTest] Â Â Â Â at java.lang.Thread.sleep(Native Method) Â Â Â Â at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101) Â Â Â Â at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940) Â Â Â Â at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003) Â Â 2) Thread[id=2970, name=TEST-OverseerTest.testShardLeaderChange-seed#[A819205877C572F6]-EventThread, state=WAITING, group=TGRP-OverseerTest] Â Â Â Â at sun.misc.Unsafe.park(Native Method) Â Â Â Â at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) Â Â Â Â at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) Â Â Â Â at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) Â Â Â Â at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.OverseerTest:
  1) Thread[id=2969, name=TEST-OverseerTest.testShardLeaderChange-seed#[A819205877C572F6]-SendThread(127.0.0.1:33880), state=TIMED_WAITING, group=TGRP-OverseerTest]
    at java.lang.Thread.sleep(Native Method)
    at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
    at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
  2) Thread[id=2970, name=TEST-OverseerTest.testShardLeaderChange-seed#[A819205877C572F6]-EventThread, state=WAITING, group=TGRP-OverseerTest]
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
    at __randomizedtesting.SeedInfo.seed([A819205877C572F6]:0)
FAILED: Â junit.framework.TestSuite.org.apache.solr.cloud.OverseerTest
Error Message:
There are still zombie threads that couldn't be terminated: Â Â 1) Thread[id=2969, name=TEST-OverseerTest.testShardLeaderChange-seed#[A819205877C572F6]-SendThread(127.0.0.1:33880), state=TIMED_WAITING, group=TGRP-OverseerTest] Â Â Â Â at java.lang.Thread.sleep(Native Method) Â Â Â Â at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101) Â Â Â Â at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940) Â Â Â Â at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
  1) Thread[id=2969, name=TEST-OverseerTest.testShardLeaderChange-seed#[A819205877C572F6]-SendThread(127.0.0.1:33880), state=TIMED_WAITING, group=TGRP-OverseerTest]
    at java.lang.Thread.sleep(Native Method)
    at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
    at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
    at __randomizedtesting.SeedInfo.seed([A819205877C572F6]:0)
REGRESSION: Â org.apache.solr.cloud.OverseerTest.testShardLeaderChange
Error Message:
KeeperErrorCode = NodeExists for /live_nodes/node1
Stack Trace:
org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /live_nodes/node1
    at __randomizedtesting.SeedInfo.seed([A819205877C572F6:764AA7AF6D5D8707]:0)
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
    at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
    at org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:432)
    at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
    at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:429)
    at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:386)
    at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:373)
    at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:350)
    at org.apache.solr.cloud.OverseerTest$MockZKController.<init>(OverseerTest.java:85)
    at org.apache.solr.cloud.OverseerTest.testShardLeaderChange(OverseerTest.java:746)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:483)
    at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
    at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
    at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
    at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
    at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
    at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
    at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
    at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
    at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
    at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
    at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
    at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
    at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
    at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
    at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
    at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
    at java.lang.Thread.run(Thread.java:745)
FAILED: Â org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch
Error Message:
shard2 is not consistent.  Got 153 from http://127.0.0.1:51995/collection1lastClient and got 132 from http://127.0.0.1:36739/collection1
Stack Trace:
java.lang.AssertionError: shard2 is not consistent.  Got 153 from http://127.0.0.1:51995/collection1lastClient and got 132 from http://127.0.0.1:36739/collection1
    at __randomizedtesting.SeedInfo.seed([A819205877C572F6:29FFAE40009A12CA]:0)
    at org.junit.Assert.fail(Assert.java:93)
    at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1139)
    at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1118)
    at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:150)
    at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:483)
    at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
    at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
    at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
    at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
    at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
    at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
    at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
    at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
    at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
    at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
    at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
    at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
    at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
    at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
    at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
    at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
    at java.lang.Thread.run(Thread.java:745)
Build Log:
[...truncated 11133 lines...]
  [junit4] Suite: org.apache.solr.cloud.OverseerTest
  [junit4]  2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.OverseerTest-A819205877C572F6-001/init-core-data-001
  [junit4]  2> 443102 T2522 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
  [junit4]  2> 443103 T2522 oas.SolrTestCaseJ4.initCore ####initCore
  [junit4]  2> 443103 T2522 oas.SolrTestCaseJ4.initCore ####initCore end
  [junit4]  2> 443107 T2522 oas.SolrTestCaseJ4.setUp ###Starting testStateChange
  [junit4]  2> 443107 T2522 oasc.ZkTestServer.run STARTING ZK TEST SERVER
  [junit4]  1> client port:0.0.0.0/0.0.0.0:0
  [junit4]  2> 443108 T2523 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
  [junit4]  2> 443208 T2522 oasc.ZkTestServer.run start zk server on port:45124
  [junit4]  2> 443209 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 443282 T2529 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ed72a5 name:ZooKeeperConnection Watcher:127.0.0.1:45124/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 443283 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 443284 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 443287 T2531 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6366c4 name:ZooKeeperConnection Watcher:127.0.0.1:45124 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 443287 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 443290 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 443292 T2533 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@60240b name:ZooKeeperConnection Watcher:127.0.0.1:45124 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 443292 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 443293 T2522 oascc.SolrZkClient.makePath makePath: /solr
  [junit4]  2> 443298 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes
  [junit4]  2> 443301 T2522 oascc.SolrZkClient.makePath makePath: /clusterstate.json
  [junit4]  2> 443304 T2522 oascc.SolrZkClient.makePath makePath: /aliases.json
  [junit4]  2> 443306 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
  [junit4]  2> 443311 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 443312 T2535 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@68e786 name:ZooKeeperConnection Watcher:127.0.0.1:45124/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 443313 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 443314 T2522 oascc.SolrZkClient.makePath makePath: /overseer_elect
  [junit4]  2> 443318 T2522 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
  [junit4]  2> 443324 T2522 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:45124_solr
  [junit4]  2> 443324 T2522 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
  [junit4]  2> 443328 T2522 oasc.Overseer.start Overseer (id=92053395863699459-127.0.0.1:45124_solr-n_0000000000) starting
  [junit4]  2> 443331 T2522 oascc.SolrZkClient.makePath makePath: /overseer/queue
  [junit4]  2> 443336 T2522 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
  [junit4]  2> 443340 T2522 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
  [junit4]  2> 443345 T2522 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
  [junit4]  2> 443350 T2522 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
  [junit4]  2> 443357 T2522 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
  [junit4]  2> 443365 T2537 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
  [junit4]  2> 443366 T2536 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
  [junit4]  2> 443368 T2522 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
  [junit4]  2> 443369 T2536 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
  [junit4]  2> 443370 T2536 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
  [junit4]  2>      "operation":"state",
  [junit4]  2>      "base_url":"http://127.0.0.1/solr",
  [junit4]  2>      "node_name":"node1",
  [junit4]  2>      "collection":"collection1",
  [junit4]  2>      "core":"core1",
  [junit4]  2>      "roles":"",
  [junit4]  2>      "state":"recovering"}
  [junit4]  2> 443370 T2536 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
  [junit4]  2> 443374 T2535 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
  [junit4]  2> 443375 T2529 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
  [junit4]  2> 443420 T2522 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
  [junit4]  2> 443422 T2535 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
  [junit4]  2> 443424 T2536 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
  [junit4]  2> 443425 T2536 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
  [junit4]  2>      "operation":"state",
  [junit4]  2>      "base_url":"http://127.0.0.1/solr",
  [junit4]  2>      "node_name":"node1",
  [junit4]  2>      "collection":"collection1",
  [junit4]  2>      "core":"core1",
  [junit4]  2>      "roles":"",
  [junit4]  2>      "state":"active"}
  [junit4]  2> 443426 T2536 oasc.Overseer$ClusterStateUpdater.updateState node=core_node1 is already registered
  [junit4]  2> 443426 T2536 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
  [junit4]  2> 443431 T2535 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
  [junit4]  2> 443534 T2529 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
  [junit4]  2> 443575 T2522 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:45124 45124
  [junit4]  2> 443988 T2523 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:45124 45124
  [junit4]  2> 443991 T2522 oas.SolrTestCaseJ4.tearDown ###Ending testStateChange
  [junit4]  2> 443992 T2536 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:45124_solr
  [junit4]  2> 443994 T2538 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
  [junit4]  2>     at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
  [junit4]  2>     at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
  [junit4]  2>     at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
  [junit4]  2>     at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
  [junit4]  2>     at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
  [junit4]  2>     at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
  [junit4]  2>     at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
  [junit4]  2>     at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:316)
  [junit4]  2>     at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
  [junit4]  2>     at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:288)
  [junit4]  2>
  [junit4]  2> 444000 T2522 oas.SolrTestCaseJ4.setUp ###Starting testShardAssignment
  [junit4]  2> 444001 T2522 oasc.ZkTestServer.run STARTING ZK TEST SERVER
  [junit4]  1> client port:0.0.0.0/0.0.0.0:0
  [junit4]  2> 444002 T2539 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
  [junit4]  2> 444101 T2522 oasc.ZkTestServer.run start zk server on port:57742
  [junit4]  2> 444103 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 444107 T2545 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@9fc604 name:ZooKeeperConnection Watcher:127.0.0.1:57742 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 444108 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 444111 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 444112 T2547 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@104f2e0 name:ZooKeeperConnection Watcher:127.0.0.1:57742 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 444113 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 444114 T2522 oascc.SolrZkClient.makePath makePath: /solr
  [junit4]  2> 444119 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 444121 T2549 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@896251 name:ZooKeeperConnection Watcher:127.0.0.1:57742/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 444121 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 444122 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes
  [junit4]  2> 444125 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 444127 T2551 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@10c3e42 name:ZooKeeperConnection Watcher:127.0.0.1:57742/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 444127 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 444128 T2522 oascc.SolrZkClient.makePath makePath: /overseer_elect
  [junit4]  2> 444132 T2522 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
  [junit4]  2> 444137 T2522 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:57742_solr
  [junit4]  2> 444138 T2522 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
  [junit4]  2> 444141 T2522 oasc.Overseer.start Overseer (id=92053395922223107-127.0.0.1:57742_solr-n_0000000000) starting
  [junit4]  2> 444144 T2522 oascc.SolrZkClient.makePath makePath: /overseer/queue
  [junit4]  2> 444149 T2522 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
  [junit4]  2> 444153 T2522 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
  [junit4]  2> 444157 T2522 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
  [junit4]  2> 444162 T2522 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
  [junit4]  2> 444166 T2522 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
  [junit4]  2> 444173 T2553 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
  [junit4]  2> 444175 T2522 oascc.SolrZkClient.makePath makePath: /clusterstate.json
  [junit4]  2> 444175 T2552 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
  [junit4]  2> 444177 T2522 oascc.SolrZkClient.makePath makePath: /aliases.json
  [junit4]  2> 444179 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
  [junit4]  2> 444183 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 444185 T2555 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1196b81 name:ZooKeeperConnection Watcher:127.0.0.1:57742/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 444185 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 444187 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
  [junit4]  2> 444190 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1
  [junit4]  2> 444194 T2549 oascc.ZkStateReader$3.process Updating live nodes... (1)
  [junit4]  2> 444194 T2555 oascc.ZkStateReader$3.process Updating live nodes... (1)
  [junit4]  2> 444197 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
  [junit4]  2> 444199 T2552 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
  [junit4]  2> 444199 T2552 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
  [junit4]  2>      "operation":"state",
  [junit4]  2>      "state":"active",
  [junit4]  2>      "node_name":"127.0.0.1",
  [junit4]  2>      "core":"core1",
  [junit4]  2>      "core_node_name":"node1",
  [junit4]  2>      "collection":"collection1",
  [junit4]  2>      "numShards":"3",
  [junit4]  2>      "base_url":"http://127.0.0.1/solr/"}
  [junit4]  2> 444200 T2552 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
  [junit4]  2> 444200 T2552 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
  [junit4]  2> 444204 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
  [junit4]  2> 444205 T2555 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
  [junit4]  2> 444205 T2549 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
  [junit4]  2> 444697 T2522 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
  [junit4]  2> 444712 T2522 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
  [junit4]  2> 444719 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
  [junit4]  2> 444721 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
  [junit4]  2> 444721 T2552 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
  [junit4]  2> 444727 T2552 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
  [junit4]  2>      "operation":"state",
  [junit4]  2>      "state":"active",
  [junit4]  2>      "node_name":"127.0.0.1",
  [junit4]  2>      "core":"core2",
  [junit4]  2>      "core_node_name":"node2",
  [junit4]  2>      "collection":"collection1",
  [junit4]  2>      "numShards":"3",
  [junit4]  2>      "base_url":"http://127.0.0.1/solr/"}
  [junit4]  2> 444728 T2552 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
  [junit4]  2> 444728 T2552 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
  [junit4]  2> 444731 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
  [junit4]  2> 444834 T2555 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
  [junit4]  2> 444834 T2549 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
  [junit4]  2> 445222 T2522 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
  [junit4]  2> 445233 T2522 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
  [junit4]  2> 445240 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
  [junit4]  2> 445240 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
  [junit4]  2> 445241 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
  [junit4]  2> 445242 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
  [junit4]  2> 445242 T2552 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
  [junit4]  2> 445248 T2552 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
  [junit4]  2>      "operation":"state",
  [junit4]  2>      "state":"active",
  [junit4]  2>      "node_name":"127.0.0.1",
  [junit4]  2>      "core":"core3",
  [junit4]  2>      "core_node_name":"node3",
  [junit4]  2>      "collection":"collection1",
  [junit4]  2>      "numShards":"3",
  [junit4]  2>      "base_url":"http://127.0.0.1/solr/"}
  [junit4]  2> 445248 T2552 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
  [junit4]  2> 445248 T2552 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
  [junit4]  2> 445251 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
  [junit4]  2> 445354 T2549 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
  [junit4]  2> 445354 T2555 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
  [junit4]  2> 445742 T2522 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
  [junit4]  2> 445753 T2522 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
  [junit4]  2> 445764 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
  [junit4]  2> 445765 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
  [junit4]  2> 445765 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
  [junit4]  2> 445766 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
  [junit4]  2> 445767 T2552 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
  [junit4]  2> 445771 T2552 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
  [junit4]  2>      "operation":"state",
  [junit4]  2>      "state":"active",
  [junit4]  2>      "node_name":"127.0.0.1",
  [junit4]  2>      "core":"core4",
  [junit4]  2>      "core_node_name":"node4",
  [junit4]  2>      "collection":"collection1",
  [junit4]  2>      "numShards":"3",
  [junit4]  2>      "base_url":"http://127.0.0.1/solr/"}
  [junit4]  2> 445771 T2552 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
  [junit4]  2> 445771 T2552 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
  [junit4]  2> 445773 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
  [junit4]  2> 445876 T2555 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
  [junit4]  2> 445876 T2549 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
  [junit4]  2> 446267 T2522 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
  [junit4]  2> 446276 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
  [junit4]  2> 446276 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
  [junit4]  2> 446276 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
  [junit4]  2> 446278 T2552 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
  [junit4]  2> 446279 T2552 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
  [junit4]  2>      "operation":"state",
  [junit4]  2>      "state":"active",
  [junit4]  2>      "node_name":"127.0.0.1",
  [junit4]  2>      "core":"core5",
  [junit4]  2>      "core_node_name":"node5",
  [junit4]  2>      "collection":"collection1",
  [junit4]  2>      "numShards":"3",
  [junit4]  2>      "base_url":"http://127.0.0.1/solr/"}
  [junit4]  2> 446280 T2552 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
  [junit4]  2> 446280 T2552 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
  [junit4]  2> 446283 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
  [junit4]  2> 446386 T2555 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
  [junit4]  2> 446386 T2549 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
  [junit4]  2> 446776 T2522 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
  [junit4]  2> 446787 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
  [junit4]  2> 446788 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
  [junit4]  2> 446788 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
  [junit4]  2> 446790 T2552 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
  [junit4]  2> 446791 T2552 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
  [junit4]  2>      "operation":"state",
  [junit4]  2>      "state":"active",
  [junit4]  2>      "node_name":"127.0.0.1",
  [junit4]  2>      "core":"core6",
  [junit4]  2>      "core_node_name":"node6",
  [junit4]  2>      "collection":"collection1",
  [junit4]  2>      "numShards":"3",
  [junit4]  2>      "base_url":"http://127.0.0.1/solr/"}
  [junit4]  2> 446791 T2552 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
  [junit4]  2> 446791 T2552 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
  [junit4]  2> 446795 T2551 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
  [junit4]  2> 446899 T2549 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
  [junit4]  2> 446899 T2555 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
  [junit4]  2> 447288 T2522 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
  [junit4]  2> 447298 T2555 oascc.ZkStateReader$3.process Updating live nodes... (0)
  [junit4]  2> 447301 T2522 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:57742 57742
  [junit4]  2> 447869 T2539 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:57742 57742
  [junit4]  2> 447871 T2522 oas.SolrTestCaseJ4.tearDown ###Ending testShardAssignment
  [junit4]  2> 447872 T2552 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:57742_solr
  [junit4]  2> 447875 T2556 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
  [junit4]  2>     at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
  [junit4]  2>     at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
  [junit4]  2>     at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
  [junit4]  2>     at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
  [junit4]  2>     at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
  [junit4]  2>     at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
  [junit4]  2>     at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
  [junit4]  2>     at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:316)
  [junit4]  2>     at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
  [junit4]  2>     at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:288)
  [junit4]  2>
  [junit4]  2> 447881 T2522 oas.SolrTestCaseJ4.setUp ###Starting testShardAssignmentBigger
  [junit4]  2> 447882 T2522 oasc.ZkTestServer.run STARTING ZK TEST SERVER
  [junit4]  1> client port:0.0.0.0/0.0.0.0:0
  [junit4]  2> 447883 T2557 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
  [junit4]  2> 447983 T2522 oasc.ZkTestServer.run start zk server on port:50146
  [junit4]  2> 447984 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 447988 T2563 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@daa195 name:ZooKeeperConnection Watcher:127.0.0.1:50146 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 447989 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 447993 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 447994 T2565 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f8aac name:ZooKeeperConnection Watcher:127.0.0.1:50146 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 447995 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 447995 T2522 oascc.SolrZkClient.makePath makePath: /solr
  [junit4]  2> 448000 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 448002 T2567 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@141291c name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 448002 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 448003 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes
  [junit4]  2> 448006 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 448008 T2569 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@49f99d name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 448008 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 448009 T2522 oascc.SolrZkClient.makePath makePath: /overseer_elect
  [junit4]  2> 448012 T2522 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
  [junit4]  2> 448018 T2522 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:50146_solr
  [junit4]  2> 448018 T2522 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
  [junit4]  2> 448021 T2522 oasc.Overseer.start Overseer (id=92053396176633859-127.0.0.1:50146_solr-n_0000000000) starting
  [junit4]  2> 448024 T2522 oascc.SolrZkClient.makePath makePath: /overseer/queue
  [junit4]  2> 448029 T2522 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
  [junit4]  2> 448034 T2522 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
  [junit4]  2> 448038 T2522 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
  [junit4]  2> 448043 T2522 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
  [junit4]  2> 448048 T2522 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
  [junit4]  2> 448055 T2571 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
  [junit4]  2> 448056 T2522 oascc.SolrZkClient.makePath makePath: /clusterstate.json
  [junit4]  2> 448056 T2570 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
  [junit4]  2> 448058 T2522 oascc.SolrZkClient.makePath makePath: /aliases.json
  [junit4]  2> 448061 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
  [junit4]  2> 448065 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 448066 T2573 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1242e08 name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 448067 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 448069 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
  [junit4]  2> 448072 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node0
  [junit4]  2> 448076 T2573 oascc.ZkStateReader$3.process Updating live nodes... (1)
  [junit4]  2> 448076 T2567 oascc.ZkStateReader$3.process Updating live nodes... (1)
  [junit4]  2> 448077 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 448078 T2575 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1fc812a name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 448078 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 448080 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
  [junit4]  2> 448083 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
  [junit4]  2> 448087 T2573 oascc.ZkStateReader$3.process Updating live nodes... (2)
  [junit4]  2> 448087 T2567 oascc.ZkStateReader$3.process Updating live nodes... (2)
  [junit4]  2> 448087 T2575 oascc.ZkStateReader$3.process Updating live nodes... (2)
  [junit4]  2> 448088 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 448089 T2577 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1714ec0 name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 448090 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 448091 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
  [junit4]  2> 448095 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node2
  [junit4]  2> 448099 T2573 oascc.ZkStateReader$3.process Updating live nodes... (3)
  [junit4]  2> 448099 T2577 oascc.ZkStateReader$3.process Updating live nodes... (3)
  [junit4]  2> 448099 T2575 oascc.ZkStateReader$3.process Updating live nodes... (3)
  [junit4]  2> 448100 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 448099 T2567 oascc.ZkStateReader$3.process Updating live nodes... (3)
  [junit4]  2> 448102 T2579 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@fbc5e2 name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 448102 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 448104 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
  [junit4]  2> 448107 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node3
  [junit4]  2> 448112 T2577 oascc.ZkStateReader$3.process Updating live nodes... (4)
  [junit4]  2> 448112 T2575 oascc.ZkStateReader$3.process Updating live nodes... (4)
  [junit4]  2> 448112 T2567 oascc.ZkStateReader$3.process Updating live nodes... (4)
  [junit4]  2> 448112 T2573 oascc.ZkStateReader$3.process Updating live nodes... (4)
  [junit4]  2> 448113 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 448113 T2579 oascc.ZkStateReader$3.process Updating live nodes... (4)
  [junit4]  2> 448115 T2581 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@75e1db name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 448115 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 448117 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
  [junit4]  2> 448121 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node4
  [junit4]  2> 448126 T2577 oascc.ZkStateReader$3.process Updating live nodes... (5)
  [junit4]  2> 448126 T2575 oascc.ZkStateReader$3.process Updating live nodes... (5)
  [junit4]  2> 448126 T2573 oascc.ZkStateReader$3.process Updating live nodes... (5)
  [junit4]  2> 448127 T2581 oascc.ZkStateReader$3.process Updating live nodes... (5)
  [junit4]  2> 448126 T2567 oascc.ZkStateReader$3.process Updating live nodes... (5)
  [junit4]  2> 448126 T2579 oascc.ZkStateReader$3.process Updating live nodes... (5)
  [junit4]  2> 448127 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 448129 T2583 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@11dc02e name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 448130 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 448132 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
  [junit4]  2> 448136 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node5
  [junit4]  2> 448139 T2575 oascc.ZkStateReader$3.process Updating live nodes... (6)
  [junit4]  2> 448140 T2577 oascc.ZkStateReader$3.process Updating live nodes... (6)
  [junit4]  2> 448140 T2581 oascc.ZkStateReader$3.process Updating live nodes... (6)
  [junit4]  2> 448140 T2573 oascc.ZkStateReader$3.process Updating live nodes... (6)
  [junit4]  2> 448140 T2579 oascc.ZkStateReader$3.process Updating live nodes... (6)
  [junit4]  2> 448141 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 448141 T2583 oascc.ZkStateReader$3.process Updating live nodes... (6)
  [junit4]  2> 448140 T2567 oascc.ZkStateReader$3.process Updating live nodes... (6)
  [junit4]  2> 448143 T2585 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6d9f03 name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 448143 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 448145 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
  [junit4]  2> 448148 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node6
  [junit4]  2> 448153 T2575 oascc.ZkStateReader$3.process Updating live nodes... (7)
  [junit4]  2> 448153 T2577 oascc.ZkStateReader$3.process Updating live nodes... (7)
  [junit4]  2> 448153 T2573 oascc.ZkStateReader$3.process Updating live nodes... (7)
  [junit4]  2> 448153 T2585 oascc.ZkStateReader$3.process Updating live nodes... (7)
  [junit4]  2> 448153 T2583 oascc.ZkStateReader$3.process Updating live nodes... (7)
  [junit4]  2> 448153 T2579 oascc.ZkStateReader$3.process Updating live nodes... (7)
  [junit4]  2> 448153 T2581 oascc.ZkStateReader$3.process Updating live nodes... (7)
  [junit4]  2> 448153 T2567 oascc.ZkStateReader$3.process Updating live nodes... (7)
  [junit4]  2> 448154 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 448156 T2587 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@958679 name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 448157 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 448159 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
  [junit4]  2> 448162 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node7
  [junit4]  2> 448167 T2567 oascc.ZkStateReader$3.process Updating live nodes... (8)
  [junit4]  2> 448167 T2577 oascc.ZkStateReader$3.process Updating live nodes... (8)
  [junit4]  2> 448168 T2585 oascc.ZkStateReader$3.process Updating live nodes... (8)
  [junit4]  2> 448168 T2583 oascc.ZkStateReader$3.process Updating live nodes... (8)
  [junit4]  2> 448168 T2581 oascc.ZkStateReader$3.process Updating live nodes... (8)
  [junit4]  2> 448168 T2579 oascc.ZkStateReader$3.process Updating live nodes... (8)
  [junit4]  2> 448168 T2573 oascc.ZkStateReader$3.process Updating live nodes... (8)
  [junit4]  2> 448167 T2587 oascc.ZkStateReader$3.process Updating live nodes... (8)
  [junit4]  2> 448167 T2575 oascc.ZkStateReader$3.process Updating live nodes... (8)
  [junit4]  2> 448169 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 448171 T2589 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@169d949 name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 448171 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 448174 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
  [junit4]  2> 448177 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node8
  [junit4]  2> 448182 T2589 oascc.ZkStateReader$3.process Updating live nodes... (9)
  [junit4]  2> 448183 T2585 oascc.ZkStateReader$3.process Updating live nodes... (9)
  [junit4]  2> 448183 T2583 oascc.ZkStateReader$3.process Updating live nodes... (9)
  [junit4]  2> 448183 T2587 oascc.ZkStateReader$3.process Updating live nodes... (9)
  [junit4]  2> 448183 T2573 oascc.ZkStateReader$3.process Updating live nodes... (9)
  [junit4]  2> 448183 T2579 oascc.ZkStateReader$3.process Updating live nodes... (9)
  [junit4]  2> 448183 T2581 oascc.ZkStateReader$3.process Updating live nodes... (9)
  [junit4]  2> 448183 T2575 oascc.ZkStateReader$3.process Updating live nodes... (9)
  [junit4]  2> 448183 T2577 oascc.ZkStateReader$3.process Updating live nodes... (9)
  [junit4]  2> 448182 T2567 oascc.ZkStateReader$3.process Updating live nodes... (9)
  [junit4]  2> 448185 T2591 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2e78fa name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 448184 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 448188 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 448190 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
  [junit4]  2> 448192 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node9
  [junit4]  2> 448195 T2589 oascc.ZkStateReader$3.process Updating live nodes... (10)
  [junit4]  2> 448195 T2577 oascc.ZkStateReader$3.process Updating live nodes... (10)
  [junit4]  2> 448195 T2567 oascc.ZkStateReader$3.process Updating live nodes... (10)
  [junit4]  2> 448196 T2573 oascc.ZkStateReader$3.process Updating live nodes... (10)
  [junit4]  2> 448195 T2575 oascc.ZkStateReader$3.process Updating live nodes... (10)
  [junit4]  2> 448195 T2587 oascc.ZkStateReader$3.process Updating live nodes... (10)
  [junit4]  2> 448195 T2579 oascc.ZkStateReader$3.process Updating live nodes... (10)
  [junit4]  2> 448195 T2583 oascc.ZkStateReader$3.process Updating live nodes... (10)
  [junit4]  2> 448196 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 448198 T2593 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@156c3b name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 448196 T2591 oascc.ZkStateReader$3.process Updating live nodes... (10)
  [junit4]  2> 448196 T2585 oascc.ZkStateReader$3.process Updating live nodes... (10)
  [junit4]  2> 448196 T2581 oascc.ZkStateReader$3.process Updating live nodes... (10)
  [junit4]  2> 448198 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 448200 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
  [junit4]  2> 448203 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node10
  [junit4]  2> 448207 T2575 oascc.ZkStateReader$3.process Updating live nodes... (11)
  [junit4]  2> 448207 T2583 oascc.ZkStateReader$3.process Updating live nodes... (11)
  [junit4]  2> 448208 T2579 oascc.ZkStateReader$3.process Updating live nodes... (11)
  [junit4]  2> 448207 T2577 oascc.ZkStateReader$3.process Updating live nodes... (11)
  [junit4]  2> 448207 T2573 oascc.ZkStateReader$3.process Updating live nodes... (11)
  [junit4]  2> 448207 T2589 oascc.ZkStateReader$3.process Updating live nodes... (11)
  [junit4]  2> 448207 T2587 oascc.ZkStateReader$3.process Updating live nodes... (11)
  [junit4]  2> 448209 T2585 oascc.ZkStateReader$3.process Updating live nodes... (11)
  [junit4]  2> 448208 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 448208 T2581 oascc.ZkStateReader$3.process Updating live nodes... (11)
  [junit4]  2> 448208 T2593 oascc.ZkStateReader$3.process Updating live nodes... (11)
  [junit4]  2> 448208 T2591 oascc.ZkStateReader$3.process Updating live nodes... (11)
  [junit4]  2> 448208 T2567 oascc.ZkStateReader$3.process Updating live nodes... (11)
  [junit4]  2> 448211 T2595 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f1978e name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 448212 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 448213 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
  [junit4]  2> 448216 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node11
  [junit4]  2> 448220 T2587 oascc.ZkStateReader$3.process Updating live nodes... (12)
  [junit4]  2> 448220 T2567 oascc.ZkStateReader$3.process Updating live nodes... (12)
  [junit4]  2> 448221 T2583 oascc.ZkStateReader$3.process Updating live nodes... (12)
  [junit4]  2> 448220 T2589 oascc.ZkStateReader$3.process Updating live nodes... (12)
  [junit4]  2> 448222 T2595 oascc.ZkStateReader$3.process Updating live nodes... (12)
  [junit4]  2> 448220 T2575 oascc.ZkStateReader$3.process Updating live nodes... (12)
  [junit4]  2> 448222 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 448222 T2593 oascc.ZkStateReader$3.process Updating live nodes... (12)
  [junit4]  2> 448222 T2581 oascc.ZkStateReader$3.process Updating live nodes... (12)
  [junit4]  2> 448224 T2597 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@175be88 name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 448221 T2591 oascc.ZkStateReader$3.process Updating live nodes... (12)
  [junit4]  2> 448221 T2573 oascc.ZkStateReader$3.process Updating live nodes... (12)
  [junit4]  2> 448221 T2577 oascc.ZkStateReader$3.process Updating live nodes... (12)
  [junit4]  2> 448221 T2585 oascc.ZkStateReader$3.process Updating live nodes... (12)
  [junit4]  2> 448221 T2579 oascc.ZkStateReader$3.process Updating live nodes... (12)
  [junit4]  2> 448224 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 448228 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
  [junit4]  2> 448232 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node12
  [junit4]  2> 448236 T2573 oascc.ZkStateReader$3.process Updating live nodes... (13)
  [junit4]  2> 448236 T2577 oascc.ZkStateReader$3.process Updating live nodes... (13)
  [junit4]  2> 448237 T2595 oascc.ZkStateReader$3.process Updating live nodes... (13)
  [junit4]  2> 448237 T2593 oascc.ZkStateReader$3.process Updating live nodes... (13)
  [junit4]  2> 448237 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 448238 T2581 oascc.ZkStateReader$3.process Updating live nodes... (13)
  [junit4]  2> 448238 T2597 oascc.ZkStateReader$3.process Updating live nodes... (13)
  [junit4]  2> 448237 T2583 oascc.ZkStateReader$3.process Updating live nodes... (13)
  [junit4]  2> 448239 T2599 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@dde0f4 name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 448237 T2589 oascc.ZkStateReader$3.process Updating live nodes... (13)
  [junit4]  2> 448237 T2575 oascc.ZkStateReader$3.process Updating live nodes... (13)
  [junit4]  2> 448237 T2587 oascc.ZkStateReader$3.process Updating live nodes... (13)
  [junit4]  2> 448237 T2585 oascc.ZkStateReader$3.process Updating live nodes... (13)
  [junit4]  2> 448236 T2567 oascc.ZkStateReader$3.process Updating live nodes... (13)
  [junit4]  2> 448236 T2579 oascc.ZkStateReader$3.process Updating live nodes... (13)
  [junit4]  2> 448236 T2591 oascc.ZkStateReader$3.process Updating live nodes... (13)
  [junit4]  2> 448240 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 448244 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
  [junit4]  2> 448248 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node13
  [junit4]  2> 448254 T2587 oascc.ZkStateReader$3.process Updating live nodes... (14)
  [junit4]  2> 448255 T2577 oascc.ZkStateReader$3.process Updating live nodes... (14)
  [junit4]  2> 448255 T2579 oascc.ZkStateReader$3.process Updating live nodes... (14)
  [junit4]  2> 448255 T2595 oascc.ZkStateReader$3.process Updating live nodes... (14)
  [junit4]  2> 448255 T2573 oascc.ZkStateReader$3.process Updating live nodes... (14)
  [junit4]  2> 448256 T2597 oascc.ZkStateReader$3.process Updating live nodes... (14)
  [junit4]  2> 448256 T2599 oascc.ZkStateReader$3.process Updating live nodes... (14)
  [junit4]  2> 448256 T2593 oascc.ZkStateReader$3.process Updating live nodes... (14)
  [junit4]  2> 448256 T2585 oascc.ZkStateReader$3.process Updating live nodes... (14)
  [junit4]  2> 448256 T2581 oascc.ZkStateReader$3.process Updating live nodes... (14)
  [junit4]  2> 448256 T2583 oascc.ZkStateReader$3.process Updating live nodes... (14)
  [junit4]  2> 448256 T2567 oascc.ZkStateReader$3.process Updating live nodes... (14)
  [junit4]  2> 448256 T2575 oascc.ZkStateReader$3.process Updating live nodes... (14)
  [junit4]  2> 448256 T2589 oascc.ZkStateReader$3.process Updating live nodes... (14)
  [junit4]  2> 448256 T2591 oascc.ZkStateReader$3.process Updating live nodes... (14)
  [junit4]  2> 448259 T2601 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@19e3760 name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 448257 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 448262 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 448264 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
  [junit4]  2> 448268 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node14
  [junit4]  2> 448273 T2573 oascc.ZkStateReader$3.process Updating live nodes... (15)
  [junit4]  2> 448273 T2587 oascc.ZkStateReader$3.process Updating live nodes... (15)
  [junit4]  2> 448273 T2579 oascc.ZkStateReader$3.process Updating live nodes... (15)
  [junit4]  2> 448273 T2591 oascc.ZkStateReader$3.process Updating live nodes... (15)
  [junit4]  2> 448273 T2577 oascc.ZkStateReader$3.process Updating live nodes... (15)
  [junit4]  2> 448273 T2595 oascc.ZkStateReader$3.process Updating live nodes... (15)
  [junit4]  2> 448275 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 448275 T2599 oascc.ZkStateReader$3.process Updating live nodes... (15)
  [junit4]  2> 448275 T2583 oascc.ZkStateReader$3.process Updating live nodes... (15)
  [junit4]  2> 448275 T2585 oascc.ZkStateReader$3.process Updating live nodes... (15)
  [junit4]  2> 448275 T2597 oascc.ZkStateReader$3.process Updating live nodes... (15)
  [junit4]  2> 448275 T2601 oascc.ZkStateReader$3.process Updating live nodes... (15)
  [junit4]  2> 448275 T2567 oascc.ZkStateReader$3.process Updating live nodes... (15)
  [junit4]  2> 448275 T2575 oascc.ZkStateReader$3.process Updating live nodes... (15)
  [junit4]  2> 448275 T2581 oascc.ZkStateReader$3.process Updating live nodes... (15)
  [junit4]  2> 448274 T2593 oascc.ZkStateReader$3.process Updating live nodes... (15)
  [junit4]  2> 448273 T2589 oascc.ZkStateReader$3.process Updating live nodes... (15)
  [junit4]  2> 448277 T2603 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1818cfa name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 448281 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 448283 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
  [junit4]  2> 448286 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node15
  [junit4]  2> 448291 T2577 oascc.ZkStateReader$3.process Updating live nodes... (16)
  [junit4]  2> 448292 T2573 oascc.ZkStateReader$3.process Updating live nodes... (16)
  [junit4]  2> 448291 T2587 oascc.ZkStateReader$3.process Updating live nodes... (16)
  [junit4]  2> 448293 T2579 oascc.ZkStateReader$3.process Updating live nodes... (16)
  [junit4]  2> 448293 T2575 oascc.ZkStateReader$3.process Updating live nodes... (16)
  [junit4]  2> 448293 T2601 oascc.ZkStateReader$3.process Updating live nodes... (16)
  [junit4]  2> 448294 T2599 oascc.ZkStateReader$3.process Updating live nodes... (16)
  [junit4]  2> 448293 T2591 oascc.ZkStateReader$3.process Updating live nodes... (16)
  [junit4]  2> 448293 T2597 oascc.ZkStateReader$3.process Updating live nodes... (16)
  [junit4]  2> 448293 T2583 oascc.ZkStateReader$3.process Updating live nodes... (16)
  [junit4]  2> 448293 T2567 oascc.ZkStateReader$3.process Updating live nodes... (16)
  [junit4]  2> 448293 T2589 oascc.ZkStateReader$3.process Updating live nodes... (16)
  [junit4]  2> 448295 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 448294 T2603 oascc.ZkStateReader$3.process Updating live nodes... (16)
  [junit4]  2> 448294 T2581 oascc.ZkStateReader$3.process Updating live nodes... (16)
  [junit4]  2> 448294 T2585 oascc.ZkStateReader$3.process Updating live nodes... (16)
  [junit4]  2> 448294 T2595 oascc.ZkStateReader$3.process Updating live nodes... (16)
  [junit4]  2> 448294 T2593 oascc.ZkStateReader$3.process Updating live nodes... (16)
  [junit4]  2> 448297 T2605 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@154e83c name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 448299 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 448300 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
  [junit4]  2> 448304 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node16
  [junit4]  2> 448313 T2587 oascc.ZkStateReader$3.process Updating live nodes... (17)
  [junit4]  2> 448313 T2589 oascc.ZkStateReader$3.process Updating live nodes... (17)
  [junit4]  2> 448314 T2595 oascc.ZkStateReader$3.process Updating live nodes... (17)
  [junit4]  2> 448315 T2575 oascc.ZkStateReader$3.process Updating live nodes... (17)
  [junit4]  2> 448314 T2585 oascc.ZkStateReader$3.process Updating live nodes... (17)
  [junit4]  2> 448316 T2591 oascc.ZkStateReader$3.process Updating live nodes... (17)
  [junit4]  2> 448313 T2573 oascc.ZkStateReader$3.process Updating live nodes... (17)
  [junit4]  2> 448313 T2577 oascc.ZkStateReader$3.process Updating live nodes... (17)
  [junit4]  2> 448313 T2593 oascc.ZkStateReader$3.process Updating live nodes... (17)
  [junit4]  2> 448317 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 448316 T2599 oascc.ZkStateReader$3.process Updating live nodes... (17)
  [junit4]  2> 448316 T2581 oascc.ZkStateReader$3.process Updating live nodes... (17)
  [junit4]  2> 448316 T2605 oascc.ZkStateReader$3.process Updating live nodes... (17)
  [junit4]  2> 448316 T2603 oascc.ZkStateReader$3.process Updating live nodes... (17)
  [junit4]  2> 448316 T2583 oascc.ZkStateReader$3.process Updating live nodes... (17)
  [junit4]  2> 448316 T2567 oascc.ZkStateReader$3.process Updating live nodes... (17)
  [junit4]  2> 448315 T2601 oascc.ZkStateReader$3.process Updating live nodes... (17)
  [junit4]  2> 448315 T2597 oascc.ZkStateReader$3.process Updating live nodes... (17)
  [junit4]  2> 448315 T2579 oascc.ZkStateReader$3.process Updating live nodes... (17)
  [junit4]  2> 448319 T2607 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d963a2 name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 448323 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 448324 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
  [junit4]  2> 448328 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node17
  [junit4]  2> 448333 T2573 oascc.ZkStateReader$3.process Updating live nodes... (18)
  [junit4]  2> 448333 T2577 oascc.ZkStateReader$3.process Updating live nodes... (18)
  [junit4]  2> 448334 T2601 oascc.ZkStateReader$3.process Updating live nodes... (18)
  [junit4]  2> 448334 T2607 oascc.ZkStateReader$3.process Updating live nodes... (18)
  [junit4]  2> 448335 T2567 oascc.ZkStateReader$3.process Updating live nodes... (18)
  [junit4]  2> 448335 T2583 oascc.ZkStateReader$3.process Updating live nodes... (18)
  [junit4]  2> 448335 T2603 oascc.ZkStateReader$3.process Updating live nodes... (18)
  [junit4]  2> 448334 T2589 oascc.ZkStateReader$3.process Updating live nodes... (18)
  [junit4]  2> 448336 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 448334 T2595 oascc.ZkStateReader$3.process Updating live nodes... (18)
  [junit4]  2> 448334 T2579 oascc.ZkStateReader$3.process Updating live nodes... (18)
  [junit4]  2> 448334 T2575 oascc.ZkStateReader$3.process Updating live nodes... (18)
  [junit4]  2> 448338 T2609 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@826a2e name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 448333 T2587 oascc.ZkStateReader$3.process Updating live nodes... (18)
  [junit4]  2> 448333 T2591 oascc.ZkStateReader$3.process Updating live nodes... (18)
  [junit4]  2> 448338 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 448336 T2585 oascc.ZkStateReader$3.process Updating live nodes... (18)
  [junit4]  2> 448335 T2593 oascc.ZkStateReader$3.process Updating live nodes... (18)
  [junit4]  2> 448335 T2599 oascc.ZkStateReader$3.process Updating live nodes... (18)
  [junit4]  2> 448335 T2581 oascc.ZkStateReader$3.process Updating live nodes... (18)
  [junit4]  2> 448335 T2605 oascc.ZkStateReader$3.process Updating live nodes... (18)
  [junit4]  2> 448335 T2597 oascc.ZkStateReader$3.process Updating live nodes... (18)
  [junit4]  2> 448341 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
  [junit4]  2> 448346 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node18
  [junit4]  2> 448350 T2573 oascc.ZkStateReader$3.process Updating live nodes... (19)
  [junit4]  2> 448350 T2577 oascc.ZkStateReader$3.process Updating live nodes... (19)
  [junit4]  2> 448351 T2607 oascc.ZkStateReader$3.process Updating live nodes... (19)
  [junit4]  2> 448352 T2587 oascc.ZkStateReader$3.process Updating live nodes... (19)
  [junit4]  2> 448352 T2593 oascc.ZkStateReader$3.process Updating live nodes... (19)
  [junit4]  2> 448351 T2601 oascc.ZkStateReader$3.process Updating live nodes... (19)
  [junit4]  2> 448353 T2579 oascc.ZkStateReader$3.process Updating live nodes... (19)
  [junit4]  2> 448353 T2581 oascc.ZkStateReader$3.process Updating live nodes... (19)
  [junit4]  2> 448351 T2595 oascc.ZkStateReader$3.process Updating live nodes... (19)
  [junit4]  2> 448353 T2605 oascc.ZkStateReader$3.process Updating live nodes... (19)
  [junit4]  2> 448354 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 448353 T2609 oascc.ZkStateReader$3.process Updating live nodes... (19)
  [junit4]  2> 448353 T2603 oascc.ZkStateReader$3.process Updating live nodes... (19)
  [junit4]  2> 448353 T2597 oascc.ZkStateReader$3.process Updating live nodes... (19)
  [junit4]  2> 448353 T2589 oascc.ZkStateReader$3.process Updating live nodes... (19)
  [junit4]  2> 448353 T2567 oascc.ZkStateReader$3.process Updating live nodes... (19)
  [junit4]  2> 448353 T2591 oascc.ZkStateReader$3.process Updating live nodes... (19)
  [junit4]  2> 448353 T2575 oascc.ZkStateReader$3.process Updating live nodes... (19)
  [junit4]  2> 448352 T2599 oascc.ZkStateReader$3.process Updating live nodes... (19)
  [junit4]  2> 448352 T2583 oascc.ZkStateReader$3.process Updating live nodes... (19)
  [junit4]  2> 448352 T2585 oascc.ZkStateReader$3.process Updating live nodes... (19)
  [junit4]  2> 448356 T2611 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b5c498 name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 448359 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 448362 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
  [junit4]  2> 448366 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node19
  [junit4]  2> 448371 T2587 oascc.ZkStateReader$3.process Updating live nodes... (20)
  [junit4]  2> 448371 T2591 oascc.ZkStateReader$3.process Updating live nodes... (20)
  [junit4]  2> 448372 T2579 oascc.ZkStateReader$3.process Updating live nodes... (20)
  [junit4]  2> 448372 T2575 oascc.ZkStateReader$3.process Updating live nodes... (20)
  [junit4]  2> 448372 T2577 oascc.ZkStateReader$3.process Updating live nodes... (20)
  [junit4]  2> 448373 T2581 oascc.ZkStateReader$3.process Updating live nodes... (20)
  [junit4]  2> 448372 T2589 oascc.ZkStateReader$3.process Updating live nodes... (20)
  [junit4]  2> 448372 T2573 oascc.ZkStateReader$3.process Updating live nodes... (20)
  [junit4]  2> 448372 T2595 oascc.ZkStateReader$3.process Updating live nodes... (20)
  [junit4]  2> 448374 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 448373 T2609 oascc.ZkStateReader$3.process Updating live nodes... (20)
  [junit4]  2> 448373 T2611 oascc.ZkStateReader$3.process Updating live nodes... (20)
  [junit4]  2> 448373 T2603 oascc.ZkStateReader$3.process Updating live nodes... (20)
  [junit4]  2> 448373 T2585 oascc.ZkStateReader$3.process Updating live nodes... (20)
  [junit4]  2> 448373 T2593 oascc.ZkStateReader$3.process Updating live nodes... (20)
  [junit4]  2> 448373 T2567 oascc.ZkStateReader$3.process Updating live nodes... (20)
  [junit4]  2> 448373 T2607 oascc.ZkStateReader$3.process Updating live nodes... (20)
  [junit4]  2> 448372 T2599 oascc.ZkStateReader$3.process Updating live nodes... (20)
  [junit4]  2> 448372 T2597 oascc.ZkStateReader$3.process Updating live nodes... (20)
  [junit4]  2> 448372 T2601 oascc.ZkStateReader$3.process Updating live nodes... (20)
  [junit4]  2> 448372 T2583 oascc.ZkStateReader$3.process Updating live nodes... (20)
  [junit4]  2> 448372 T2605 oascc.ZkStateReader$3.process Updating live nodes... (20)
  [junit4]  2> 448376 T2613 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@18ed79a name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 448380 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 448382 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
  [junit4]  2> 448385 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node20
  [junit4]  2> 448390 T2577 oascc.ZkStateReader$3.process Updating live nodes... (21)
  [junit4]  2> 448390 T2579 oascc.ZkStateReader$3.process Updating live nodes... (21)
  [junit4]  2> 448391 T2589 oascc.ZkStateReader$3.process Updating live nodes... (21)
  [junit4]  2> 448391 T2601 oascc.ZkStateReader$3.process Updating live nodes... (21)
  [junit4]  2> 448391 T2599 oascc.ZkStateReader$3.process Updating live nodes... (21)
  [junit4]  2> 448391 T2573 oascc.ZkStateReader$3.process Updating live nodes... (21)
  [junit4]  2> 448392 T2581 oascc.ZkStateReader$3.process Updating live nodes... (21)
  [junit4]  2> 448391 T2605 oascc.ZkStateReader$3.process Updating live nodes... (21)
  [junit4]  2> 448390 T2591 oascc.ZkStateReader$3.process Updating live nodes... (21)
  [junit4]  2> 448390 T2595 oascc.ZkStateReader$3.process Updating live nodes... (21)
  [junit4]  2> 448393 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 448392 T2603 oascc.ZkStateReader$3.process Updating live nodes... (21)
  [junit4]  2> 448392 T2609 oascc.ZkStateReader$3.process Updating live nodes... (21)
  [junit4]  2> 448392 T2611 oascc.ZkStateReader$3.process Updating live nodes... (21)
  [junit4]  2> 448392 T2593 oascc.ZkStateReader$3.process Updating live nodes... (21)
  [junit4]  2> 448392 T2585 oascc.ZkStateReader$3.process Updating live nodes... (21)
  [junit4]  2> 448392 T2567 oascc.ZkStateReader$3.process Updating live nodes... (21)
  [junit4]  2> 448392 T2575 oascc.ZkStateReader$3.process Updating live nodes... (21)
  [junit4]  2> 448391 T2613 oascc.ZkStateReader$3.process Updating live nodes... (21)
  [junit4]  2> 448391 T2583 oascc.ZkStateReader$3.process Updating live nodes... (21)
  [junit4]  2> 448391 T2607 oascc.ZkStateReader$3.process Updating live nodes... (21)
  [junit4]  2> 448391 T2597 oascc.ZkStateReader$3.process Updating live nodes... (21)
  [junit4]  2> 448391 T2587 oascc.ZkStateReader$3.process Updating live nodes... (21)
  [junit4]  2> 448395 T2615 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5f6bba name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 448399 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 448401 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
  [junit4]  2> 448404 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node21
  [junit4]  2> 448409 T2587 oascc.ZkStateReader$3.process Updating live nodes... (22)
  [junit4]  2> 448410 T2573 oascc.ZkStateReader$3.process Updating live nodes... (22)
  [junit4]  2> 448410 T2595 oascc.ZkStateReader$3.process Updating live nodes... (22)
  [junit4]  2> 448410 T2577 oascc.ZkStateReader$3.process Updating live nodes... (22)
  [junit4]  2> 448409 T2591 oascc.ZkStateReader$3.process Updating live nodes... (22)
  [junit4]  2> 448412 T2607 oascc.ZkStateReader$3.process Updating live nodes... (22)
  [junit4]  2> 448412 T2593 oascc.ZkStateReader$3.process Updating live nodes... (22)
  [junit4]  2> 448412 T2599 oascc.ZkStateReader$3.process Updating live nodes... (22)
  [junit4]  2> 448413 T2603 oascc.ZkStateReader$3.process Updating live nodes... (22)
  [junit4]  2> 448413 T2609 oascc.ZkStateReader$3.process Updating live nodes... (22)
  [junit4]  2> 448412 T2567 oascc.ZkStateReader$3.process Updating live nodes... (22)
  [junit4]  2> 448412 T2613 oascc.ZkStateReader$3.process Updating live nodes... (22)
  [junit4]  2> 448412 T2575 oascc.ZkStateReader$3.process Updating live nodes... (22)
  [junit4]  2> 448411 T2583 oascc.ZkStateReader$3.process Updating live nodes... (22)
  [junit4]  2> 448411 T2601 oascc.ZkStateReader$3.process Updating live nodes... (22)
  [junit4]  2> 448411 T2597 oascc.ZkStateReader$3.process Updating live nodes... (22)
  [junit4]  2> 448411 T2579 oascc.ZkStateReader$3.process Updating live nodes... (22)
  [junit4]  2> 448411 T2605 oascc.ZkStateReader$3.process Updating live nodes... (22)
  [junit4]  2> 448411 T2589 oascc.ZkStateReader$3.process Updating live nodes... (22)
  [junit4]  2> 448416 T2617 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@13fabc5 name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
  [junit4]  2> 448414 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 448413 T2615 oascc.ZkStateReader$3.process Updating live nodes... (22)
  [junit4]  2> 448413 T2611 oascc.ZkStateReader$3.process Updating live nodes... (22)
  [junit4]  2> 448412 T2581 oascc.ZkStateReader$3.process Updating live nodes... (22)
  [junit4]  2> 448412 T2585 oascc.ZkStateReader$3.process Updating live nodes... (22)
  [junit4]  2> 448418 T2522 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
  [junit4]  2> 448421 T2522 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
  [junit4]  2> 448426 T2522 oascc.SolrZkClient.makePath makePath: /live_nodes/node22
  [junit4]  2> 448431 T2587 oascc.ZkStateReader$3.process Updating live nodes... (23)
  [junit4]  2> 448432 T2577 oascc.ZkStateReader$3.process Updating live nodes... (23)
  [junit4]  2> 448432 T2579 oascc.ZkStateReader$3.process Updating live nodes... (23)
  [junit4]  2> 448432 T2605 oascc.ZkStateReader$3.process Updating live nodes... (23)
  [junit4]  2> 448433 T2611 oascc.ZkStateReader$3.process Updating live nodes... (23)
  [junit4]  2> 448432 T2595 oascc.ZkStateReader$3.process Updating live nodes... (23)
  [junit4]  2> 448432 T2591 oascc.ZkStateReader$3.process Updating live nodes... (23)
  [junit4]  2> 448434 T2609 oascc.ZkStateReader$3.process Updating live nodes... (23)
  [junit4]  2> 448432 T2573 oascc.ZkStateReader$3.process Updating live nodes... (23)
  [junit4]  2> 448435 T2522 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
  [junit4]  2> 448434 T2603 oascc.ZkStateReader$3.process Updating live nodes... (23)
  [junit4]  2> 448434 T2617 oascc.ZkStateReader$3.process Updating live nodes... (23)
  [junit4]  2> 448434 T2581 oascc.ZkStateReader$3.process Updating live nodes... (23)
  [junit4]  2> 448437 T2619 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@71217e name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event WatchedEvent state:SyncConnected type:None path:null
[...truncated too long message...]
queue/qn-
  [junit4]  2>     at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
  [junit4]  2>     at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
  [junit4]  2>     at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
  [junit4]  2>     at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:243)
  [junit4]  2>     at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:240)
  [junit4]  2>     at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
  [junit4]  2>     at org.apache.solr.common.cloud.SolrZkClient.create(SolrZkClient.java:240)
  [junit4]  2>     at org.apache.solr.cloud.DistributedQueue.createData(DistributedQueue.java:346)
  [junit4]  2>     at org.apache.solr.cloud.DistributedQueue.offer(DistributedQueue.java:330)
  [junit4]  2>     at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:198)
  [junit4]  2>     at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:163)
  [junit4]  2>     at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:125)
  [junit4]  2>     at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:314)
  [junit4]  2>     at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:221)
  [junit4]  2>     at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:452)
  [junit4]  2>     at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:216)
  [junit4]  2>     at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:163)
  [junit4]  2>     at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:125)
  [junit4]  2>     at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
  [junit4]  2>     at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:358)
  [junit4]  2>     at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
  [junit4]  2>     at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
  [junit4]  2>
  [junit4]  2> 750036 T3044 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
  [junit4]  2> 750037 T3044 oasc.LeaderElector$ElectionWatcher.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
  [junit4]  2>     at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
  [junit4]  2>     at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
  [junit4]  2>     at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
  [junit4]  2>     at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
  [junit4]  2>     at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
  [junit4]  2>     at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
  [junit4]  2>     at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
  [junit4]  2>     at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
  [junit4]  2>     at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
  [junit4]  2>     at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:358)
  [junit4]  2>     at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
  [junit4]  2>     at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
  [junit4]  2>
  [junit4]  2> 750038 T3044 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
  [junit4]  2> 750038 T3044 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
  [junit4]  2> 750121 T3116 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
  [junit4]  2> 750206 T2787 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
  [junit4]  2> 750206 T2787 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
  [junit4]  2> 750207 T2787 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
  [junit4]  2> 750207 T2787 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
  [junit4]  2> 750207 T2787 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
  [junit4]  2> 750206 T3156 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
  [junit4]  2>     at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
  [junit4]  2>     at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
  [junit4]  2>     at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
  [junit4]  2>     at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
  [junit4]  2>     at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
  [junit4]  2>     at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
  [junit4]  2>     at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
  [junit4]  2>     at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:316)
  [junit4]  2>     at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
  [junit4]  2>     at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:288)
  [junit4]  2>
  [junit4]  2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-A819205877C572F6-001
  [junit4]  2> NOTE: test params are: codec=Lucene41, sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=ar_JO, timezone=Africa/Ndjamena
  [junit4]  2> NOTE: Linux 3.8.0-42-generic i386/Oracle Corporation 1.8.0_05 (32-bit)/cpus=8,threads=1,free=118175464,total=317980672
  [junit4]  2> NOTE: All tests run in this JVM: [TestRecovery, OpenCloseCoreStressTest, HardAutoCommitTest, DeleteReplicaTest, IndexSchemaRuntimeFieldTest, TestXIncludeConfig, TestFreeTextSuggestions, CollectionsAPIDistributedZkTest, TestFieldTypeResource, BadCopyFieldTest, CopyFieldTest, TestCSVLoader, UniqFieldsUpdateProcessorFactoryTest, TestIndexingPerformance, TestDocumentBuilder, AutoCommitTest, AtomicUpdatesTest, TestQueryTypes, TestCollapseQParserPlugin, BasicDistributedZk2Test, TestWriterPerf, TestCoreContainer, TestComplexPhraseQParserPlugin, TestDistribDocBasedVersion, TestFastLRUCache, TestDynamicFieldResource, DOMUtilTest, QueryResultKeyTest, DistribDocExpirationUpdateProcessorTest, UpdateRequestProcessorFactoryTest, TimeZoneUtilsTest, SchemaVersionSpecificBehaviorTest, PreAnalyzedUpdateProcessorTest, SOLR749Test, TestNonDefinedSimilarityFactory, TestFileDictionaryLookup, MultiThreadedOCPTest, TestRTGBase, QueryParsingTest, OutputWriterTest, LoggingHandlerTest, TestInfoStreamLogging, TestPseudoReturnFields, TestFastWriter, TestReplicationHandlerBackup, AssignTest, TermVectorComponentTest, SolrRequestParserTest, TestPerFieldSimilarity, TestAtomicUpdateErrorCases, TestManagedResource, TermVectorComponentDistributedTest, TestReRankQParserPlugin, TestSerializedLuceneMatchVersion, TestChildDocTransformer, TestNoOpRegenerator, TestGroupingSearch, TestFieldTypeCollectionResource, TestBinaryField, StatsComponentTest, CursorMarkTest, BasicZkTest, SuggesterFSTTest, BJQParserTest, CurrencyFieldXmlFileTest, DistributedQueryElevationComponentTest, TestDistributedMissingSort, SearchHandlerTest, TestLFUCache, TestSolrJ, FileBasedSpellCheckerTest, TestRealTimeGet, RequestHandlersTest, AnalysisAfterCoreReloadTest, SolrInfoMBeanTest, TestDistributedGrouping, TestExtendedDismaxParser, URLClassifyProcessorTest, SuggestComponentTest, ChaosMonkeySafeLeaderTest]
  [junit4] Completed on J1 in 74.71s, 1 test, 1 failure <<< FAILURES!
[...truncated 59 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1299: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:923: There were test failures: 405 suites, 1656 tests, 3 suite-level errors, 1 error, 1 failure, 954 ignored (13 assumptions)
Total time: 65 minutes 33 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 32bit/jdk1.8.0_05 -client -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
---------------------------------------------------------------------
To unsubscribe, e-mail:Â dev-unsubscribe@lucene.apache.org
For additional commands, e-mail:Â dev-help@lucene.apache.org
--
Regards,
Shalin Shekhar Mangar.
Re: [JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.8.0_05) - Build #
10616 - Still Failing!
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
I was looking into the OverseerTest failure and the following caught my
attention.
This error can be due to a slow machine but it also happens on fast
machines if you try to do a lot of writes very fast on ZooKeeper which is
what the testShardLeaderChange does. Perhaps we should add a small wait
between operations?
[junit4] 2> 472241 T2893 oazsp.FileTxnLog.commit WARN fsync-ing
the write ahead log in SyncThread:0 took 11588ms which will adversely
effect operation latency. See the ZooKeeper troubleshooting guide
On Sun, Jul 6, 2014 at 10:46 AM, Policeman Jenkins Server <
jenkins@thetaphi.de> wrote:
> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/10616/
> Java: 32bit/jdk1.8.0_05 -client -XX:+UseParallelGC
>
> 5 tests failed.
> FAILED: junit.framework.TestSuite.org.apache.solr.cloud.OverseerTest
>
> Error Message:
> ERROR: SolrZkClient opens=165 closes=164
>
> Stack Trace:
> java.lang.AssertionError: ERROR: SolrZkClient opens=165 closes=164
> at __randomizedtesting.SeedInfo.seed([A819205877C572F6]:0)
> at org.junit.Assert.fail(Assert.java:93)
> at
> org.apache.solr.SolrTestCaseJ4.endTrackingZkClients(SolrTestCaseJ4.java:436)
> at
> org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:182)
> at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:790)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
> at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
> at
> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
> at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
> at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
> at
> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
> at java.lang.Thread.run(Thread.java:745)
>
>
> FAILED: junit.framework.TestSuite.org.apache.solr.cloud.OverseerTest
>
> Error Message:
> 2 threads leaked from SUITE scope at org.apache.solr.cloud.OverseerTest:
> 1) Thread[id=2969,
> name=TEST-OverseerTest.testShardLeaderChange-seed#[A819205877C572F6]-SendThread(
> 127.0.0.1:33880), state=TIMED_WAITING, group=TGRP-OverseerTest]
> at java.lang.Thread.sleep(Native Method) at
> org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003) 2)
> Thread[id=2970,
> name=TEST-OverseerTest.testShardLeaderChange-seed#[A819205877C572F6]-EventThread,
> state=WAITING, group=TGRP-OverseerTest] at
> sun.misc.Unsafe.park(Native Method) at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
>
> Stack Trace:
> com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from
> SUITE scope at org.apache.solr.cloud.OverseerTest:
> 1) Thread[id=2969,
> name=TEST-OverseerTest.testShardLeaderChange-seed#[A819205877C572F6]-SendThread(
> 127.0.0.1:33880), state=TIMED_WAITING, group=TGRP-OverseerTest]
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
> 2) Thread[id=2970,
> name=TEST-OverseerTest.testShardLeaderChange-seed#[A819205877C572F6]-EventThread,
> state=WAITING, group=TGRP-OverseerTest]
> at sun.misc.Unsafe.park(Native Method)
> at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
> at __randomizedtesting.SeedInfo.seed([A819205877C572F6]:0)
>
>
> FAILED: junit.framework.TestSuite.org.apache.solr.cloud.OverseerTest
>
> Error Message:
> There are still zombie threads that couldn't be terminated: 1)
> Thread[id=2969,
> name=TEST-OverseerTest.testShardLeaderChange-seed#[A819205877C572F6]-SendThread(
> 127.0.0.1:33880), state=TIMED_WAITING, group=TGRP-OverseerTest]
> at java.lang.Thread.sleep(Native Method) at
> org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
>
> Stack Trace:
> com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie
> threads that couldn't be terminated:
> 1) Thread[id=2969,
> name=TEST-OverseerTest.testShardLeaderChange-seed#[A819205877C572F6]-SendThread(
> 127.0.0.1:33880), state=TIMED_WAITING, group=TGRP-OverseerTest]
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
> at __randomizedtesting.SeedInfo.seed([A819205877C572F6]:0)
>
>
> REGRESSION: org.apache.solr.cloud.OverseerTest.testShardLeaderChange
>
> Error Message:
> KeeperErrorCode = NodeExists for /live_nodes/node1
>
> Stack Trace:
> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode
> = NodeExists for /live_nodes/node1
> at
> __randomizedtesting.SeedInfo.seed([A819205877C572F6:764AA7AF6D5D8707]:0)
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
> at
> org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:432)
> at
> org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
> at
> org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:429)
> at
> org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:386)
> at
> org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:373)
> at
> org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:350)
> at
> org.apache.solr.cloud.OverseerTest$MockZKController.<init>(OverseerTest.java:85)
> at
> org.apache.solr.cloud.OverseerTest.testShardLeaderChange(OverseerTest.java:746)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
> at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
> at
> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
> at
> org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
> at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
> at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> at
> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
> at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
> at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
> at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
> at
> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
> at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
> at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
> at
> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
> at java.lang.Thread.run(Thread.java:745)
>
>
> FAILED: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch
>
> Error Message:
> shard2 is not consistent. Got 153 from
> http://127.0.0.1:51995/collection1lastClient and got 132 from
> http://127.0.0.1:36739/collection1
>
> Stack Trace:
> java.lang.AssertionError: shard2 is not consistent. Got 153 from
> http://127.0.0.1:51995/collection1lastClient and got 132 from
> http://127.0.0.1:36739/collection1
> at
> __randomizedtesting.SeedInfo.seed([A819205877C572F6:29FFAE40009A12CA]:0)
> at org.junit.Assert.fail(Assert.java:93)
> at
> org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1139)
> at
> org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1118)
> at
> org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:150)
> at
> org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
> at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
> at
> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
> at
> org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
> at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
> at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> at
> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
> at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
> at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
> at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
> at
> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
> at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
> at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
> at
> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
> at java.lang.Thread.run(Thread.java:745)
>
>
>
>
> Build Log:
> [...truncated 11133 lines...]
> [junit4] Suite: org.apache.solr.cloud.OverseerTest
> [junit4] 2> Creating dataDir:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.OverseerTest-A819205877C572F6-001/init-core-data-001
> [junit4] 2> 443102 T2522 oas.SolrTestCaseJ4.buildSSLConfig Randomized
> ssl (false) and clientAuth (false)
> [junit4] 2> 443103 T2522 oas.SolrTestCaseJ4.initCore ####initCore
> [junit4] 2> 443103 T2522 oas.SolrTestCaseJ4.initCore ####initCore end
> [junit4] 2> 443107 T2522 oas.SolrTestCaseJ4.setUp ###Starting
> testStateChange
> [junit4] 2> 443107 T2522 oasc.ZkTestServer.run STARTING ZK TEST SERVER
> [junit4] 1> client port:0.0.0.0/0.0.0.0:0
> [junit4] 2> 443108 T2523 oasc.ZkTestServer$ZKServerMain.runFromConfig
> Starting server
> [junit4] 2> 443208 T2522 oasc.ZkTestServer.run start zk server on
> port:45124
> [junit4] 2> 443209 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 443282 T2529 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@ed72a5
> name:ZooKeeperConnection Watcher:127.0.0.1:45124/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 443283 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 443284 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 443287 T2531 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@6366c4
> name:ZooKeeperConnection Watcher:127.0.0.1:45124 got event WatchedEvent
> state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 443287 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 443290 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 443292 T2533 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@60240b
> name:ZooKeeperConnection Watcher:127.0.0.1:45124 got event WatchedEvent
> state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 443292 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 443293 T2522 oascc.SolrZkClient.makePath makePath: /solr
> [junit4] 2> 443298 T2522 oascc.SolrZkClient.makePath makePath:
> /live_nodes
> [junit4] 2> 443301 T2522 oascc.SolrZkClient.makePath makePath:
> /clusterstate.json
> [junit4] 2> 443304 T2522 oascc.SolrZkClient.makePath makePath:
> /aliases.json
> [junit4] 2> 443306 T2522
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
> [junit4] 2> 443311 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 443312 T2535 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@68e786
> name:ZooKeeperConnection Watcher:127.0.0.1:45124/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 443313 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 443314 T2522 oascc.SolrZkClient.makePath makePath:
> /overseer_elect
> [junit4] 2> 443318 T2522 oascc.SolrZkClient.makePath makePath:
> /overseer_elect/election
> [junit4] 2> 443324 T2522
> oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader
> 127.0.0.1:45124_solr
> [junit4] 2> 443324 T2522 oascc.SolrZkClient.makePath makePath:
> /overseer_elect/leader
> [junit4] 2> 443328 T2522 oasc.Overseer.start Overseer
> (id=92053395863699459-127.0.0.1:45124_solr-n_0000000000) starting
> [junit4] 2> 443331 T2522 oascc.SolrZkClient.makePath makePath:
> /overseer/queue
> [junit4] 2> 443336 T2522 oascc.SolrZkClient.makePath makePath:
> /overseer/queue-work
> [junit4] 2> 443340 T2522 oascc.SolrZkClient.makePath makePath:
> /overseer/collection-map-failure
> [junit4] 2> 443345 T2522 oascc.SolrZkClient.makePath makePath:
> /overseer/collection-map-running
> [junit4] 2> 443350 T2522 oascc.SolrZkClient.makePath makePath:
> /overseer/collection-map-completed
> [junit4] 2> 443357 T2522 oascc.SolrZkClient.makePath makePath:
> /overseer/collection-queue-work
> [junit4] 2> 443365 T2537 oasc.OverseerCollectionProcessor.run Process
> current queue of collection creations
> [junit4] 2> 443366 T2536 oasc.Overseer$ClusterStateUpdater.run
> Starting to work on the main queue
> [junit4] 2> 443368 T2522 oascc.ZkStateReader.updateClusterState
> Updating cloud state from ZooKeeper...
> [junit4] 2> 443369 T2536 oascc.ZkStateReader.updateClusterState
> Updating cloud state from ZooKeeper...
> [junit4] 2> 443370 T2536
> oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null
> message={
> [junit4] 2> "operation":"state",
> [junit4] 2> "base_url":"http://127.0.0.1/solr",
> [junit4] 2> "node_name":"node1",
> [junit4] 2> "collection":"collection1",
> [junit4] 2> "core":"core1",
> [junit4] 2> "roles":"",
> [junit4] 2> "state":"recovering"}
> [junit4] 2> 443370 T2536
> oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard
> shard=shard1
> [junit4] 2> 443374 T2535
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
> [junit4] 2> 443375 T2529 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
> [junit4] 2> 443420 T2522 oascc.ZkStateReader.updateClusterState
> Updating cloud state from ZooKeeper...
> [junit4] 2> 443422 T2535
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
> [junit4] 2> 443424 T2536 oascc.ZkStateReader.updateClusterState
> Updating cloud state from ZooKeeper...
> [junit4] 2> 443425 T2536
> oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null
> message={
> [junit4] 2> "operation":"state",
> [junit4] 2> "base_url":"http://127.0.0.1/solr",
> [junit4] 2> "node_name":"node1",
> [junit4] 2> "collection":"collection1",
> [junit4] 2> "core":"core1",
> [junit4] 2> "roles":"",
> [junit4] 2> "state":"active"}
> [junit4] 2> 443426 T2536
> oasc.Overseer$ClusterStateUpdater.updateState node=core_node1 is already
> registered
> [junit4] 2> 443426 T2536
> oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already
> registered
> [junit4] 2> 443431 T2535
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
> [junit4] 2> 443534 T2529 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
> [junit4] 2> 443575 T2522 oasc.ZkTestServer.send4LetterWord connecting
> to 127.0.0.1:45124 45124
> [junit4] 2> 443988 T2523 oasc.ZkTestServer.send4LetterWord connecting
> to 127.0.0.1:45124 45124
> [junit4] 2> 443991 T2522 oas.SolrTestCaseJ4.tearDown ###Ending
> testStateChange
> [junit4] 2> 443992 T2536 oasc.Overseer$ClusterStateUpdater.run
> Overseer Loop exiting : 127.0.0.1:45124_solr
> [junit4] 2> 443994 T2538
> oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not
> read the data org.apache.zookeeper.KeeperException$SessionExpiredException:
> KeeperErrorCode = Session expired for /overseer_elect/leader
> [junit4] 2> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
> [junit4] 2> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> [junit4] 2> at
> org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
> [junit4] 2> at
> org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
> [junit4] 2> at
> org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
> [junit4] 2> at
> org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
> [junit4] 2> at
> org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
> [junit4] 2> at
> org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:316)
> [junit4] 2> at
> org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
> [junit4] 2> at
> org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:288)
> [junit4] 2>
> [junit4] 2> 444000 T2522 oas.SolrTestCaseJ4.setUp ###Starting
> testShardAssignment
> [junit4] 2> 444001 T2522 oasc.ZkTestServer.run STARTING ZK TEST SERVER
> [junit4] 1> client port:0.0.0.0/0.0.0.0:0
> [junit4] 2> 444002 T2539 oasc.ZkTestServer$ZKServerMain.runFromConfig
> Starting server
> [junit4] 2> 444101 T2522 oasc.ZkTestServer.run start zk server on
> port:57742
> [junit4] 2> 444103 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 444107 T2545 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@9fc604
> name:ZooKeeperConnection Watcher:127.0.0.1:57742 got event WatchedEvent
> state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 444108 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 444111 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 444112 T2547 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@104f2e0
> name:ZooKeeperConnection Watcher:127.0.0.1:57742 got event WatchedEvent
> state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 444113 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 444114 T2522 oascc.SolrZkClient.makePath makePath: /solr
> [junit4] 2> 444119 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 444121 T2549 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@896251
> name:ZooKeeperConnection Watcher:127.0.0.1:57742/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 444121 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 444122 T2522 oascc.SolrZkClient.makePath makePath:
> /live_nodes
> [junit4] 2> 444125 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 444127 T2551 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@10c3e42
> name:ZooKeeperConnection Watcher:127.0.0.1:57742/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 444127 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 444128 T2522 oascc.SolrZkClient.makePath makePath:
> /overseer_elect
> [junit4] 2> 444132 T2522 oascc.SolrZkClient.makePath makePath:
> /overseer_elect/election
> [junit4] 2> 444137 T2522
> oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader
> 127.0.0.1:57742_solr
> [junit4] 2> 444138 T2522 oascc.SolrZkClient.makePath makePath:
> /overseer_elect/leader
> [junit4] 2> 444141 T2522 oasc.Overseer.start Overseer
> (id=92053395922223107-127.0.0.1:57742_solr-n_0000000000) starting
> [junit4] 2> 444144 T2522 oascc.SolrZkClient.makePath makePath:
> /overseer/queue
> [junit4] 2> 444149 T2522 oascc.SolrZkClient.makePath makePath:
> /overseer/queue-work
> [junit4] 2> 444153 T2522 oascc.SolrZkClient.makePath makePath:
> /overseer/collection-map-failure
> [junit4] 2> 444157 T2522 oascc.SolrZkClient.makePath makePath:
> /overseer/collection-map-running
> [junit4] 2> 444162 T2522 oascc.SolrZkClient.makePath makePath:
> /overseer/collection-map-completed
> [junit4] 2> 444166 T2522 oascc.SolrZkClient.makePath makePath:
> /overseer/collection-queue-work
> [junit4] 2> 444173 T2553 oasc.OverseerCollectionProcessor.run Process
> current queue of collection creations
> [junit4] 2> 444175 T2522 oascc.SolrZkClient.makePath makePath:
> /clusterstate.json
> [junit4] 2> 444175 T2552 oasc.Overseer$ClusterStateUpdater.run
> Starting to work on the main queue
> [junit4] 2> 444177 T2522 oascc.SolrZkClient.makePath makePath:
> /aliases.json
> [junit4] 2> 444179 T2522
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
> [junit4] 2> 444183 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 444185 T2555 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@1196b81
> name:ZooKeeperConnection Watcher:127.0.0.1:57742/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 444185 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 444187 T2522
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
> [junit4] 2> 444190 T2522 oascc.SolrZkClient.makePath makePath:
> /live_nodes/127.0.0.1
> [junit4] 2> 444194 T2549 oascc.ZkStateReader$3.process Updating live
> nodes... (1)
> [junit4] 2> 444194 T2555 oascc.ZkStateReader$3.process Updating live
> nodes... (1)
> [junit4] 2> 444197 T2551
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
> [junit4] 2> 444199 T2552 oascc.ZkStateReader.updateClusterState
> Updating cloud state from ZooKeeper...
> [junit4] 2> 444199 T2552
> oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3
> message={
> [junit4] 2> "operation":"state",
> [junit4] 2> "state":"active",
> [junit4] 2> "node_name":"127.0.0.1",
> [junit4] 2> "core":"core1",
> [junit4] 2> "core_node_name":"node1",
> [junit4] 2> "collection":"collection1",
> [junit4] 2> "numShards":"3",
> [junit4] 2> "base_url":"http://127.0.0.1/solr/"}
> [junit4] 2> 444200 T2552
> oasc.Overseer$ClusterStateUpdater.createCollection Create collection
> collection1 with shards [shard1, shard2, shard3]
> [junit4] 2> 444200 T2552
> oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard
> shard=shard2
> [junit4] 2> 444204 T2551
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
> [junit4] 2> 444205 T2555 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> [junit4] 2> 444205 T2549 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> [junit4] 2> 444697 T2522 oascc.SolrZkClient.makePath makePath:
> /collections/collection1/leader_elect/shard2/election
> [junit4] 2> 444712 T2522 oascc.SolrZkClient.makePath makePath:
> /collections/collection1/leaders/shard2
> [junit4] 2> 444719 T2551
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
> [junit4] 2> 444721 T2551
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
> [junit4] 2> 444721 T2552 oascc.ZkStateReader.updateClusterState
> Updating cloud state from ZooKeeper...
> [junit4] 2> 444727 T2552
> oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3
> message={
> [junit4] 2> "operation":"state",
> [junit4] 2> "state":"active",
> [junit4] 2> "node_name":"127.0.0.1",
> [junit4] 2> "core":"core2",
> [junit4] 2> "core_node_name":"node2",
> [junit4] 2> "collection":"collection1",
> [junit4] 2> "numShards":"3",
> [junit4] 2> "base_url":"http://127.0.0.1/solr/"}
> [junit4] 2> 444728 T2552
> oasc.Overseer$ClusterStateUpdater.updateState Collection already exists
> with numShards=3
> [junit4] 2> 444728 T2552
> oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard
> shard=shard3
> [junit4] 2> 444731 T2551
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
> [junit4] 2> 444834 T2555 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> [junit4] 2> 444834 T2549 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> [junit4] 2> 445222 T2522 oascc.SolrZkClient.makePath makePath:
> /collections/collection1/leader_elect/shard3/election
> [junit4] 2> 445233 T2522 oascc.SolrZkClient.makePath makePath:
> /collections/collection1/leaders/shard3
> [junit4] 2> 445240 T2551
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
> [junit4] 2> 445240 T2551
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
> [junit4] 2> 445241 T2551
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
> [junit4] 2> 445242 T2551
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
> [junit4] 2> 445242 T2552 oascc.ZkStateReader.updateClusterState
> Updating cloud state from ZooKeeper...
> [junit4] 2> 445248 T2552
> oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3
> message={
> [junit4] 2> "operation":"state",
> [junit4] 2> "state":"active",
> [junit4] 2> "node_name":"127.0.0.1",
> [junit4] 2> "core":"core3",
> [junit4] 2> "core_node_name":"node3",
> [junit4] 2> "collection":"collection1",
> [junit4] 2> "numShards":"3",
> [junit4] 2> "base_url":"http://127.0.0.1/solr/"}
> [junit4] 2> 445248 T2552
> oasc.Overseer$ClusterStateUpdater.updateState Collection already exists
> with numShards=3
> [junit4] 2> 445248 T2552
> oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard
> shard=shard1
> [junit4] 2> 445251 T2551
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
> [junit4] 2> 445354 T2549 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> [junit4] 2> 445354 T2555 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> [junit4] 2> 445742 T2522 oascc.SolrZkClient.makePath makePath:
> /collections/collection1/leader_elect/shard1/election
> [junit4] 2> 445753 T2522 oascc.SolrZkClient.makePath makePath:
> /collections/collection1/leaders/shard1
> [junit4] 2> 445764 T2551
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
> [junit4] 2> 445765 T2551
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
> [junit4] 2> 445765 T2551
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
> [junit4] 2> 445766 T2551
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
> [junit4] 2> 445767 T2552 oascc.ZkStateReader.updateClusterState
> Updating cloud state from ZooKeeper...
> [junit4] 2> 445771 T2552
> oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3
> message={
> [junit4] 2> "operation":"state",
> [junit4] 2> "state":"active",
> [junit4] 2> "node_name":"127.0.0.1",
> [junit4] 2> "core":"core4",
> [junit4] 2> "core_node_name":"node4",
> [junit4] 2> "collection":"collection1",
> [junit4] 2> "numShards":"3",
> [junit4] 2> "base_url":"http://127.0.0.1/solr/"}
> [junit4] 2> 445771 T2552
> oasc.Overseer$ClusterStateUpdater.updateState Collection already exists
> with numShards=3
> [junit4] 2> 445771 T2552
> oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard
> shard=shard2
> [junit4] 2> 445773 T2551
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
> [junit4] 2> 445876 T2555 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> [junit4] 2> 445876 T2549 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> [junit4] 2> 446267 T2522 oascc.SolrZkClient.makePath makePath:
> /collections/collection1/leader_elect/shard2/election
> [junit4] 2> 446276 T2551
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
> [junit4] 2> 446276 T2551
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
> [junit4] 2> 446276 T2551
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
> [junit4] 2> 446278 T2552 oascc.ZkStateReader.updateClusterState
> Updating cloud state from ZooKeeper...
> [junit4] 2> 446279 T2552
> oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3
> message={
> [junit4] 2> "operation":"state",
> [junit4] 2> "state":"active",
> [junit4] 2> "node_name":"127.0.0.1",
> [junit4] 2> "core":"core5",
> [junit4] 2> "core_node_name":"node5",
> [junit4] 2> "collection":"collection1",
> [junit4] 2> "numShards":"3",
> [junit4] 2> "base_url":"http://127.0.0.1/solr/"}
> [junit4] 2> 446280 T2552
> oasc.Overseer$ClusterStateUpdater.updateState Collection already exists
> with numShards=3
> [junit4] 2> 446280 T2552
> oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard
> shard=shard3
> [junit4] 2> 446283 T2551
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
> [junit4] 2> 446386 T2555 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> [junit4] 2> 446386 T2549 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> [junit4] 2> 446776 T2522 oascc.SolrZkClient.makePath makePath:
> /collections/collection1/leader_elect/shard3/election
> [junit4] 2> 446787 T2551
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
> [junit4] 2> 446788 T2551
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
> [junit4] 2> 446788 T2551
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
> [junit4] 2> 446790 T2552 oascc.ZkStateReader.updateClusterState
> Updating cloud state from ZooKeeper...
> [junit4] 2> 446791 T2552
> oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3
> message={
> [junit4] 2> "operation":"state",
> [junit4] 2> "state":"active",
> [junit4] 2> "node_name":"127.0.0.1",
> [junit4] 2> "core":"core6",
> [junit4] 2> "core_node_name":"node6",
> [junit4] 2> "collection":"collection1",
> [junit4] 2> "numShards":"3",
> [junit4] 2> "base_url":"http://127.0.0.1/solr/"}
> [junit4] 2> 446791 T2552
> oasc.Overseer$ClusterStateUpdater.updateState Collection already exists
> with numShards=3
> [junit4] 2> 446791 T2552
> oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard
> shard=shard1
> [junit4] 2> 446795 T2551
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
> [junit4] 2> 446899 T2549 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> [junit4] 2> 446899 T2555 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> [junit4] 2> 447288 T2522 oascc.SolrZkClient.makePath makePath:
> /collections/collection1/leader_elect/shard1/election
> [junit4] 2> 447298 T2555 oascc.ZkStateReader$3.process Updating live
> nodes... (0)
> [junit4] 2> 447301 T2522 oasc.ZkTestServer.send4LetterWord connecting
> to 127.0.0.1:57742 57742
> [junit4] 2> 447869 T2539 oasc.ZkTestServer.send4LetterWord connecting
> to 127.0.0.1:57742 57742
> [junit4] 2> 447871 T2522 oas.SolrTestCaseJ4.tearDown ###Ending
> testShardAssignment
> [junit4] 2> 447872 T2552 oasc.Overseer$ClusterStateUpdater.run
> Overseer Loop exiting : 127.0.0.1:57742_solr
> [junit4] 2> 447875 T2556
> oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not
> read the data org.apache.zookeeper.KeeperException$SessionExpiredException:
> KeeperErrorCode = Session expired for /overseer_elect/leader
> [junit4] 2> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
> [junit4] 2> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> [junit4] 2> at
> org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
> [junit4] 2> at
> org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
> [junit4] 2> at
> org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
> [junit4] 2> at
> org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
> [junit4] 2> at
> org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
> [junit4] 2> at
> org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:316)
> [junit4] 2> at
> org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
> [junit4] 2> at
> org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:288)
> [junit4] 2>
> [junit4] 2> 447881 T2522 oas.SolrTestCaseJ4.setUp ###Starting
> testShardAssignmentBigger
> [junit4] 2> 447882 T2522 oasc.ZkTestServer.run STARTING ZK TEST SERVER
> [junit4] 1> client port:0.0.0.0/0.0.0.0:0
> [junit4] 2> 447883 T2557 oasc.ZkTestServer$ZKServerMain.runFromConfig
> Starting server
> [junit4] 2> 447983 T2522 oasc.ZkTestServer.run start zk server on
> port:50146
> [junit4] 2> 447984 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 447988 T2563 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@daa195
> name:ZooKeeperConnection Watcher:127.0.0.1:50146 got event WatchedEvent
> state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 447989 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 447993 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 447994 T2565 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@f8aac
> name:ZooKeeperConnection Watcher:127.0.0.1:50146 got event WatchedEvent
> state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 447995 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 447995 T2522 oascc.SolrZkClient.makePath makePath: /solr
> [junit4] 2> 448000 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 448002 T2567 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@141291c
> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 448002 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 448003 T2522 oascc.SolrZkClient.makePath makePath:
> /live_nodes
> [junit4] 2> 448006 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 448008 T2569 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@49f99d
> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 448008 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 448009 T2522 oascc.SolrZkClient.makePath makePath:
> /overseer_elect
> [junit4] 2> 448012 T2522 oascc.SolrZkClient.makePath makePath:
> /overseer_elect/election
> [junit4] 2> 448018 T2522
> oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader
> 127.0.0.1:50146_solr
> [junit4] 2> 448018 T2522 oascc.SolrZkClient.makePath makePath:
> /overseer_elect/leader
> [junit4] 2> 448021 T2522 oasc.Overseer.start Overseer
> (id=92053396176633859-127.0.0.1:50146_solr-n_0000000000) starting
> [junit4] 2> 448024 T2522 oascc.SolrZkClient.makePath makePath:
> /overseer/queue
> [junit4] 2> 448029 T2522 oascc.SolrZkClient.makePath makePath:
> /overseer/queue-work
> [junit4] 2> 448034 T2522 oascc.SolrZkClient.makePath makePath:
> /overseer/collection-map-failure
> [junit4] 2> 448038 T2522 oascc.SolrZkClient.makePath makePath:
> /overseer/collection-map-running
> [junit4] 2> 448043 T2522 oascc.SolrZkClient.makePath makePath:
> /overseer/collection-map-completed
> [junit4] 2> 448048 T2522 oascc.SolrZkClient.makePath makePath:
> /overseer/collection-queue-work
> [junit4] 2> 448055 T2571 oasc.OverseerCollectionProcessor.run Process
> current queue of collection creations
> [junit4] 2> 448056 T2522 oascc.SolrZkClient.makePath makePath:
> /clusterstate.json
> [junit4] 2> 448056 T2570 oasc.Overseer$ClusterStateUpdater.run
> Starting to work on the main queue
> [junit4] 2> 448058 T2522 oascc.SolrZkClient.makePath makePath:
> /aliases.json
> [junit4] 2> 448061 T2522
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
> [junit4] 2> 448065 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 448066 T2573 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@1242e08
> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 448067 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 448069 T2522
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
> [junit4] 2> 448072 T2522 oascc.SolrZkClient.makePath makePath:
> /live_nodes/node0
> [junit4] 2> 448076 T2573 oascc.ZkStateReader$3.process Updating live
> nodes... (1)
> [junit4] 2> 448076 T2567 oascc.ZkStateReader$3.process Updating live
> nodes... (1)
> [junit4] 2> 448077 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 448078 T2575 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@1fc812a
> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 448078 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 448080 T2522
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
> [junit4] 2> 448083 T2522 oascc.SolrZkClient.makePath makePath:
> /live_nodes/node1
> [junit4] 2> 448087 T2573 oascc.ZkStateReader$3.process Updating live
> nodes... (2)
> [junit4] 2> 448087 T2567 oascc.ZkStateReader$3.process Updating live
> nodes... (2)
> [junit4] 2> 448087 T2575 oascc.ZkStateReader$3.process Updating live
> nodes... (2)
> [junit4] 2> 448088 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 448089 T2577 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@1714ec0
> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 448090 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 448091 T2522
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
> [junit4] 2> 448095 T2522 oascc.SolrZkClient.makePath makePath:
> /live_nodes/node2
> [junit4] 2> 448099 T2573 oascc.ZkStateReader$3.process Updating live
> nodes... (3)
> [junit4] 2> 448099 T2577 oascc.ZkStateReader$3.process Updating live
> nodes... (3)
> [junit4] 2> 448099 T2575 oascc.ZkStateReader$3.process Updating live
> nodes... (3)
> [junit4] 2> 448100 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 448099 T2567 oascc.ZkStateReader$3.process Updating live
> nodes... (3)
> [junit4] 2> 448102 T2579 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@fbc5e2
> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 448102 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 448104 T2522
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
> [junit4] 2> 448107 T2522 oascc.SolrZkClient.makePath makePath:
> /live_nodes/node3
> [junit4] 2> 448112 T2577 oascc.ZkStateReader$3.process Updating live
> nodes... (4)
> [junit4] 2> 448112 T2575 oascc.ZkStateReader$3.process Updating live
> nodes... (4)
> [junit4] 2> 448112 T2567 oascc.ZkStateReader$3.process Updating live
> nodes... (4)
> [junit4] 2> 448112 T2573 oascc.ZkStateReader$3.process Updating live
> nodes... (4)
> [junit4] 2> 448113 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 448113 T2579 oascc.ZkStateReader$3.process Updating live
> nodes... (4)
> [junit4] 2> 448115 T2581 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@75e1db
> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 448115 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 448117 T2522
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
> [junit4] 2> 448121 T2522 oascc.SolrZkClient.makePath makePath:
> /live_nodes/node4
> [junit4] 2> 448126 T2577 oascc.ZkStateReader$3.process Updating live
> nodes... (5)
> [junit4] 2> 448126 T2575 oascc.ZkStateReader$3.process Updating live
> nodes... (5)
> [junit4] 2> 448126 T2573 oascc.ZkStateReader$3.process Updating live
> nodes... (5)
> [junit4] 2> 448127 T2581 oascc.ZkStateReader$3.process Updating live
> nodes... (5)
> [junit4] 2> 448126 T2567 oascc.ZkStateReader$3.process Updating live
> nodes... (5)
> [junit4] 2> 448126 T2579 oascc.ZkStateReader$3.process Updating live
> nodes... (5)
> [junit4] 2> 448127 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 448129 T2583 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@11dc02e
> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 448130 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 448132 T2522
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
> [junit4] 2> 448136 T2522 oascc.SolrZkClient.makePath makePath:
> /live_nodes/node5
> [junit4] 2> 448139 T2575 oascc.ZkStateReader$3.process Updating live
> nodes... (6)
> [junit4] 2> 448140 T2577 oascc.ZkStateReader$3.process Updating live
> nodes... (6)
> [junit4] 2> 448140 T2581 oascc.ZkStateReader$3.process Updating live
> nodes... (6)
> [junit4] 2> 448140 T2573 oascc.ZkStateReader$3.process Updating live
> nodes... (6)
> [junit4] 2> 448140 T2579 oascc.ZkStateReader$3.process Updating live
> nodes... (6)
> [junit4] 2> 448141 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 448141 T2583 oascc.ZkStateReader$3.process Updating live
> nodes... (6)
> [junit4] 2> 448140 T2567 oascc.ZkStateReader$3.process Updating live
> nodes... (6)
> [junit4] 2> 448143 T2585 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@6d9f03
> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 448143 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 448145 T2522
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
> [junit4] 2> 448148 T2522 oascc.SolrZkClient.makePath makePath:
> /live_nodes/node6
> [junit4] 2> 448153 T2575 oascc.ZkStateReader$3.process Updating live
> nodes... (7)
> [junit4] 2> 448153 T2577 oascc.ZkStateReader$3.process Updating live
> nodes... (7)
> [junit4] 2> 448153 T2573 oascc.ZkStateReader$3.process Updating live
> nodes... (7)
> [junit4] 2> 448153 T2585 oascc.ZkStateReader$3.process Updating live
> nodes... (7)
> [junit4] 2> 448153 T2583 oascc.ZkStateReader$3.process Updating live
> nodes... (7)
> [junit4] 2> 448153 T2579 oascc.ZkStateReader$3.process Updating live
> nodes... (7)
> [junit4] 2> 448153 T2581 oascc.ZkStateReader$3.process Updating live
> nodes... (7)
> [junit4] 2> 448153 T2567 oascc.ZkStateReader$3.process Updating live
> nodes... (7)
> [junit4] 2> 448154 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 448156 T2587 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@958679
> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 448157 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 448159 T2522
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
> [junit4] 2> 448162 T2522 oascc.SolrZkClient.makePath makePath:
> /live_nodes/node7
> [junit4] 2> 448167 T2567 oascc.ZkStateReader$3.process Updating live
> nodes... (8)
> [junit4] 2> 448167 T2577 oascc.ZkStateReader$3.process Updating live
> nodes... (8)
> [junit4] 2> 448168 T2585 oascc.ZkStateReader$3.process Updating live
> nodes... (8)
> [junit4] 2> 448168 T2583 oascc.ZkStateReader$3.process Updating live
> nodes... (8)
> [junit4] 2> 448168 T2581 oascc.ZkStateReader$3.process Updating live
> nodes... (8)
> [junit4] 2> 448168 T2579 oascc.ZkStateReader$3.process Updating live
> nodes... (8)
> [junit4] 2> 448168 T2573 oascc.ZkStateReader$3.process Updating live
> nodes... (8)
> [junit4] 2> 448167 T2587 oascc.ZkStateReader$3.process Updating live
> nodes... (8)
> [junit4] 2> 448167 T2575 oascc.ZkStateReader$3.process Updating live
> nodes... (8)
> [junit4] 2> 448169 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 448171 T2589 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@169d949
> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 448171 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 448174 T2522
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
> [junit4] 2> 448177 T2522 oascc.SolrZkClient.makePath makePath:
> /live_nodes/node8
> [junit4] 2> 448182 T2589 oascc.ZkStateReader$3.process Updating live
> nodes... (9)
> [junit4] 2> 448183 T2585 oascc.ZkStateReader$3.process Updating live
> nodes... (9)
> [junit4] 2> 448183 T2583 oascc.ZkStateReader$3.process Updating live
> nodes... (9)
> [junit4] 2> 448183 T2587 oascc.ZkStateReader$3.process Updating live
> nodes... (9)
> [junit4] 2> 448183 T2573 oascc.ZkStateReader$3.process Updating live
> nodes... (9)
> [junit4] 2> 448183 T2579 oascc.ZkStateReader$3.process Updating live
> nodes... (9)
> [junit4] 2> 448183 T2581 oascc.ZkStateReader$3.process Updating live
> nodes... (9)
> [junit4] 2> 448183 T2575 oascc.ZkStateReader$3.process Updating live
> nodes... (9)
> [junit4] 2> 448183 T2577 oascc.ZkStateReader$3.process Updating live
> nodes... (9)
> [junit4] 2> 448182 T2567 oascc.ZkStateReader$3.process Updating live
> nodes... (9)
> [junit4] 2> 448185 T2591 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@2e78fa
> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 448184 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 448188 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 448190 T2522
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
> [junit4] 2> 448192 T2522 oascc.SolrZkClient.makePath makePath:
> /live_nodes/node9
> [junit4] 2> 448195 T2589 oascc.ZkStateReader$3.process Updating live
> nodes... (10)
> [junit4] 2> 448195 T2577 oascc.ZkStateReader$3.process Updating live
> nodes... (10)
> [junit4] 2> 448195 T2567 oascc.ZkStateReader$3.process Updating live
> nodes... (10)
> [junit4] 2> 448196 T2573 oascc.ZkStateReader$3.process Updating live
> nodes... (10)
> [junit4] 2> 448195 T2575 oascc.ZkStateReader$3.process Updating live
> nodes... (10)
> [junit4] 2> 448195 T2587 oascc.ZkStateReader$3.process Updating live
> nodes... (10)
> [junit4] 2> 448195 T2579 oascc.ZkStateReader$3.process Updating live
> nodes... (10)
> [junit4] 2> 448195 T2583 oascc.ZkStateReader$3.process Updating live
> nodes... (10)
> [junit4] 2> 448196 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 448198 T2593 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@156c3b
> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 448196 T2591 oascc.ZkStateReader$3.process Updating live
> nodes... (10)
> [junit4] 2> 448196 T2585 oascc.ZkStateReader$3.process Updating live
> nodes... (10)
> [junit4] 2> 448196 T2581 oascc.ZkStateReader$3.process Updating live
> nodes... (10)
> [junit4] 2> 448198 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 448200 T2522
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
> [junit4] 2> 448203 T2522 oascc.SolrZkClient.makePath makePath:
> /live_nodes/node10
> [junit4] 2> 448207 T2575 oascc.ZkStateReader$3.process Updating live
> nodes... (11)
> [junit4] 2> 448207 T2583 oascc.ZkStateReader$3.process Updating live
> nodes... (11)
> [junit4] 2> 448208 T2579 oascc.ZkStateReader$3.process Updating live
> nodes... (11)
> [junit4] 2> 448207 T2577 oascc.ZkStateReader$3.process Updating live
> nodes... (11)
> [junit4] 2> 448207 T2573 oascc.ZkStateReader$3.process Updating live
> nodes... (11)
> [junit4] 2> 448207 T2589 oascc.ZkStateReader$3.process Updating live
> nodes... (11)
> [junit4] 2> 448207 T2587 oascc.ZkStateReader$3.process Updating live
> nodes... (11)
> [junit4] 2> 448209 T2585 oascc.ZkStateReader$3.process Updating live
> nodes... (11)
> [junit4] 2> 448208 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 448208 T2581 oascc.ZkStateReader$3.process Updating live
> nodes... (11)
> [junit4] 2> 448208 T2593 oascc.ZkStateReader$3.process Updating live
> nodes... (11)
> [junit4] 2> 448208 T2591 oascc.ZkStateReader$3.process Updating live
> nodes... (11)
> [junit4] 2> 448208 T2567 oascc.ZkStateReader$3.process Updating live
> nodes... (11)
> [junit4] 2> 448211 T2595 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@f1978e
> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 448212 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 448213 T2522
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
> [junit4] 2> 448216 T2522 oascc.SolrZkClient.makePath makePath:
> /live_nodes/node11
> [junit4] 2> 448220 T2587 oascc.ZkStateReader$3.process Updating live
> nodes... (12)
> [junit4] 2> 448220 T2567 oascc.ZkStateReader$3.process Updating live
> nodes... (12)
> [junit4] 2> 448221 T2583 oascc.ZkStateReader$3.process Updating live
> nodes... (12)
> [junit4] 2> 448220 T2589 oascc.ZkStateReader$3.process Updating live
> nodes... (12)
> [junit4] 2> 448222 T2595 oascc.ZkStateReader$3.process Updating live
> nodes... (12)
> [junit4] 2> 448220 T2575 oascc.ZkStateReader$3.process Updating live
> nodes... (12)
> [junit4] 2> 448222 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 448222 T2593 oascc.ZkStateReader$3.process Updating live
> nodes... (12)
> [junit4] 2> 448222 T2581 oascc.ZkStateReader$3.process Updating live
> nodes... (12)
> [junit4] 2> 448224 T2597 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@175be88
> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 448221 T2591 oascc.ZkStateReader$3.process Updating live
> nodes... (12)
> [junit4] 2> 448221 T2573 oascc.ZkStateReader$3.process Updating live
> nodes... (12)
> [junit4] 2> 448221 T2577 oascc.ZkStateReader$3.process Updating live
> nodes... (12)
> [junit4] 2> 448221 T2585 oascc.ZkStateReader$3.process Updating live
> nodes... (12)
> [junit4] 2> 448221 T2579 oascc.ZkStateReader$3.process Updating live
> nodes... (12)
> [junit4] 2> 448224 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 448228 T2522
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
> [junit4] 2> 448232 T2522 oascc.SolrZkClient.makePath makePath:
> /live_nodes/node12
> [junit4] 2> 448236 T2573 oascc.ZkStateReader$3.process Updating live
> nodes... (13)
> [junit4] 2> 448236 T2577 oascc.ZkStateReader$3.process Updating live
> nodes... (13)
> [junit4] 2> 448237 T2595 oascc.ZkStateReader$3.process Updating live
> nodes... (13)
> [junit4] 2> 448237 T2593 oascc.ZkStateReader$3.process Updating live
> nodes... (13)
> [junit4] 2> 448237 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 448238 T2581 oascc.ZkStateReader$3.process Updating live
> nodes... (13)
> [junit4] 2> 448238 T2597 oascc.ZkStateReader$3.process Updating live
> nodes... (13)
> [junit4] 2> 448237 T2583 oascc.ZkStateReader$3.process Updating live
> nodes... (13)
> [junit4] 2> 448239 T2599 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@dde0f4
> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 448237 T2589 oascc.ZkStateReader$3.process Updating live
> nodes... (13)
> [junit4] 2> 448237 T2575 oascc.ZkStateReader$3.process Updating live
> nodes... (13)
> [junit4] 2> 448237 T2587 oascc.ZkStateReader$3.process Updating live
> nodes... (13)
> [junit4] 2> 448237 T2585 oascc.ZkStateReader$3.process Updating live
> nodes... (13)
> [junit4] 2> 448236 T2567 oascc.ZkStateReader$3.process Updating live
> nodes... (13)
> [junit4] 2> 448236 T2579 oascc.ZkStateReader$3.process Updating live
> nodes... (13)
> [junit4] 2> 448236 T2591 oascc.ZkStateReader$3.process Updating live
> nodes... (13)
> [junit4] 2> 448240 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 448244 T2522
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
> [junit4] 2> 448248 T2522 oascc.SolrZkClient.makePath makePath:
> /live_nodes/node13
> [junit4] 2> 448254 T2587 oascc.ZkStateReader$3.process Updating live
> nodes... (14)
> [junit4] 2> 448255 T2577 oascc.ZkStateReader$3.process Updating live
> nodes... (14)
> [junit4] 2> 448255 T2579 oascc.ZkStateReader$3.process Updating live
> nodes... (14)
> [junit4] 2> 448255 T2595 oascc.ZkStateReader$3.process Updating live
> nodes... (14)
> [junit4] 2> 448255 T2573 oascc.ZkStateReader$3.process Updating live
> nodes... (14)
> [junit4] 2> 448256 T2597 oascc.ZkStateReader$3.process Updating live
> nodes... (14)
> [junit4] 2> 448256 T2599 oascc.ZkStateReader$3.process Updating live
> nodes... (14)
> [junit4] 2> 448256 T2593 oascc.ZkStateReader$3.process Updating live
> nodes... (14)
> [junit4] 2> 448256 T2585 oascc.ZkStateReader$3.process Updating live
> nodes... (14)
> [junit4] 2> 448256 T2581 oascc.ZkStateReader$3.process Updating live
> nodes... (14)
> [junit4] 2> 448256 T2583 oascc.ZkStateReader$3.process Updating live
> nodes... (14)
> [junit4] 2> 448256 T2567 oascc.ZkStateReader$3.process Updating live
> nodes... (14)
> [junit4] 2> 448256 T2575 oascc.ZkStateReader$3.process Updating live
> nodes... (14)
> [junit4] 2> 448256 T2589 oascc.ZkStateReader$3.process Updating live
> nodes... (14)
> [junit4] 2> 448256 T2591 oascc.ZkStateReader$3.process Updating live
> nodes... (14)
> [junit4] 2> 448259 T2601 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@19e3760
> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 448257 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 448262 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 448264 T2522
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
> [junit4] 2> 448268 T2522 oascc.SolrZkClient.makePath makePath:
> /live_nodes/node14
> [junit4] 2> 448273 T2573 oascc.ZkStateReader$3.process Updating live
> nodes... (15)
> [junit4] 2> 448273 T2587 oascc.ZkStateReader$3.process Updating live
> nodes... (15)
> [junit4] 2> 448273 T2579 oascc.ZkStateReader$3.process Updating live
> nodes... (15)
> [junit4] 2> 448273 T2591 oascc.ZkStateReader$3.process Updating live
> nodes... (15)
> [junit4] 2> 448273 T2577 oascc.ZkStateReader$3.process Updating live
> nodes... (15)
> [junit4] 2> 448273 T2595 oascc.ZkStateReader$3.process Updating live
> nodes... (15)
> [junit4] 2> 448275 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 448275 T2599 oascc.ZkStateReader$3.process Updating live
> nodes... (15)
> [junit4] 2> 448275 T2583 oascc.ZkStateReader$3.process Updating live
> nodes... (15)
> [junit4] 2> 448275 T2585 oascc.ZkStateReader$3.process Updating live
> nodes... (15)
> [junit4] 2> 448275 T2597 oascc.ZkStateReader$3.process Updating live
> nodes... (15)
> [junit4] 2> 448275 T2601 oascc.ZkStateReader$3.process Updating live
> nodes... (15)
> [junit4] 2> 448275 T2567 oascc.ZkStateReader$3.process Updating live
> nodes... (15)
> [junit4] 2> 448275 T2575 oascc.ZkStateReader$3.process Updating live
> nodes... (15)
> [junit4] 2> 448275 T2581 oascc.ZkStateReader$3.process Updating live
> nodes... (15)
> [junit4] 2> 448274 T2593 oascc.ZkStateReader$3.process Updating live
> nodes... (15)
> [junit4] 2> 448273 T2589 oascc.ZkStateReader$3.process Updating live
> nodes... (15)
> [junit4] 2> 448277 T2603 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@1818cfa
> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 448281 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 448283 T2522
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
> [junit4] 2> 448286 T2522 oascc.SolrZkClient.makePath makePath:
> /live_nodes/node15
> [junit4] 2> 448291 T2577 oascc.ZkStateReader$3.process Updating live
> nodes... (16)
> [junit4] 2> 448292 T2573 oascc.ZkStateReader$3.process Updating live
> nodes... (16)
> [junit4] 2> 448291 T2587 oascc.ZkStateReader$3.process Updating live
> nodes... (16)
> [junit4] 2> 448293 T2579 oascc.ZkStateReader$3.process Updating live
> nodes... (16)
> [junit4] 2> 448293 T2575 oascc.ZkStateReader$3.process Updating live
> nodes... (16)
> [junit4] 2> 448293 T2601 oascc.ZkStateReader$3.process Updating live
> nodes... (16)
> [junit4] 2> 448294 T2599 oascc.ZkStateReader$3.process Updating live
> nodes... (16)
> [junit4] 2> 448293 T2591 oascc.ZkStateReader$3.process Updating live
> nodes... (16)
> [junit4] 2> 448293 T2597 oascc.ZkStateReader$3.process Updating live
> nodes... (16)
> [junit4] 2> 448293 T2583 oascc.ZkStateReader$3.process Updating live
> nodes... (16)
> [junit4] 2> 448293 T2567 oascc.ZkStateReader$3.process Updating live
> nodes... (16)
> [junit4] 2> 448293 T2589 oascc.ZkStateReader$3.process Updating live
> nodes... (16)
> [junit4] 2> 448295 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 448294 T2603 oascc.ZkStateReader$3.process Updating live
> nodes... (16)
> [junit4] 2> 448294 T2581 oascc.ZkStateReader$3.process Updating live
> nodes... (16)
> [junit4] 2> 448294 T2585 oascc.ZkStateReader$3.process Updating live
> nodes... (16)
> [junit4] 2> 448294 T2595 oascc.ZkStateReader$3.process Updating live
> nodes... (16)
> [junit4] 2> 448294 T2593 oascc.ZkStateReader$3.process Updating live
> nodes... (16)
> [junit4] 2> 448297 T2605 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@154e83c
> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 448299 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 448300 T2522
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
> [junit4] 2> 448304 T2522 oascc.SolrZkClient.makePath makePath:
> /live_nodes/node16
> [junit4] 2> 448313 T2587 oascc.ZkStateReader$3.process Updating live
> nodes... (17)
> [junit4] 2> 448313 T2589 oascc.ZkStateReader$3.process Updating live
> nodes... (17)
> [junit4] 2> 448314 T2595 oascc.ZkStateReader$3.process Updating live
> nodes... (17)
> [junit4] 2> 448315 T2575 oascc.ZkStateReader$3.process Updating live
> nodes... (17)
> [junit4] 2> 448314 T2585 oascc.ZkStateReader$3.process Updating live
> nodes... (17)
> [junit4] 2> 448316 T2591 oascc.ZkStateReader$3.process Updating live
> nodes... (17)
> [junit4] 2> 448313 T2573 oascc.ZkStateReader$3.process Updating live
> nodes... (17)
> [junit4] 2> 448313 T2577 oascc.ZkStateReader$3.process Updating live
> nodes... (17)
> [junit4] 2> 448313 T2593 oascc.ZkStateReader$3.process Updating live
> nodes... (17)
> [junit4] 2> 448317 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 448316 T2599 oascc.ZkStateReader$3.process Updating live
> nodes... (17)
> [junit4] 2> 448316 T2581 oascc.ZkStateReader$3.process Updating live
> nodes... (17)
> [junit4] 2> 448316 T2605 oascc.ZkStateReader$3.process Updating live
> nodes... (17)
> [junit4] 2> 448316 T2603 oascc.ZkStateReader$3.process Updating live
> nodes... (17)
> [junit4] 2> 448316 T2583 oascc.ZkStateReader$3.process Updating live
> nodes... (17)
> [junit4] 2> 448316 T2567 oascc.ZkStateReader$3.process Updating live
> nodes... (17)
> [junit4] 2> 448315 T2601 oascc.ZkStateReader$3.process Updating live
> nodes... (17)
> [junit4] 2> 448315 T2597 oascc.ZkStateReader$3.process Updating live
> nodes... (17)
> [junit4] 2> 448315 T2579 oascc.ZkStateReader$3.process Updating live
> nodes... (17)
> [junit4] 2> 448319 T2607 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@d963a2
> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 448323 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 448324 T2522
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
> [junit4] 2> 448328 T2522 oascc.SolrZkClient.makePath makePath:
> /live_nodes/node17
> [junit4] 2> 448333 T2573 oascc.ZkStateReader$3.process Updating live
> nodes... (18)
> [junit4] 2> 448333 T2577 oascc.ZkStateReader$3.process Updating live
> nodes... (18)
> [junit4] 2> 448334 T2601 oascc.ZkStateReader$3.process Updating live
> nodes... (18)
> [junit4] 2> 448334 T2607 oascc.ZkStateReader$3.process Updating live
> nodes... (18)
> [junit4] 2> 448335 T2567 oascc.ZkStateReader$3.process Updating live
> nodes... (18)
> [junit4] 2> 448335 T2583 oascc.ZkStateReader$3.process Updating live
> nodes... (18)
> [junit4] 2> 448335 T2603 oascc.ZkStateReader$3.process Updating live
> nodes... (18)
> [junit4] 2> 448334 T2589 oascc.ZkStateReader$3.process Updating live
> nodes... (18)
> [junit4] 2> 448336 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 448334 T2595 oascc.ZkStateReader$3.process Updating live
> nodes... (18)
> [junit4] 2> 448334 T2579 oascc.ZkStateReader$3.process Updating live
> nodes... (18)
> [junit4] 2> 448334 T2575 oascc.ZkStateReader$3.process Updating live
> nodes... (18)
> [junit4] 2> 448338 T2609 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@826a2e
> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 448333 T2587 oascc.ZkStateReader$3.process Updating live
> nodes... (18)
> [junit4] 2> 448333 T2591 oascc.ZkStateReader$3.process Updating live
> nodes... (18)
> [junit4] 2> 448338 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 448336 T2585 oascc.ZkStateReader$3.process Updating live
> nodes... (18)
> [junit4] 2> 448335 T2593 oascc.ZkStateReader$3.process Updating live
> nodes... (18)
> [junit4] 2> 448335 T2599 oascc.ZkStateReader$3.process Updating live
> nodes... (18)
> [junit4] 2> 448335 T2581 oascc.ZkStateReader$3.process Updating live
> nodes... (18)
> [junit4] 2> 448335 T2605 oascc.ZkStateReader$3.process Updating live
> nodes... (18)
> [junit4] 2> 448335 T2597 oascc.ZkStateReader$3.process Updating live
> nodes... (18)
> [junit4] 2> 448341 T2522
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
> [junit4] 2> 448346 T2522 oascc.SolrZkClient.makePath makePath:
> /live_nodes/node18
> [junit4] 2> 448350 T2573 oascc.ZkStateReader$3.process Updating live
> nodes... (19)
> [junit4] 2> 448350 T2577 oascc.ZkStateReader$3.process Updating live
> nodes... (19)
> [junit4] 2> 448351 T2607 oascc.ZkStateReader$3.process Updating live
> nodes... (19)
> [junit4] 2> 448352 T2587 oascc.ZkStateReader$3.process Updating live
> nodes... (19)
> [junit4] 2> 448352 T2593 oascc.ZkStateReader$3.process Updating live
> nodes... (19)
> [junit4] 2> 448351 T2601 oascc.ZkStateReader$3.process Updating live
> nodes... (19)
> [junit4] 2> 448353 T2579 oascc.ZkStateReader$3.process Updating live
> nodes... (19)
> [junit4] 2> 448353 T2581 oascc.ZkStateReader$3.process Updating live
> nodes... (19)
> [junit4] 2> 448351 T2595 oascc.ZkStateReader$3.process Updating live
> nodes... (19)
> [junit4] 2> 448353 T2605 oascc.ZkStateReader$3.process Updating live
> nodes... (19)
> [junit4] 2> 448354 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 448353 T2609 oascc.ZkStateReader$3.process Updating live
> nodes... (19)
> [junit4] 2> 448353 T2603 oascc.ZkStateReader$3.process Updating live
> nodes... (19)
> [junit4] 2> 448353 T2597 oascc.ZkStateReader$3.process Updating live
> nodes... (19)
> [junit4] 2> 448353 T2589 oascc.ZkStateReader$3.process Updating live
> nodes... (19)
> [junit4] 2> 448353 T2567 oascc.ZkStateReader$3.process Updating live
> nodes... (19)
> [junit4] 2> 448353 T2591 oascc.ZkStateReader$3.process Updating live
> nodes... (19)
> [junit4] 2> 448353 T2575 oascc.ZkStateReader$3.process Updating live
> nodes... (19)
> [junit4] 2> 448352 T2599 oascc.ZkStateReader$3.process Updating live
> nodes... (19)
> [junit4] 2> 448352 T2583 oascc.ZkStateReader$3.process Updating live
> nodes... (19)
> [junit4] 2> 448352 T2585 oascc.ZkStateReader$3.process Updating live
> nodes... (19)
> [junit4] 2> 448356 T2611 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@1b5c498
> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 448359 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 448362 T2522
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
> [junit4] 2> 448366 T2522 oascc.SolrZkClient.makePath makePath:
> /live_nodes/node19
> [junit4] 2> 448371 T2587 oascc.ZkStateReader$3.process Updating live
> nodes... (20)
> [junit4] 2> 448371 T2591 oascc.ZkStateReader$3.process Updating live
> nodes... (20)
> [junit4] 2> 448372 T2579 oascc.ZkStateReader$3.process Updating live
> nodes... (20)
> [junit4] 2> 448372 T2575 oascc.ZkStateReader$3.process Updating live
> nodes... (20)
> [junit4] 2> 448372 T2577 oascc.ZkStateReader$3.process Updating live
> nodes... (20)
> [junit4] 2> 448373 T2581 oascc.ZkStateReader$3.process Updating live
> nodes... (20)
> [junit4] 2> 448372 T2589 oascc.ZkStateReader$3.process Updating live
> nodes... (20)
> [junit4] 2> 448372 T2573 oascc.ZkStateReader$3.process Updating live
> nodes... (20)
> [junit4] 2> 448372 T2595 oascc.ZkStateReader$3.process Updating live
> nodes... (20)
> [junit4] 2> 448374 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 448373 T2609 oascc.ZkStateReader$3.process Updating live
> nodes... (20)
> [junit4] 2> 448373 T2611 oascc.ZkStateReader$3.process Updating live
> nodes... (20)
> [junit4] 2> 448373 T2603 oascc.ZkStateReader$3.process Updating live
> nodes... (20)
> [junit4] 2> 448373 T2585 oascc.ZkStateReader$3.process Updating live
> nodes... (20)
> [junit4] 2> 448373 T2593 oascc.ZkStateReader$3.process Updating live
> nodes... (20)
> [junit4] 2> 448373 T2567 oascc.ZkStateReader$3.process Updating live
> nodes... (20)
> [junit4] 2> 448373 T2607 oascc.ZkStateReader$3.process Updating live
> nodes... (20)
> [junit4] 2> 448372 T2599 oascc.ZkStateReader$3.process Updating live
> nodes... (20)
> [junit4] 2> 448372 T2597 oascc.ZkStateReader$3.process Updating live
> nodes... (20)
> [junit4] 2> 448372 T2601 oascc.ZkStateReader$3.process Updating live
> nodes... (20)
> [junit4] 2> 448372 T2583 oascc.ZkStateReader$3.process Updating live
> nodes... (20)
> [junit4] 2> 448372 T2605 oascc.ZkStateReader$3.process Updating live
> nodes... (20)
> [junit4] 2> 448376 T2613 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@18ed79a
> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 448380 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 448382 T2522
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
> [junit4] 2> 448385 T2522 oascc.SolrZkClient.makePath makePath:
> /live_nodes/node20
> [junit4] 2> 448390 T2577 oascc.ZkStateReader$3.process Updating live
> nodes... (21)
> [junit4] 2> 448390 T2579 oascc.ZkStateReader$3.process Updating live
> nodes... (21)
> [junit4] 2> 448391 T2589 oascc.ZkStateReader$3.process Updating live
> nodes... (21)
> [junit4] 2> 448391 T2601 oascc.ZkStateReader$3.process Updating live
> nodes... (21)
> [junit4] 2> 448391 T2599 oascc.ZkStateReader$3.process Updating live
> nodes... (21)
> [junit4] 2> 448391 T2573 oascc.ZkStateReader$3.process Updating live
> nodes... (21)
> [junit4] 2> 448392 T2581 oascc.ZkStateReader$3.process Updating live
> nodes... (21)
> [junit4] 2> 448391 T2605 oascc.ZkStateReader$3.process Updating live
> nodes... (21)
> [junit4] 2> 448390 T2591 oascc.ZkStateReader$3.process Updating live
> nodes... (21)
> [junit4] 2> 448390 T2595 oascc.ZkStateReader$3.process Updating live
> nodes... (21)
> [junit4] 2> 448393 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 448392 T2603 oascc.ZkStateReader$3.process Updating live
> nodes... (21)
> [junit4] 2> 448392 T2609 oascc.ZkStateReader$3.process Updating live
> nodes... (21)
> [junit4] 2> 448392 T2611 oascc.ZkStateReader$3.process Updating live
> nodes... (21)
> [junit4] 2> 448392 T2593 oascc.ZkStateReader$3.process Updating live
> nodes... (21)
> [junit4] 2> 448392 T2585 oascc.ZkStateReader$3.process Updating live
> nodes... (21)
> [junit4] 2> 448392 T2567 oascc.ZkStateReader$3.process Updating live
> nodes... (21)
> [junit4] 2> 448392 T2575 oascc.ZkStateReader$3.process Updating live
> nodes... (21)
> [junit4] 2> 448391 T2613 oascc.ZkStateReader$3.process Updating live
> nodes... (21)
> [junit4] 2> 448391 T2583 oascc.ZkStateReader$3.process Updating live
> nodes... (21)
> [junit4] 2> 448391 T2607 oascc.ZkStateReader$3.process Updating live
> nodes... (21)
> [junit4] 2> 448391 T2597 oascc.ZkStateReader$3.process Updating live
> nodes... (21)
> [junit4] 2> 448391 T2587 oascc.ZkStateReader$3.process Updating live
> nodes... (21)
> [junit4] 2> 448395 T2615 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@5f6bba
> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 448399 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 448401 T2522
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
> [junit4] 2> 448404 T2522 oascc.SolrZkClient.makePath makePath:
> /live_nodes/node21
> [junit4] 2> 448409 T2587 oascc.ZkStateReader$3.process Updating live
> nodes... (22)
> [junit4] 2> 448410 T2573 oascc.ZkStateReader$3.process Updating live
> nodes... (22)
> [junit4] 2> 448410 T2595 oascc.ZkStateReader$3.process Updating live
> nodes... (22)
> [junit4] 2> 448410 T2577 oascc.ZkStateReader$3.process Updating live
> nodes... (22)
> [junit4] 2> 448409 T2591 oascc.ZkStateReader$3.process Updating live
> nodes... (22)
> [junit4] 2> 448412 T2607 oascc.ZkStateReader$3.process Updating live
> nodes... (22)
> [junit4] 2> 448412 T2593 oascc.ZkStateReader$3.process Updating live
> nodes... (22)
> [junit4] 2> 448412 T2599 oascc.ZkStateReader$3.process Updating live
> nodes... (22)
> [junit4] 2> 448413 T2603 oascc.ZkStateReader$3.process Updating live
> nodes... (22)
> [junit4] 2> 448413 T2609 oascc.ZkStateReader$3.process Updating live
> nodes... (22)
> [junit4] 2> 448412 T2567 oascc.ZkStateReader$3.process Updating live
> nodes... (22)
> [junit4] 2> 448412 T2613 oascc.ZkStateReader$3.process Updating live
> nodes... (22)
> [junit4] 2> 448412 T2575 oascc.ZkStateReader$3.process Updating live
> nodes... (22)
> [junit4] 2> 448411 T2583 oascc.ZkStateReader$3.process Updating live
> nodes... (22)
> [junit4] 2> 448411 T2601 oascc.ZkStateReader$3.process Updating live
> nodes... (22)
> [junit4] 2> 448411 T2597 oascc.ZkStateReader$3.process Updating live
> nodes... (22)
> [junit4] 2> 448411 T2579 oascc.ZkStateReader$3.process Updating live
> nodes... (22)
> [junit4] 2> 448411 T2605 oascc.ZkStateReader$3.process Updating live
> nodes... (22)
> [junit4] 2> 448411 T2589 oascc.ZkStateReader$3.process Updating live
> nodes... (22)
> [junit4] 2> 448416 T2617 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@13fabc5
> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 448414 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 448413 T2615 oascc.ZkStateReader$3.process Updating live
> nodes... (22)
> [junit4] 2> 448413 T2611 oascc.ZkStateReader$3.process Updating live
> nodes... (22)
> [junit4] 2> 448412 T2581 oascc.ZkStateReader$3.process Updating live
> nodes... (22)
> [junit4] 2> 448412 T2585 oascc.ZkStateReader$3.process Updating live
> nodes... (22)
> [junit4] 2> 448418 T2522 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 448421 T2522
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
> [junit4] 2> 448426 T2522 oascc.SolrZkClient.makePath makePath:
> /live_nodes/node22
> [junit4] 2> 448431 T2587 oascc.ZkStateReader$3.process Updating live
> nodes... (23)
> [junit4] 2> 448432 T2577 oascc.ZkStateReader$3.process Updating live
> nodes... (23)
> [junit4] 2> 448432 T2579 oascc.ZkStateReader$3.process Updating live
> nodes... (23)
> [junit4] 2> 448432 T2605 oascc.ZkStateReader$3.process Updating live
> nodes... (23)
> [junit4] 2> 448433 T2611 oascc.ZkStateReader$3.process Updating live
> nodes... (23)
> [junit4] 2> 448432 T2595 oascc.ZkStateReader$3.process Updating live
> nodes... (23)
> [junit4] 2> 448432 T2591 oascc.ZkStateReader$3.process Updating live
> nodes... (23)
> [junit4] 2> 448434 T2609 oascc.ZkStateReader$3.process Updating live
> nodes... (23)
> [junit4] 2> 448432 T2573 oascc.ZkStateReader$3.process Updating live
> nodes... (23)
> [junit4] 2> 448435 T2522 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 448434 T2603 oascc.ZkStateReader$3.process Updating live
> nodes... (23)
> [junit4] 2> 448434 T2617 oascc.ZkStateReader$3.process Updating live
> nodes... (23)
> [junit4] 2> 448434 T2581 oascc.ZkStateReader$3.process Updating live
> nodes... (23)
> [junit4] 2> 448437 T2619 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@71217e
> name:ZooKeeperConnection Watcher:127.0.0.1:50146/solr got event
> WatchedEvent state:SyncConnected type:None path:null
>
> [...truncated too long message...]
>
> queue/qn-
> [junit4] 2> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
> [junit4] 2> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> [junit4] 2> at
> org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
> [junit4] 2> at
> org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:243)
> [junit4] 2> at
> org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:240)
> [junit4] 2> at
> org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
> [junit4] 2> at
> org.apache.solr.common.cloud.SolrZkClient.create(SolrZkClient.java:240)
> [junit4] 2> at
> org.apache.solr.cloud.DistributedQueue.createData(DistributedQueue.java:346)
> [junit4] 2> at
> org.apache.solr.cloud.DistributedQueue.offer(DistributedQueue.java:330)
> [junit4] 2> at
> org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:198)
> [junit4] 2> at
> org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:163)
> [junit4] 2> at
> org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:125)
> [junit4] 2> at
> org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:314)
> [junit4] 2> at
> org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:221)
> [junit4] 2> at
> org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:452)
> [junit4] 2> at
> org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:216)
> [junit4] 2> at
> org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:163)
> [junit4] 2> at
> org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:125)
> [junit4] 2> at
> org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
> [junit4] 2> at
> org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:358)
> [junit4] 2> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
> [junit4] 2> at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
> [junit4] 2>
> [junit4] 2> 750036 T3044 oascc.ZkStateReader$3.process WARN ZooKeeper
> watch triggered, but Solr cannot talk to ZK
> [junit4] 2> 750037 T3044 oasc.LeaderElector$ElectionWatcher.process
> WARN org.apache.zookeeper.KeeperException$SessionExpiredException:
> KeeperErrorCode = Session expired for /overseer_elect/election
> [junit4] 2> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
> [junit4] 2> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> [junit4] 2> at
> org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
> [junit4] 2> at
> org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
> [junit4] 2> at
> org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
> [junit4] 2> at
> org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
> [junit4] 2> at
> org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
> [junit4] 2> at
> org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
> [junit4] 2> at
> org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
> [junit4] 2> at
> org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:358)
> [junit4] 2> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
> [junit4] 2> at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
> [junit4] 2>
> [junit4] 2> 750038 T3044 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
> [junit4] 2> 750038 T3044 oascc.ZkStateReader$2.process WARN ZooKeeper
> watch triggered, but Solr cannot talk to ZK
> [junit4] 2> 750121 T3116 oascc.ZkStateReader$3.process WARN ZooKeeper
> watch triggered, but Solr cannot talk to ZK
> [junit4] 2> 750206 T2787 oascc.ZkStateReader$3.process WARN ZooKeeper
> watch triggered, but Solr cannot talk to ZK
> [junit4] 2> 750206 T2787
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
> [junit4] 2> 750207 T2787
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
> [junit4] 2> 750207 T2787
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
> [junit4] 2> 750207 T2787
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
> [junit4] 2> 750206 T3156
> oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not
> read the data org.apache.zookeeper.KeeperException$SessionExpiredException:
> KeeperErrorCode = Session expired for /overseer_elect/leader
> [junit4] 2> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
> [junit4] 2> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> [junit4] 2> at
> org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
> [junit4] 2> at
> org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
> [junit4] 2> at
> org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
> [junit4] 2> at
> org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
> [junit4] 2> at
> org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
> [junit4] 2> at
> org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:316)
> [junit4] 2> at
> org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
> [junit4] 2> at
> org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:288)
> [junit4] 2>
> [junit4] 2> NOTE: leaving temporary files on disk at:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-A819205877C572F6-001
> [junit4] 2> NOTE: test params are: codec=Lucene41,
> sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=ar_JO,
> timezone=Africa/Ndjamena
> [junit4] 2> NOTE: Linux 3.8.0-42-generic i386/Oracle Corporation
> 1.8.0_05 (32-bit)/cpus=8,threads=1,free=118175464,total=317980672
> [junit4] 2> NOTE: All tests run in this JVM: [TestRecovery,
> OpenCloseCoreStressTest, HardAutoCommitTest, DeleteReplicaTest,
> IndexSchemaRuntimeFieldTest, TestXIncludeConfig, TestFreeTextSuggestions,
> CollectionsAPIDistributedZkTest, TestFieldTypeResource, BadCopyFieldTest,
> CopyFieldTest, TestCSVLoader, UniqFieldsUpdateProcessorFactoryTest,
> TestIndexingPerformance, TestDocumentBuilder, AutoCommitTest,
> AtomicUpdatesTest, TestQueryTypes, TestCollapseQParserPlugin,
> BasicDistributedZk2Test, TestWriterPerf, TestCoreContainer,
> TestComplexPhraseQParserPlugin, TestDistribDocBasedVersion,
> TestFastLRUCache, TestDynamicFieldResource, DOMUtilTest,
> QueryResultKeyTest, DistribDocExpirationUpdateProcessorTest,
> UpdateRequestProcessorFactoryTest, TimeZoneUtilsTest,
> SchemaVersionSpecificBehaviorTest, PreAnalyzedUpdateProcessorTest,
> SOLR749Test, TestNonDefinedSimilarityFactory, TestFileDictionaryLookup,
> MultiThreadedOCPTest, TestRTGBase, QueryParsingTest, OutputWriterTest,
> LoggingHandlerTest, TestInfoStreamLogging, TestPseudoReturnFields,
> TestFastWriter, TestReplicationHandlerBackup, AssignTest,
> TermVectorComponentTest, SolrRequestParserTest, TestPerFieldSimilarity,
> TestAtomicUpdateErrorCases, TestManagedResource,
> TermVectorComponentDistributedTest, TestReRankQParserPlugin,
> TestSerializedLuceneMatchVersion, TestChildDocTransformer,
> TestNoOpRegenerator, TestGroupingSearch, TestFieldTypeCollectionResource,
> TestBinaryField, StatsComponentTest, CursorMarkTest, BasicZkTest,
> SuggesterFSTTest, BJQParserTest, CurrencyFieldXmlFileTest,
> DistributedQueryElevationComponentTest, TestDistributedMissingSort,
> SearchHandlerTest, TestLFUCache, TestSolrJ, FileBasedSpellCheckerTest,
> TestRealTimeGet, RequestHandlersTest, AnalysisAfterCoreReloadTest,
> SolrInfoMBeanTest, TestDistributedGrouping, TestExtendedDismaxParser,
> URLClassifyProcessorTest, SuggestComponentTest, ChaosMonkeySafeLeaderTest]
> [junit4] Completed on J1 in 74.71s, 1 test, 1 failure <<< FAILURES!
>
> [...truncated 59 lines...]
> BUILD FAILED
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:467: The
> following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:447: The
> following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:45: The
> following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The
> following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The
> following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:496:
> The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1299:
> The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:923:
> There were test failures: 405 suites, 1656 tests, 3 suite-level errors, 1
> error, 1 failure, 954 ignored (13 assumptions)
>
> Total time: 65 minutes 33 seconds
> Build step 'Invoke Ant' marked build as failure
> [description-setter] Description set: Java: 32bit/jdk1.8.0_05 -client
> -XX:+UseParallelGC
> Archiving artifacts
> Recording test results
> Email was triggered for: Failure - Any
> Sending email for trigger: Failure - Any
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
--
Regards,
Shalin Shekhar Mangar.