You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/05/19 12:35:36 UTC
[JENKINS] Lucene-Solr-4.8-Linux (32bit/jdk1.8.0_05) - Build # 198 -
Failure!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.8-Linux/198/
Java: 32bit/jdk1.8.0_05 -client -XX:+UseParallelGC
1 tests failed.
REGRESSION: org.apache.solr.cloud.OverseerTest.testShardLeaderChange
Error Message:
Unexpected shard leader coll:collection1 shard:shard1 expected:<core[1]> but was:<core[4]>
Stack Trace:
org.junit.ComparisonFailure: Unexpected shard leader coll:collection1 shard:shard1 expected:<core[1]> but was:<core[4]>
at __randomizedtesting.SeedInfo.seed([68EA34FC2C11F88D:B6B9B30B36890D7C]:0)
at org.junit.Assert.assertEquals(Assert.java:125)
at org.apache.solr.cloud.OverseerTest.verifyShardLeader(OverseerTest.java:596)
at org.apache.solr.cloud.OverseerTest.testShardLeaderChange(OverseerTest.java:755)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
at java.lang.Thread.run(Thread.java:745)
Build Log:
[...truncated 11083 lines...]
[junit4] Suite: org.apache.solr.cloud.OverseerTest
[junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/build/solr-core/test/J0/./solr.cloud.OverseerTest-68EA34FC2C11F88D-001/init-core-data-001
[junit4] 2> 350478 T1335 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (true)
[junit4] 2> 350479 T1335 oas.SolrTestCaseJ4.initCore ####initCore
[junit4] 2> 350479 T1335 oas.SolrTestCaseJ4.initCore ####initCore end
[junit4] 2> 350481 T1335 oas.SolrTestCaseJ4.setUp ###Starting testPlaceholders
[junit4] 2> 350482 T1335 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 350482 T1336 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 350582 T1335 oasc.ZkTestServer.run start zk server on port:55551
[junit4] 2> 350583 T1335 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 350633 T1342 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1446f name:ZooKeeperConnection Watcher:127.0.0.1:55551/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 350634 T1335 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 350635 T1335 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 350637 T1344 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@9053e8 name:ZooKeeperConnection Watcher:127.0.0.1:55551 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 350637 T1335 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 350639 T1335 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 350640 T1346 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ff843c name:ZooKeeperConnection Watcher:127.0.0.1:55551 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 350640 T1335 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 350640 T1335 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 350645 T1335 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 350646 T1335 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 350648 T1335 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 350649 T1335 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 350651 T1335 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 350652 T1348 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@15e841b name:ZooKeeperConnection Watcher:127.0.0.1:55551/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 350652 T1335 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 350653 T1335 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 350654 T1335 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
[junit4] 2> 350656 T1342 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4] 2> 350656 T1348 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4] 2> 350656 T1335 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 350657 T1350 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@11c9b63 name:ZooKeeperConnection Watcher:127.0.0.1:55551/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 350657 T1335 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 350658 T1335 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 350659 T1335 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 350661 T1335 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:55551_solr
[junit4] 2> 350661 T1335 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 350663 T1335 oasc.Overseer.start Overseer (id=91782813357178884-127.0.0.1:55551_solr-n_0000000000) starting
[junit4] 2> 350664 T1335 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 350665 T1335 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 350667 T1335 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 350669 T1335 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 350670 T1335 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 350673 T1335 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 350678 T1352 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 350678 T1351 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 350678 T1352 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes at 127.0.0.1:55551_solr
[junit4] 2> 350680 T1351 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 350680 T1351 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=12 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"12",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 350680 T1351 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3, shard4, shard5, shard6, shard7, shard8, shard9, shard10, shard11, shard12]
[junit4] 2> 350681 T1351 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 350682 T1350 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 350683 T1348 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 350683 T1342 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 351179 T1335 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 351186 T1335 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4] 2> 351190 T1350 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 351190 T1335 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 351191 T1351 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 351195 T1342 oascc.ZkStateReader$3.process Updating live nodes... (0)
[junit4] 2> 351195 T1348 oascc.ZkStateReader$3.process Updating live nodes... (0)
[junit4] 2> 351197 T1335 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:55551 55551
[junit4] 2> 351769 T1336 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:55551 55551
[junit4] 2> 351770 T1335 oas.SolrTestCaseJ4.tearDown ###Ending testPlaceholders
[junit4] 2> 351771 T1351 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:55551_solr
[junit4] 2> 351772 T1353 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:302)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:291)
[junit4] 2>
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 351775 T1335 oas.SolrTestCaseJ4.setUp ###Starting testDoubleAssignment
[junit4] 2> 351776 T1335 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 351776 T1354 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 351876 T1335 oasc.ZkTestServer.run start zk server on port:49896
[junit4] 2> 351877 T1335 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 351879 T1360 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@8e2418 name:ZooKeeperConnection Watcher:127.0.0.1:49896/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 351879 T1335 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 351880 T1335 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 351881 T1362 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@10d030e name:ZooKeeperConnection Watcher:127.0.0.1:49896 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 351882 T1335 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 351884 T1335 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 351885 T1364 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@475d0f name:ZooKeeperConnection Watcher:127.0.0.1:49896 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 351885 T1335 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 351886 T1335 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 351888 T1335 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 351890 T1335 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 351891 T1335 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 351892 T1335 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 351894 T1335 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 351895 T1366 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1848cae name:ZooKeeperConnection Watcher:127.0.0.1:49896/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 351895 T1335 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 351896 T1335 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 351897 T1335 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
[junit4] 2> 351898 T1360 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4] 2> 351898 T1366 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4] 2> 351899 T1335 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 351900 T1368 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@71c462 name:ZooKeeperConnection Watcher:127.0.0.1:49896/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 351900 T1335 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 351901 T1335 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 351902 T1335 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 351904 T1335 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:49896_solr
[junit4] 2> 351904 T1335 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 351906 T1335 oasc.Overseer.start Overseer (id=91782813441982468-127.0.0.1:49896_solr-n_0000000000) starting
[junit4] 2> 351907 T1335 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 351909 T1335 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 351911 T1335 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 351915 T1335 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 351917 T1335 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 351919 T1335 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 351927 T1370 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 351927 T1369 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 351928 T1370 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes at 127.0.0.1:49896_solr
[junit4] 2> 351929 T1369 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 351929 T1369 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 351930 T1369 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1]
[junit4] 2> 351930 T1369 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 351931 T1368 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 351932 T1360 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 351932 T1366 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 352428 T1335 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 352435 T1335 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 352439 T1368 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 352439 T1335 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 352440 T1369 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 352440 T1360 oascc.ZkStateReader$3.process Updating live nodes... (0)
[junit4] 2> 352440 T1366 oascc.ZkStateReader$3.process Updating live nodes... (0)
[junit4] 2> 352441 T1335 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 352443 T1368 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 352445 T1372 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1dbab51 name:ZooKeeperConnection Watcher:127.0.0.1:49896/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 352445 T1335 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 352447 T1335 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 352448 T1335 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
[junit4] 2> 352451 T1360 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4] 2> 352451 T1372 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4] 2> 352453 T1368 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 352453 T1335 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 352453 T1369 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 352454 T1369 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
[junit4] 2> 352456 T1368 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 352457 T1335 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 352460 T1368 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 352463 T1368 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 352564 T1360 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 352564 T1372 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 352564 T1335 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 352567 T1360 oascc.ZkStateReader$3.process Updating live nodes... (0)
[junit4] 2> 352567 T1372 oascc.ZkStateReader$3.process Updating live nodes... (0)
[junit4] 2> 352568 T1335 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:49896 49896
[junit4] 2> 353250 T1354 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:49896 49896
[junit4] 2> 353251 T1335 oas.SolrTestCaseJ4.tearDown ###Ending testDoubleAssignment
[junit4] 2> 353251 T1369 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:49896_solr
[junit4] 2> 353252 T1373 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:302)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:291)
[junit4] 2>
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 353255 T1335 oas.SolrTestCaseJ4.setUp ###Starting testStateChange
[junit4] 2> 353255 T1335 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 353256 T1374 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 353356 T1335 oasc.ZkTestServer.run start zk server on port:57577
[junit4] 2> 353357 T1335 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 353360 T1380 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1143fc0 name:ZooKeeperConnection Watcher:127.0.0.1:57577/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 353360 T1335 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 353361 T1335 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 353362 T1382 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@c03030 name:ZooKeeperConnection Watcher:127.0.0.1:57577 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 353362 T1335 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 353364 T1335 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 353365 T1384 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@56a29e name:ZooKeeperConnection Watcher:127.0.0.1:57577 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 353365 T1335 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 353366 T1335 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 353368 T1335 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 353369 T1335 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 353370 T1335 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 353371 T1335 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 353373 T1335 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 353374 T1386 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ff0c66 name:ZooKeeperConnection Watcher:127.0.0.1:57577/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 353374 T1335 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 353374 T1335 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 353376 T1335 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 353378 T1335 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:57577_solr
[junit4] 2> 353379 T1335 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 353380 T1335 oasc.Overseer.start Overseer (id=91782813538975747-127.0.0.1:57577_solr-n_0000000000) starting
[junit4] 2> 353381 T1335 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 353383 T1335 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 353384 T1335 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 353386 T1335 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 353387 T1335 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 353389 T1335 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 353393 T1388 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 353395 T1388 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes at 127.0.0.1:57577_solr
[junit4] 2> 353395 T1387 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 353396 T1335 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 353397 T1387 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 353397 T1387 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1/solr",
[junit4] 2> "node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "core":"core1",
[junit4] 2> "roles":"",
[junit4] 2> "state":"recovering"}
[junit4] 2> 353397 T1387 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 353399 T1386 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 353399 T1380 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 353447 T1335 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 353449 T1386 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 353451 T1387 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 353451 T1387 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1/solr",
[junit4] 2> "node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "core":"core1",
[junit4] 2> "roles":"",
[junit4] 2> "state":"active"}
[junit4] 2> 353452 T1387 oasc.Overseer$ClusterStateUpdater.updateState node=core_node1 is already registered
[junit4] 2> 353452 T1387 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
[junit4] 2> 353454 T1386 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 353556 T1380 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 353602 T1335 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:57577 57577
[junit4] 2> 354432 T1374 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:57577 57577
[junit4] 2> 354433 T1335 oas.SolrTestCaseJ4.tearDown ###Ending testStateChange
[junit4] 2> 354433 T1387 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:57577_solr
[junit4] 2> 354435 T1389 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:302)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:291)
[junit4] 2>
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 354438 T1335 oas.SolrTestCaseJ4.setUp ###Starting testShardAssignment
[junit4] 2> 354438 T1335 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 354439 T1390 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 354539 T1335 oasc.ZkTestServer.run start zk server on port:34547
[junit4] 2> 354540 T1335 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 354544 T1396 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@21d006 name:ZooKeeperConnection Watcher:127.0.0.1:34547 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 354545 T1335 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 354547 T1335 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 354548 T1398 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c12e82 name:ZooKeeperConnection Watcher:127.0.0.1:34547 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 354548 T1335 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 354548 T1335 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 354552 T1335 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 354554 T1400 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@522576 name:ZooKeeperConnection Watcher:127.0.0.1:34547/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 354554 T1335 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 354554 T1335 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 354556 T1335 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 354557 T1402 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7a8426 name:ZooKeeperConnection Watcher:127.0.0.1:34547/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 354557 T1335 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 354558 T1335 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 354560 T1335 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 354562 T1335 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:34547_solr
[junit4] 2> 354562 T1335 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 354564 T1335 oasc.Overseer.start Overseer (id=91782813616504835-127.0.0.1:34547_solr-n_0000000000) starting
[junit4] 2> 354565 T1335 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 354567 T1335 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 354569 T1335 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 354570 T1335 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 354572 T1335 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 354574 T1335 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 354577 T1404 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 354578 T1335 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 354578 T1404 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes at 127.0.0.1:34547_solr
[junit4] 2> 354578 T1403 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 354579 T1335 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 354580 T1335 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 354582 T1335 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 354583 T1406 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@add24f name:ZooKeeperConnection Watcher:127.0.0.1:34547/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 354584 T1335 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 354585 T1335 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 354587 T1335 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1
[junit4] 2> 354589 T1400 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4] 2> 354589 T1406 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4] 2> 354591 T1402 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 354592 T1403 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 354593 T1403 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 354593 T1403 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
[junit4] 2> 354593 T1403 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 354595 T1402 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 354595 T1400 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 354595 T1406 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 355091 T1335 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 355098 T1335 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4] 2> 355103 T1402 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 355106 T1402 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 355106 T1403 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 355109 T1403 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core2",
[junit4] 2> "core_node_name":"node2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 355110 T1403 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 355110 T1403 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
[junit4] 2> 355112 T1402 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 355215 T1406 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 355215 T1400 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 355606 T1335 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4] 2> 355611 T1335 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
[junit4] 2> 355614 T1402 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 355615 T1402 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 355615 T1402 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 355615 T1402 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 355616 T1403 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 355618 T1403 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core3",
[junit4] 2> "core_node_name":"node3",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 355618 T1403 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 355619 T1403 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 355620 T1402 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 355723 T1400 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 355723 T1406 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 356116 T1335 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 356121 T1335 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 356124 T1402 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 356124 T1402 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 356124 T1402 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 356125 T1402 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 356125 T1403 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 356127 T1403 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core4",
[junit4] 2> "core_node_name":"node4",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 356127 T1403 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 356127 T1403 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 356129 T1402 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 356231 T1400 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 356231 T1406 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 356625 T1335 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 356631 T1402 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 356631 T1402 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 356632 T1402 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 356632 T1403 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 356633 T1403 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core5",
[junit4] 2> "core_node_name":"node5",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 356633 T1403 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 356633 T1403 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
[junit4] 2> 356635 T1402 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 356737 T1406 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 356737 T1400 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 357131 T1335 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4] 2> 357137 T1402 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 357137 T1402 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 357138 T1402 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 357138 T1403 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 357139 T1403 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core6",
[junit4] 2> "core_node_name":"node6",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 357139 T1403 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 357139 T1403 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 357141 T1402 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 357247 T1400 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 357248 T1406 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 357637 T1335 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 357646 T1406 oascc.ZkStateReader$3.process Updating live nodes... (0)
[junit4] 2> 357648 T1335 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:34547 34547
[junit4] 2> 358351 T1390 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:34547 34547
[junit4] 2> 358352 T1335 oas.SolrTestCaseJ4.tearDown ###Ending testShardAssignment
[junit4] 2> 358352 T1403 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:34547_solr
[junit4] 2> 358353 T1407 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:302)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:291)
[junit4] 2>
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 358357 T1335 oas.SolrTestCaseJ4.setUp ###Starting testOverseerFailure
[junit4] 2> 358358 T1335 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 358359 T1408 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 358459 T1335 oasc.ZkTestServer.run start zk server on port:55133
[junit4] 2> 358460 T1335 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 358462 T1414 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@128af1c name:ZooKeeperConnection Watcher:127.0.0.1:55133 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 358463 T1335 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 358464 T1335 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 358465 T1416 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@11b50eb name:ZooKeeperConnection Watcher:127.0.0.1:55133 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 358465 T1335 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 358466 T1335 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 358468 T1335 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 358469 T1418 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@180d86d name:ZooKeeperConnection Watcher:127.0.0.1:55133/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 358469 T1335 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 358470 T1335 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 358471 T1335 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 358473 T1335 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 358474 T1335 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 358475 T1335 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 358476 T1420 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1156c4b name:ZooKeeperConnection Watcher:127.0.0.1:55133/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 358477 T1335 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 358478 T1335 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 358479 T1335 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
[junit4] 2> 358481 T1420 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4] 2> 358481 T1418 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4] 2> 358481 T1335 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 358482 T1422 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@18371fa name:ZooKeeperConnection Watcher:127.0.0.1:55133/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 358482 T1335 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 358483 T1335 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 358484 T1335 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 358487 T1335 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:55133_solr
[junit4] 2> 358487 T1335 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 358489 T1335 oasc.Overseer.start Overseer (id=91782813873405956-127.0.0.1:55133_solr-n_0000000000) starting
[junit4] 2> 358491 T1335 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 358495 T1335 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 358498 T1335 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 358500 T1335 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 358502 T1335 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 358504 T1335 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 358507 T1424 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 358507 T1424 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes at 127.0.0.1:55133_solr
[junit4] 2> 358507 T1423 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 359147 T1406 oasc.LeaderElector$ElectionWatcher.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard3/election
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:303)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
[junit4] 2>
[junit4] 2> 359148 T1406 oasc.LeaderElector$ElectionWatcher.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard2/election
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:303)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
[junit4] 2>
[junit4] 2> 359148 T1406 oasc.LeaderElector$ElectionWatcher.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:303)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
[junit4] 2>
[junit4] 2> 359509 T1422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 359510 T1423 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 359511 T1423 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 359511 T1423 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1]
[junit4] 2> 359511 T1423 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 359513 T1422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 359514 T1420 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 359514 T1418 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 360009 T1335 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 360017 T1335 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 360020 T1422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 360020 T1335 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 360021 T1423 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 360022 T1422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 360022 T1335 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 360023 T1423 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 360023 T1423 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
[junit4] 2> 360025 T1422 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 360127 T1420 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 360127 T1418 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 361180 T1335 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 361185 T1335 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 361186 T1426 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2eff29 name:ZooKeeperConnection Watcher:127.0.0.1:55133/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 361186 T1335 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 361186 T1423 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:55133_solr
[junit4] 2> 361188 T1427 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:302)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:291)
[junit4] 2>
[junit4] 2> 361189 T1335 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:55133_solr
[junit4] 2> 361189 T1335 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 361191 T1335 oasc.Overseer.start Overseer (id=91782813873405957-127.0.0.1:55133_solr-n_0000000001) starting
[junit4] 2> 361196 T1429 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 361197 T1428 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 361197 T1429 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes at 127.0.0.1:55133_solr
[junit4] 2> 361198 T1428 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 361199 T1428 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 361199 T1428 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
[junit4] 2> 361201 T1426 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 361201 T1418 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 361201 T1420 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 361254 T1426 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 361254 T1335 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 361255 T1428 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 361261 T1420 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 361263 T1420 oasc.LeaderElector$ElectionWatcher.process WARN org.apache.solr.common.SolrException: Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:136)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:142)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:110)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:303)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
[junit4] 2> Caused by: org.apache.solr.common.SolrException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /collections/collection1/leaders
[junit4] 2> at org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:40)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:123)
[junit4] 2> ... 6 more
[junit4] 2> Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /collections/collection1/leaders
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:432)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:429)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:386)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:373)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContextBase$1.execute(ElectionContext.java:128)
[junit4] 2> at org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:34)
[junit4] 2> ... 7 more
[junit4] 2>
[junit4] 2> 361264 T1420 oasc.LeaderElector$ElectionWatcher.process WARN org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /collections/collection1/leader_elect/shard1/election
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:303)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
[junit4] 2>
[junit4] 2> 361265 T1426 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 361265 T1420 oasc.LeaderElector$ElectionWatcher.process WARN org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /collections/collection1/leader_elect/shard1/election
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:303)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
[junit4] 2>
[junit4] 2> 361367 T1420 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 361367 T1418 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 361869 T1420 oascc.ZkStateReader$3.process Updating live nodes... (0)
[junit4] 2> 361869 T1418 oascc.ZkStateReader$3.process Updating live nodes... (0)
[junit4] 2> 361871 T1335 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:55133 55133
[junit4] 2> 362420 T1408 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:55133 55133
[junit4] 2> 362421 T1335 oas.SolrTestCaseJ4.tearDown ###Ending testOverseerFailure
[junit4] 2> 362421 T1428 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:55133_solr
[junit4] 2> 362422 T1430 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:302)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:291)
[junit4] 2>
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 362425 T1335 oas.SolrTestCaseJ4.setUp ###Starting testShardAssignmentBigger
[junit4] 2> 362426 T1335 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 362426 T1431 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 362526 T1335 oasc.ZkTestServer.run start zk server on port:39197
[junit4] 2> 362527 T1335 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 362530 T1437 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@12410f1 name:ZooKeeperConnection Watcher:127.0.0.1:39197 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 362530 T1335 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 362532 T1335 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 362533 T1439 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ef46aa name:ZooKeeperConnection Watcher:127.0.0.1:39197 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 362533 T1335 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 362534 T1335 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 362536 T1335 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 362537 T1441 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@129e94c name:ZooKeeperConnection Watcher:127.0.0.1:39197/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 362537 T1335 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 362537 T1335 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 362539 T1335 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 362540 T1443 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@699237 name:ZooKeeperConnection Watcher:127.0.0.1:39197/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 362540 T1335 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 362541 T1335 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 362542 T1335 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 362544 T1335 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:39197_solr
[junit4] 2> 362544 T1335 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 362545 T1335 oasc.Overseer.start Overseer (id=91782814139940867-127.0.0.1:39197_solr-n_0000000000) starting
[junit4] 2> 362546 T1335 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 362548 T1335 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 362550 T1335 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 362551 T1335 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 362555 T1335 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 362557 T1335 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 362560 T1445 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 362560 T1335 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 362561 T1445 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes at 127.0.0.1:39197_solr
[junit4] 2> 362561 T1444 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 362562 T1335 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 362563 T1335 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 362564 T1335 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 362565 T1447 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@143a053 name:ZooKeeperConnection Watcher:127.0.0.1:39197/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 362565 T1335 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 362566 T1335 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 362567 T1335 oascc.SolrZkClient.makePath makePath: /live_nodes/node0
[junit4] 2> 362569 T1441 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4] 2> 362569 T1447 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4] 2> 362570 T1335 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 362570 T1449 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f674c7 name:ZooKeeperConnection Watcher:127.0.0.1:39197/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 362571 T1335 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 362571 T1335 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 362573 T1335 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
[junit4] 2> 362574 T1441 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 362574 T1447 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 362574 T1449 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 362574 T1335 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 362575 T1451 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@58bdc name:ZooKeeperConnection Watcher:127.0.0.1:39197/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 362575 T1335 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 362576 T1335 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 362577 T1335 oascc.SolrZkClient.makePath makePath: /live_nodes/node2
[junit4] 2> 362579 T1441 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 362579 T1447 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 362579 T1449 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 362579 T1451 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 362580 T1335 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 362580 T1453 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f95e01 name:ZooKeeperConnection Watcher:127.0.0.1:39197/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 362580 T1335 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 362581 T1335 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 362583 T1335 oascc.SolrZkClient.makePath makePath: /live_nodes/node3
[junit4] 2> 362585 T1449 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 362585 T1447 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 362585 T1441 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 362585 T1453 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 362585 T1451 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 362586 T1335 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 362587 T1455 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@642035 name:ZooKeeperConnection Watcher:127.0.0.1:39197/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 362587 T1335 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 362588 T1335 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 362589 T1335 oascc.SolrZkClient.makePath makePath: /live_nodes/node4
[junit4] 2> 362591 T1451 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4] 2> 362591 T1453 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4] 2> 362592 T1441 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4] 2> 362591 T1447 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4] 2> 362591 T1449 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4] 2> 362592 T1455 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4] 2> 362593 T1335 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 362594 T1457 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1292072 name:ZooKeeperConnection Watcher:127.0.0.1:39197/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 362594 T1335 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 362596 T1335 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 362597 T1335 oascc.SolrZkClient.makePath makePath: /live_nodes/node5
[junit4] 2> 362598 T1449 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4] 2> 362599 T1455 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4] 2> 362598 T1441 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4] 2> 362599 T1451 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4] 2> 362599 T1447 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4] 2> 362600 T1457 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4] 2> 362600 T1453 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4] 2> 362600 T1335 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 362601 T1459 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1600fa0 name:ZooKeeperConnection Watcher:127.0.0.1:39197/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 362601 T1335 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 362602 T1335 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state
[...truncated too long message...]
ng cloud state from ZooKeeper...
[junit4] 2> 430497 T1335 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 430699 T1335 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 430901 T1335 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 431104 T1335 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 431306 T1335 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 431509 T1335 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 431763 T1757 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4] 2> 431763 T1753 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4] 2> 431764 T1757 oascc.ZkStateReader$3.process Updating live nodes... (0)
[junit4] 2> 431765 T1335 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:38291 38291
[junit4] 2> 432449 T1646 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:38291 38291
[junit4] 2> 432450 T1335 oas.SolrTestCaseJ4.tearDown ###Ending testShardLeaderChange
[junit4] 2> 432451 T1759 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:38291_solr
[junit4] 2> 432451 T1761 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:302)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:291)
[junit4] 2>
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=OverseerTest -Dtests.method=testShardLeaderChange -Dtests.seed=68EA34FC2C11F88D -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=hi -Dtests.timezone=Australia/Brisbane -Dtests.file.encoding=UTF-8
[junit4] FAILURE 61.6s J0 | OverseerTest.testShardLeaderChange <<<
[junit4] > Throwable #1: org.junit.ComparisonFailure: Unexpected shard leader coll:collection1 shard:shard1 expected:<core[1]> but was:<core[4]>
[junit4] > at __randomizedtesting.SeedInfo.seed([68EA34FC2C11F88D:B6B9B30B36890D7C]:0)
[junit4] > at org.apache.solr.cloud.OverseerTest.verifyShardLeader(OverseerTest.java:596)
[junit4] > at org.apache.solr.cloud.OverseerTest.testShardLeaderChange(OverseerTest.java:755)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 432459 T1335 oas.SolrTestCaseJ4.initCore ####initCore
[junit4] 2> 432459 T1335 oas.SolrTestCaseJ4.initCore ####initCore end
[junit4] 2> 433263 T1753 oasc.LeaderElector$ElectionWatcher.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:303)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
[junit4] 2>
[junit4] 2> 433264 T1753 oasc.LeaderElector$ElectionWatcher.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:303)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
[junit4] 2>
[junit4] 2> 435460 T1335 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4] 2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/build/solr-core/test/J0/./solr.cloud.OverseerTest-68EA34FC2C11F88D-001
[junit4] 2> NOTE: test params are: codec=CheapBastard, sim=RandomSimilarityProvider(queryNorm=false,coord=crazy): {}, locale=hi, timezone=Australia/Brisbane
[junit4] 2> NOTE: Linux 3.8.0-39-generic i386/Oracle Corporation 1.8.0_05 (32-bit)/cpus=8,threads=1,free=64394000,total=250085376
[junit4] 2> NOTE: All tests run in this JVM: [WordBreakSolrSpellCheckerTest, ZkControllerTest, LoggingHandlerTest, TestSolrDeletionPolicy1, TestElisionMultitermQuery, FileBasedSpellCheckerTest, LukeRequestHandlerTest, SuggesterTest, CoreAdminRequestStatusTest, TestReloadAndDeleteDocs, ExternalFileFieldSortTest, DocValuesTest, RegexBoostProcessorTest, TestStressUserVersions, SearchHandlerTest, TestShortCircuitedRequests, TestNonDefinedSimilarityFactory, CoreAdminCreateDiscoverTest, TestConfig, LegacyHTMLStripCharFilterTest, TestJoin, TestDocSet, TestBadConfig, FastVectorHighlighterTest, TestWriterPerf, TestConfigSets, TestLRUCache, TestPHPSerializedResponseWriter, TestReversedWildcardFilterFactory, TestSuggestSpellingConverter, FieldMutatingUpdateProcessorTest, TestTrie, ResponseLogComponentTest, TestPostingsSolrHighlighter, TestPseudoReturnFields, ResourceLoaderTest, SOLR749Test, TestComplexPhraseQParserPlugin, DOMUtilTest, TestFunctionQuery, TestSearcherReuse, DeleteShardTest, TestUniqueKeyFieldResource, ClusterStateUpdateTest, TestExceedMaxTermLength, MBeansHandlerTest, DistributedQueryElevationComponentTest, StressHdfsTest, SolrTestCaseJ4Test, CacheHeaderTest, AddBlockUpdateTest, SystemInfoHandlerTest, BasicDistributedZkTest, TestLMJelinekMercerSimilarityFactory, CopyFieldTest, LeaderElectionTest, QueryEqualityTest, TestBlendedInfixSuggestions, TestOverriddenPrefixQueryForCustomFieldType, TestJmxMonitoredMap, TestSort, TestSolrIndexConfig, TestValueSourceCache, CursorMarkTest, AutoCommitTest, DocumentBuilderTest, CoreAdminHandlerTest, CursorPagingTest, SpatialFilterTest, TestFreeTextSuggestions, OverseerTest]
[junit4] Completed on J0 in 85.00s, 9 tests, 1 failure <<< FAILURES!
[...truncated 847 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/lucene/common-build.xml:1293: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/lucene/common-build.xml:919: There were test failures: 387 suites, 1624 tests, 1 failure, 37 ignored (16 assumptions)
Total time: 58 minutes 13 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.8.0_05 -client -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure