You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/04/05 01:50:27 UTC
[JENKINS] Lucene-Solr-4.x-Linux (64bit/jdk1.7.0_51) - Build # 9885
- Failure!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/9885/
Java: 64bit/jdk1.7.0_51 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC -XX:-UseSuperWord
1 tests failed.
REGRESSION: org.apache.solr.cloud.OverseerTest.testShardLeaderChange
Error Message:
Unexpected shard leader coll:collection1 shard:shard1 expected:<core[1]> but was:<core[4]>
Stack Trace:
org.junit.ComparisonFailure: Unexpected shard leader coll:collection1 shard:shard1 expected:<core[1]> but was:<core[4]>
at __randomizedtesting.SeedInfo.seed([79EF914886CFCE36:A7BC16BF9C573BC7]:0)
at org.junit.Assert.assertEquals(Assert.java:125)
at org.apache.solr.cloud.OverseerTest.verifyShardLeader(OverseerTest.java:601)
at org.apache.solr.cloud.OverseerTest.testShardLeaderChange(OverseerTest.java:762)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
at java.lang.Thread.run(Thread.java:744)
Build Log:
[...truncated 11208 lines...]
[junit4] Suite: org.apache.solr.cloud.OverseerTest
[junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solrtest-OverseerTest-1396654591551
[junit4] 2> 647364 T3568 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
[junit4] 2> 647364 T3568 oas.SolrTestCaseJ4.initCore ####initCore
[junit4] 2> 647364 T3568 oas.SolrTestCaseJ4.initCore ####initCore end
[junit4] 2> 647368 T3568 oas.SolrTestCaseJ4.setUp ###Starting testStateChange
[junit4] 2> 647368 T3568 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 647369 T3569 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 647469 T3568 oasc.ZkTestServer.run start zk server on port:51429
[junit4] 2> 647470 T3568 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 647552 T3575 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7a285832 name:ZooKeeperConnection Watcher:127.0.0.1:51429/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 647552 T3568 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 647553 T3568 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 647555 T3577 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@fb958d5 name:ZooKeeperConnection Watcher:127.0.0.1:51429 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 647555 T3568 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 647557 T3568 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 647558 T3579 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@faec21b name:ZooKeeperConnection Watcher:127.0.0.1:51429 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 647558 T3568 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 647558 T3568 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 647560 T3568 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 647562 T3568 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 647563 T3568 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 647564 T3568 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 647566 T3568 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 647568 T3581 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2dbe1bbd name:ZooKeeperConnection Watcher:127.0.0.1:51429/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 647568 T3568 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 647569 T3568 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 647570 T3568 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 647574 T3568 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:51429_solr
[junit4] 2> 647575 T3568 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 647576 T3568 oasc.Overseer.start Overseer (id=91531155312345091-127.0.0.1:51429_solr-n_0000000000) starting
[junit4] 2> 647577 T3568 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 647580 T3568 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 647582 T3568 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 647584 T3568 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 647590 T3568 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 647593 T3568 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 647597 T3583 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 647598 T3582 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 647598 T3583 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes at 127.0.0.1:51429_solr
[junit4] 2> 647599 T3568 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 647599 T3582 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 647600 T3582 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1/solr",
[junit4] 2> "node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "core":"core1",
[junit4] 2> "roles":"",
[junit4] 2> "state":"recovering"}
[junit4] 2> 647600 T3582 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 647604 T3581 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 647604 T3575 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 647650 T3568 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 647652 T3581 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 647653 T3582 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 647654 T3582 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1/solr",
[junit4] 2> "node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "core":"core1",
[junit4] 2> "roles":"",
[junit4] 2> "state":"active"}
[junit4] 2> 647654 T3582 oasc.Overseer$ClusterStateUpdater.updateState node=core_node1 is already registered
[junit4] 2> 647654 T3582 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
[junit4] 2> 647656 T3581 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 647758 T3575 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 647804 T3568 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:51429 51429
[junit4] 2> 647974 T3569 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:51429 51429
[junit4] 2> 647976 T3568 oas.SolrTestCaseJ4.tearDown ###Ending testStateChange
[junit4] 2> 647976 T3582 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:51429_solr
[junit4] 2> 647977 T3584 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:302)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:291)
[junit4] 2>
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 647980 T3568 oas.SolrTestCaseJ4.setUp ###Starting testDoubleAssignment
[junit4] 2> 647981 T3568 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 647981 T3585 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 648081 T3568 oasc.ZkTestServer.run start zk server on port:51788
[junit4] 2> 648082 T3568 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 648084 T3591 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4a3839da name:ZooKeeperConnection Watcher:127.0.0.1:51788/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 648084 T3568 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 648085 T3568 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 648087 T3593 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5ed82ee3 name:ZooKeeperConnection Watcher:127.0.0.1:51788 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 648087 T3568 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 648098 T3568 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 648099 T3595 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4b141ef4 name:ZooKeeperConnection Watcher:127.0.0.1:51788 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 648099 T3568 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 648099 T3568 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 648104 T3568 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 648105 T3568 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 648107 T3568 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 648108 T3568 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 648111 T3568 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 648114 T3597 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1635b0 name:ZooKeeperConnection Watcher:127.0.0.1:51788/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 648114 T3568 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 648115 T3568 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 648117 T3568 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
[junit4] 2> 648119 T3597 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4] 2> 648119 T3591 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4] 2> 648121 T3568 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 648122 T3599 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7cee30db name:ZooKeeperConnection Watcher:127.0.0.1:51788/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 648123 T3568 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 648123 T3568 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 648125 T3568 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 648128 T3568 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:51788_solr
[junit4] 2> 648129 T3568 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 648130 T3568 oasc.Overseer.start Overseer (id=91531155352584196-127.0.0.1:51788_solr-n_0000000000) starting
[junit4] 2> 648132 T3568 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 648134 T3568 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 648137 T3568 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 648139 T3568 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 648144 T3568 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 648147 T3568 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 648152 T3601 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 648153 T3600 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 648153 T3601 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes at 127.0.0.1:51788_solr
[junit4] 2> 648155 T3600 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 648155 T3600 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 648156 T3600 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1]
[junit4] 2> 648156 T3600 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 648158 T3599 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 648159 T3591 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 648162 T3597 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 648654 T3568 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 648663 T3568 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 648667 T3599 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 648668 T3568 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 648669 T3600 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 648669 T3591 oascc.ZkStateReader$3.process Updating live nodes... (0)
[junit4] 2> 648670 T3568 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 648671 T3597 oascc.ZkStateReader$3.process Updating live nodes... (0)
[junit4] 2> 648672 T3603 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ac82b72 name:ZooKeeperConnection Watcher:127.0.0.1:51788/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 648672 T3568 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 648672 T3599 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 648673 T3568 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 648675 T3568 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
[junit4] 2> 648677 T3591 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4] 2> 648677 T3603 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4] 2> 648679 T3599 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 648679 T3568 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 648679 T3600 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 648680 T3600 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
[junit4] 2> 648681 T3599 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 648683 T3568 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 648687 T3599 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 648690 T3599 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 648792 T3591 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 648792 T3603 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 648794 T3568 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 648796 T3591 oascc.ZkStateReader$3.process Updating live nodes... (0)
[junit4] 2> 648796 T3603 oascc.ZkStateReader$3.process Updating live nodes... (0)
[junit4] 2> 648798 T3568 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:51788 51788
[junit4] 2> 648956 T3585 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:51788 51788
[junit4] 2> 648957 T3568 oas.SolrTestCaseJ4.tearDown ###Ending testDoubleAssignment
[junit4] 2> 648957 T3600 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:51788_solr
[junit4] 2> 648958 T3604 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:302)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:291)
[junit4] 2>
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 648964 T3568 oas.SolrTestCaseJ4.setUp ###Starting testBadQueueItem
[junit4] 2> 648964 T3568 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 648964 T3605 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 649064 T3568 oasc.ZkTestServer.run start zk server on port:39845
[junit4] 2> 649065 T3568 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 649068 T3611 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7f584ce0 name:ZooKeeperConnection Watcher:127.0.0.1:39845 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 649068 T3568 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 649085 T3568 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 649087 T3613 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@70e382c5 name:ZooKeeperConnection Watcher:127.0.0.1:39845 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 649087 T3568 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 649087 T3568 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 649089 T3568 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 649092 T3615 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7630882a name:ZooKeeperConnection Watcher:127.0.0.1:39845/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 649093 T3568 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 649093 T3568 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 649097 T3568 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 649098 T3617 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@32beab36 name:ZooKeeperConnection Watcher:127.0.0.1:39845/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 649098 T3568 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 649099 T3568 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 649101 T3568 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 649105 T3568 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:39845_solr
[junit4] 2> 649105 T3568 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 649107 T3568 oasc.Overseer.start Overseer (id=91531155418185731-127.0.0.1:39845_solr-n_0000000000) starting
[junit4] 2> 649108 T3568 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 649111 T3568 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 649114 T3568 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 649116 T3568 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 649119 T3568 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 649125 T3568 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 649130 T3619 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 649130 T3568 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 649130 T3619 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes at 127.0.0.1:39845_solr
[junit4] 2> 649130 T3618 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 649132 T3568 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 649133 T3568 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 649135 T3568 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 649136 T3621 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@722aef5e name:ZooKeeperConnection Watcher:127.0.0.1:39845/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 649137 T3568 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 649137 T3568 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 649139 T3568 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1
[junit4] 2> 649141 T3615 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4] 2> 649142 T3621 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4] 2> 649143 T3617 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 649144 T3618 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 649145 T3618 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 649145 T3618 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
[junit4] 2> 649145 T3618 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
[junit4] 2> 649147 T3617 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 649147 T3615 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 649147 T3621 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 649643 T3568 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4] 2> 649652 T3568 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
[junit4] 2> 649656 T3617 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 649657 T3618 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 649658 T3617 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 649660 T3618 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core2",
[junit4] 2> "core_node_name":"node2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 649660 T3618 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 649660 T3618 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 649662 T3617 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 649765 T3615 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 649765 T3621 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 650158 T3568 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 650165 T3568 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 650170 T3617 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 650170 T3617 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 650170 T3617 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 650171 T3617 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 650171 T3618 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 650174 T3618 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core3",
[junit4] 2> "core_node_name":"node3",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 650174 T3618 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 650174 T3618 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 650176 T3617 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 650278 T3615 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 650278 T3621 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 650671 T3568 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 650684 T3568 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4] 2> 650691 T3617 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 650691 T3617 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 650692 T3617 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 650694 T3618 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 650698 T3617 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 650801 T3621 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 650801 T3615 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 650845 T3617 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 650845 T3617 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 650846 T3617 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 650849 T3618 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 650850 T3617 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 650851 T3618 oasc.Overseer$ClusterStateUpdater.checkKeyExistence ERROR Skipping invalid Overseer message because it has no collection specified: {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core0",
[junit4] 2> "core_node_name":"node0",
[junit4] 2> "collection":"",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 650857 T3618 oasc.Overseer$ClusterStateUpdater.checkKeyExistence ERROR Skipping invalid Overseer message because it has no collection specified: {
[junit4] 2> "operation":"deletecore",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core0",
[junit4] 2> "core_node_name":"node0",
[junit4] 2> "collection":""}
[junit4] 2> 650862 T3617 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 650864 T3618 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection2",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 650864 T3618 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection2 with shards [shard1, shard2, shard3]
[junit4] 2> 650865 T3618 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
[junit4] 2> 650869 T3617 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 650971 T3615 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 650971 T3621 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 651354 T3568 oascc.SolrZkClient.makePath makePath: /collections/collection2/leader_elect/shard3/election
[junit4] 2> 651363 T3568 oascc.SolrZkClient.makePath makePath: /collections/collection2/leaders/shard3
[junit4] 2> 651366 T3617 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 651367 T3617 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 651367 T3617 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 651368 T3617 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 651368 T3618 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 651371 T3618 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core2",
[junit4] 2> "core_node_name":"node2",
[junit4] 2> "collection":"collection2",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 651371 T3618 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 651371 T3618 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 651373 T3617 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 651475 T3615 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 651475 T3621 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 651868 T3568 oascc.SolrZkClient.makePath makePath: /collections/collection2/leader_elect/shard1/election
[junit4] 2> 651874 T3568 oascc.SolrZkClient.makePath makePath: /collections/collection2/leaders/shard1
[junit4] 2> 651877 T3617 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 651877 T3617 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 651877 T3617 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 651878 T3617 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 651878 T3618 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 651882 T3618 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core3",
[junit4] 2> "core_node_name":"node3",
[junit4] 2> "collection":"collection2",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 651883 T3618 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 651883 T3618 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 651884 T3617 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 651987 T3621 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 651987 T3615 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 652378 T3568 oascc.SolrZkClient.makePath makePath: /collections/collection2/leader_elect/shard2/election
[junit4] 2> 652384 T3568 oascc.SolrZkClient.makePath makePath: /collections/collection2/leaders/shard2
[junit4] 2> 652388 T3617 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 652388 T3617 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 652388 T3617 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 652389 T3618 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 652391 T3617 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 652493 T3615 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 652493 T3621 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 652539 T3606 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x1452f19d4950002, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2>
[junit4] 2> 652540 T3621 oascc.ZkStateReader$3.process Updating live nodes... (0)
[junit4] 2> 652542 T3568 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:39845 39845
[junit4] 2> 652718 T3605 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:39845 39845
[junit4] 2> 652719 T3568 oas.SolrTestCaseJ4.tearDown ###Ending testBadQueueItem
[junit4] 2> 652720 T3618 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:39845_solr
[junit4] 2> 652721 T3622 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:302)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:291)
[junit4] 2>
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 652725 T3568 oas.SolrTestCaseJ4.setUp ###Starting testShardAssignment
[junit4] 2> 652725 T3568 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 652726 T3623 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 652826 T3568 oasc.ZkTestServer.run start zk server on port:52484
[junit4] 2> 652827 T3568 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 652830 T3629 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@25701a5e name:ZooKeeperConnection Watcher:127.0.0.1:52484 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 652830 T3568 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 652875 T3568 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 652876 T3631 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7a6b161d name:ZooKeeperConnection Watcher:127.0.0.1:52484 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 652876 T3568 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 652876 T3568 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 652879 T3568 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 652880 T3633 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1312f5c6 name:ZooKeeperConnection Watcher:127.0.0.1:52484/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 652880 T3568 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 652880 T3568 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 652882 T3568 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 652883 T3635 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@284aefe name:ZooKeeperConnection Watcher:127.0.0.1:52484/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 652884 T3568 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 652884 T3568 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 652886 T3568 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 652888 T3568 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:52484_solr
[junit4] 2> 652889 T3568 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 652890 T3568 oasc.Overseer.start Overseer (id=91531155664142339-127.0.0.1:52484_solr-n_0000000000) starting
[junit4] 2> 652894 T3568 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 652897 T3568 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 652899 T3568 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 652902 T3568 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 652904 T3568 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 652910 T3568 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 652915 T3637 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 652915 T3568 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 652916 T3637 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes at 127.0.0.1:52484_solr
[junit4] 2> 652916 T3636 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 652920 T3568 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 652922 T3568 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 652923 T3568 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 652925 T3639 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f252adb name:ZooKeeperConnection Watcher:127.0.0.1:52484/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 652925 T3568 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 652926 T3568 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 652928 T3568 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1
[junit4] 2> 652930 T3639 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4] 2> 652930 T3633 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4] 2> 652934 T3635 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 652935 T3636 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 652935 T3636 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 652936 T3636 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
[junit4] 2> 652936 T3636 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
[junit4] 2> 652938 T3635 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 652940 T3633 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 652940 T3639 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 653434 T3568 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4] 2> 653451 T3568 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
[junit4] 2> 653456 T3635 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 653457 T3636 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 653458 T3635 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 653464 T3636 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core2",
[junit4] 2> "core_node_name":"node2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 653464 T3636 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 653464 T3636 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 653467 T3635 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 653571 T3633 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 653571 T3639 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 653958 T3568 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 653964 T3568 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 653967 T3635 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 653967 T3635 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 653968 T3635 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 653969 T3635 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 653969 T3636 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 653972 T3636 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core3",
[junit4] 2> "core_node_name":"node3",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 653972 T3636 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 653972 T3636 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 653973 T3635 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 654075 T3633 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 654075 T3639 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 654469 T3568 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 654476 T3568 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4] 2> 654479 T3635 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 654480 T3635 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 654480 T3635 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 654481 T3635 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 654481 T3636 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 654484 T3636 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core4",
[junit4] 2> "core_node_name":"node4",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 654484 T3636 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 654484 T3636 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
[junit4] 2> 654487 T3635 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 654588 T3633 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 654588 T3639 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 654981 T3568 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4] 2> 654987 T3635 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 654988 T3635 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 654988 T3635 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 654988 T3636 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 654989 T3636 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core5",
[junit4] 2> "core_node_name":"node5",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 654989 T3636 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 654989 T3636 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 654995 T3635 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 655097 T3639 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 655097 T3633 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 655487 T3568 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 655493 T3635 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 655493 T3635 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 655494 T3635 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 655495 T3636 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 655495 T3636 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core6",
[junit4] 2> "core_node_name":"node6",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 655495 T3636 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 655496 T3636 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 655498 T3635 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 655599 T3639 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 655599 T3633 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 655993 T3568 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 656000 T3639 oascc.ZkStateReader$3.process Updating live nodes... (0)
[junit4] 2> 656002 T3568 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:52484 52484
[junit4] 2> 656319 T3623 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:52484 52484
[junit4] 2> 656320 T3568 oas.SolrTestCaseJ4.tearDown ###Ending testShardAssignment
[junit4] 2> 656321 T3636 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:52484_solr
[junit4] 2> 656322 T3640 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:302)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:291)
[junit4] 2>
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 656328 T3568 oas.SolrTestCaseJ4.setUp ###Starting testPlaceholders
[junit4] 2> 656328 T3568 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 656329 T3641 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 656430 T3568 oasc.ZkTestServer.run start zk server on port:52723
[junit4] 2> 656432 T3568 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 656435 T3647 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@47c157c name:ZooKeeperConnection Watcher:127.0.0.1:52723/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 656436 T3568 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 656436 T3568 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 656438 T3649 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@35498167 name:ZooKeeperConnection Watcher:127.0.0.1:52723 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 656438 T3568 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 656455 T3568 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 656456 T3651 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@32c388da name:ZooKeeperConnection Watcher:127.0.0.1:52723 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 656456 T3568 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 656456 T3568 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 656458 T3568 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 656459 T3568 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 656460 T3568 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 656462 T3568 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 656464 T3568 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 656467 T3653 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4d8e81f3 name:ZooKeeperConnection Watcher:127.0.0.1:52723/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 656467 T3568 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 656468 T3568 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 656470 T3568 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
[junit4] 2> 656472 T3653 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4] 2> 656472 T3647 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4] 2> 656472 T3568 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 656473 T3655 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@66c2db4c name:ZooKeeperConnection Watcher:127.0.0.1:52723/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 656474 T3568 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 656474 T3568 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 656476 T3568 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 656478 T3568 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:52723_solr
[junit4] 2> 656479 T3568 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 656480 T3568 oasc.Overseer.start Overseer (id=91531155900399620-127.0.0.1:52723_solr-n_0000000000) starting
[junit4] 2> 656481 T3568 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 656483 T3568 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 656485 T3568 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 656488 T3568 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 656490 T3568 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 656494 T3568 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 656498 T3657 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 656499 T3657 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes at 127.0.0.1:52723_solr
[junit4] 2> 656499 T3656 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 656501 T3656 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 656501 T3656 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=12 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"12",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 656501 T3656 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3, shard4, shard5, shard6, shard7, shard8, shard9, shard10, shard11, shard12]
[junit4] 2> 656501 T3656 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard8
[junit4] 2> 656503 T3655 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 656504 T3653 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 656504 T3647 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 657000 T3568 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard8/election
[junit4] 2> 657007 T3568 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard8
[junit4] 2> 657018 T3655 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 657030 T3568 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 657031 T3656 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 657032 T3647 oascc.ZkStateReader$3.process Updating live nodes... (0)
[junit4] 2> 657032 T3653 oascc.ZkStateReader$3.process Updating live nodes... (0)
[junit4] 2> 657033 T3568 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:52723 52723
[junit4] 2> 657153 T3641 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:52723 52723
[junit4] 2> 657154 T3568 oas.SolrTestCaseJ4.tearDown ###Ending testPlaceholders
[junit4] 2> 657154 T3656 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:52723_solr
[junit4] 2> 657155 T3658 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:302)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:291)
[junit4] 2>
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 657159 T3568 oas.SolrTestCaseJ4.setUp ###Starting testReplay
[junit4] 2> 657159 T3568 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 657160 T3659 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 657260 T3568 oasc.ZkTestServer.run start zk server on port:42495
[junit4] 2> 657261 T3568 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 657322 T3665 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7dbacf76 name:ZooKeeperConnection Watcher:127.0.0.1:42495/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 657322 T3568 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 657323 T3568 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 657326 T3667 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@738b06f1 name:ZooKeeperConnection Watcher:127.0.0.1:42495 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 657326 T3568 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 657344 T3568 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 657346 T3669 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@cb0d564 name:ZooKeeperConnection Watcher:127.0.0.1:42495 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 657346 T3568 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 657346 T3568 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 657348 T3568 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 657349 T3568 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 657351 T3568 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 657352 T3568 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 657354 T3568 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 657356 T3568 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 657358 T3671 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@53c1c985 name:ZooKeeperConnection Watcher:127.0.0.1:42495/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 657358 T3568 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 657358 T3568 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 657360 T3568 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 657362 T3568 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:42495_solr
[junit4] 2> 657362 T3568 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 657364 T3568 oasc.Overseer.start Overseer (id=91531155954991107-127.0.0.1:42495_solr-n_0000000000) starting
[junit4] 2> 657365 T3568 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 657367 T3568 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 657370 T3568 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 657374 T3568 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 657377 T3568 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 657381 T3673 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 657382 T3673 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes at 127.0.0.1:42495_solr
[junit4] 2> 657383 T3672 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 657383 T3672 oasc.Overseer$ClusterStateUpdater.run Replaying operations from work queue.
[junit4] 2> 657383 T3672 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1/solr",
[junit4] 2> "node_name":"node1",
[junit4] 2> "shard":"s1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "core":"core1",
[junit4] 2> "roles":"",
[junit4] 2> "state":"recovering"}
[junit4] 2> 657384 T3665 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 657386 T3672 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1/solr",
[junit4] 2> "node_name":"node1",
[junit4] 2> "shard":"s1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "core":"core2",
[junit4] 2> "roles":"",
[junit4] 2> "state":"recovering"}
[junit4] 2> 657387 T3665 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 657388 T3672 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 657389 T3672 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 657389 T3672 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1/solr",
[junit4] 2> "node_name":"node1",
[junit4] 2> "shard":"s1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "core":"core3",
[junit4] 2> "roles":"",
[junit4] 2> "state":"recovering"}
[junit4] 2> 657391 T3671 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 657391 T3665 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 657485 T3568 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:42495 42495
[junit4] 2> 657500 T3639 oasc.LeaderElector$ElectionWatcher.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard2/election
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:303)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
[junit4] 2>
[junit4] 2> 657742 T3659 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:42495 42495
[junit4] 2> 657742 T3639 oasc.LeaderElector$ElectionWatcher.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:303)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
[junit4] 2>
[junit4] 2> 657743 T3568 oas.SolrTestCaseJ4.tearDown ###Ending testReplay
[junit4] 2> 657743 T3639 oasc.LeaderElector$ElectionWatcher.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard3/election
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:303)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
[junit4] 2>
[junit4] 2> 657743 T3672 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:42495_solr
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 657745 T3674 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:302)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
[junit4] 2> at org.apache.solr.cloud.Overse
[...truncated too long message...]
oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4] 2> 719132 T3953 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4] 2> 719132 T3957 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4] 2> 719132 T3945 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4] 2> 719132 T3955 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4] 2> 719132 T3949 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4] 2> 719132 T3959 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4] 2> 719135 T3959 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4] 2> 719135 T3955 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4] 2> 719135 T3951 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4] 2> 719136 T3953 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4] 2> 719135 T3947 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4] 2> 719135 T3949 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4] 2> 719135 T3957 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4] 2> 719136 T3953 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4] 2> 719136 T3955 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4] 2> 719137 T3951 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4] 2> 719137 T3949 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4] 2> 719137 T3957 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4] 2> 719137 T3959 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4] 2> 719138 T3955 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 719138 T3959 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 719138 T3953 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 719138 T3951 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 719139 T3957 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 719140 T3953 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 719140 T3957 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 719140 T3955 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 719140 T3959 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 719141 T3957 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 719141 T3955 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 719141 T3959 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 719142 T3959 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4] 2> 719142 T3957 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4] 2> 719143 T3959 oascc.ZkStateReader$3.process Updating live nodes... (0)
[junit4] 2> 719143 T3568 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:57903 57903
[junit4] 2> 719206 T3813 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:57903 57903
[junit4] 2> 719208 T3568 oas.SolrTestCaseJ4.tearDown ###Ending testShardAssignmentBigger
[junit4] 2> 719208 T3826 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:57903_solr
[junit4] 2> 719209 T4026 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:302)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:291)
[junit4] 2>
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 719209 T3568 oas.SolrTestCaseJ4.initCore ####initCore
[junit4] 2> 719210 T3568 oas.SolrTestCaseJ4.initCore ####initCore end
[junit4] 2> 720358 T3847 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 720399 T3851 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 720408 T3853 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 720496 T3875 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 720551 T3899 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 720557 T3901 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 720561 T3903 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 720591 T3917 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 720622 T3937 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 722210 T3568 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4] 2> NOTE: test params are: codec=Lucene42, sim=RandomSimilarityProvider(queryNorm=true,coord=no): {}, locale=sq_AL, timezone=Australia/Yancowinna
[junit4] 2> NOTE: Linux 3.8.0-38-generic amd64/Oracle Corporation 1.7.0_51 (64-bit)/cpus=8,threads=1,free=102771168,total=256028672
[junit4] 2> NOTE: All tests run in this JVM: [DeleteShardTest, TestShardHandlerFactory, MoreLikeThisHandlerTest, HdfsSyncSliceTest, SimpleFacetsTest, TestLMDirichletSimilarityFactory, SolrIndexSplitterTest, URLClassifyProcessorTest, TestCopyFieldCollectionResource, TestCursorMarkWithoutUniqueKey, TestPHPSerializedResponseWriter, TestReloadAndDeleteDocs, SynonymTokenizerTest, TestConfig, HdfsRecoveryZkTest, TestFreeTextSuggestions, TestSolrXmlPersistence, AutoCommitTest, TestWriterPerf, UUIDFieldTest, TestPerFieldSimilarity, TestRandomMergePolicy, SyncSliceTest, PeerSyncTest, UpdateRequestProcessorFactoryTest, LoggingHandlerTest, TestCoreContainer, ParsingFieldUpdateProcessorsTest, SolrInfoMBeanTest, TestDynamicFieldResource, TestValueSourceCache, TestDistributedMissingSort, CSVRequestHandlerTest, TestBM25SimilarityFactory, CursorMarkTest, TestConfigSets, TestRealTimeGet, TestCharFilters, SOLR749Test, TestLuceneMatchVersion, DocValuesTest, TestFaceting, DirectUpdateHandlerOptimizeTest, TestRecoveryHdfs, BinaryUpdateRequestHandlerTest, WordBreakSolrSpellCheckerTest, FileBasedSpellCheckerTest, TestSolrDeletionPolicy1, TestPseudoReturnFields, TestRemoteStreaming, BasicFunctionalityTest, QueryResultKeyTest, TestSearchPerf, AnalysisErrorHandlingTest, TestSort, DistributedTermsComponentTest, TestStressUserVersions, DateMathParserTest, TermsComponentTest, TestSuggestSpellingConverter, SpatialFilterTest, TestElisionMultitermQuery, TriLevelCompositeIdRoutingTest, ClusterStateTest, DocExpirationUpdateProcessorFactoryTest, TestMergePolicyConfig, TestNonDefinedSimilarityFactory, CacheHeaderTest, ChangedSchemaMergeTest, TestReversedWildcardFilterFactory, AnalysisAfterCoreReloadTest, FastVectorHighlighterTest, ShowFileRequestHandlerTest, TestManagedSchema, TestSearcherReuse, SampleTest, TestClassNameShortening, TestCollapseQParserPlugin, SpellPossibilityIteratorTest, LeaderElectionTest, TestInfoStreamLogging, HdfsDirectoryTest, UnloadDistributedZkTest, TestSurroundQueryParser, ScriptEngineTest, TestReplicationHandler, TestHighFrequencyDictionaryFactory, TestLFUCache, MBeansHandlerTest, ZkCLITest, BadComponentTest, CoreMergeIndexesAdminHandlerTest, BasicDistributedZkTest, ChaosMonkeySafeLeaderTest, OpenCloseCoreStressTest, OverseerTest]
[junit4] Completed on J0 in 74.87s, 9 tests, 1 failure <<< FAILURES!
[...truncated 581 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1276: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:908: There were test failures: 382 suites, 1611 tests, 1 failure, 37 ignored (16 assumptions)
Total time: 57 minutes 8 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_51 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC -XX:-UseSuperWord
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-4.x-Linux (64bit/jdk1.8.0) - Build # 9886 -
Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/9886/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseParallelGC
4 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.LeaderElectionTest
Error Message:
ERROR: SolrZkClient opens=46 closes=44
Stack Trace:
java.lang.AssertionError: ERROR: SolrZkClient opens=46 closes=44
at __randomizedtesting.SeedInfo.seed([87C3BDDA0A20A364]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.SolrTestCaseJ4.endTrackingZkClients(SolrTestCaseJ4.java:469)
at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:195)
at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:790)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
at java.lang.Thread.run(Thread.java:744)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.LeaderElectionTest
Error Message:
4 threads leaked from SUITE scope at org.apache.solr.cloud.LeaderElectionTest: 1) Thread[id=4593, name=Thread-2081-EventThread, state=WAITING, group=TGRP-LeaderElectionTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) 2) Thread[id=4584, name=Thread-2081-SendThread(127.0.0.1:51799), state=TIMED_WAITING, group=TGRP-LeaderElectionTest] at java.lang.Thread.sleep(Native Method) at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003) 3) Thread[id=4585, name=Thread-2081-EventThread, state=WAITING, group=TGRP-LeaderElectionTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) 4) Thread[id=4592, name=Thread-2081-SendThread(127.0.0.1:51799), state=TIMED_WAITING, group=TGRP-LeaderElectionTest] at java.lang.Thread.sleep(Native Method) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 4 threads leaked from SUITE scope at org.apache.solr.cloud.LeaderElectionTest:
1) Thread[id=4593, name=Thread-2081-EventThread, state=WAITING, group=TGRP-LeaderElectionTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
2) Thread[id=4584, name=Thread-2081-SendThread(127.0.0.1:51799), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
3) Thread[id=4585, name=Thread-2081-EventThread, state=WAITING, group=TGRP-LeaderElectionTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
4) Thread[id=4592, name=Thread-2081-SendThread(127.0.0.1:51799), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
at __randomizedtesting.SeedInfo.seed([87C3BDDA0A20A364]:0)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.LeaderElectionTest
Error Message:
There are still zombie threads that couldn't be terminated: 1) Thread[id=4584, name=Thread-2081-SendThread(127.0.0.1:51799), state=TIMED_WAITING, group=TGRP-LeaderElectionTest] at java.lang.Thread.sleep(Native Method) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994) 2) Thread[id=4592, name=Thread-2081-SendThread(127.0.0.1:51799), state=TIMED_WAITING, group=TGRP-LeaderElectionTest] at java.lang.Thread.sleep(Native Method) at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
1) Thread[id=4584, name=Thread-2081-SendThread(127.0.0.1:51799), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
2) Thread[id=4592, name=Thread-2081-SendThread(127.0.0.1:51799), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
at __randomizedtesting.SeedInfo.seed([87C3BDDA0A20A364]:0)
REGRESSION: org.apache.solr.cloud.LeaderElectionTest.testStressElection
Error Message:
Could not get leader props
Stack Trace:
java.lang.RuntimeException: Could not get leader props
at __randomizedtesting.SeedInfo.seed([87C3BDDA0A20A364:789569746D6EB277]:0)
at org.apache.solr.cloud.LeaderElectionTest.getLeaderUrl(LeaderElectionTest.java:225)
at org.apache.solr.cloud.LeaderElectionTest.getLeaderThread(LeaderElectionTest.java:325)
at org.apache.solr.cloud.LeaderElectionTest.testStressElection(LeaderElectionTest.java:434)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
at java.lang.Thread.run(Thread.java:744)
Build Log:
[...truncated 11061 lines...]
[junit4] Suite: org.apache.solr.cloud.LeaderElectionTest
[junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solrtest-LeaderElectionTest-1396662431918
[junit4] 2> 713857 T4508 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (true)
[junit4] 2> 713859 T4508 oas.SolrTestCaseJ4.setUp ###Starting testCancelElection
[junit4] 2> 713859 T4508 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 713860 T4509 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 713960 T4508 oasc.ZkTestServer.run start zk server on port:57976
[junit4] 2> 713961 T4508 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 714196 T4515 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@135866d1 name:ZooKeeperConnection Watcher:127.0.0.1:57976 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 714196 T4508 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 714199 T4508 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 714211 T4517 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@59a26ba9 name:ZooKeeperConnection Watcher:127.0.0.1:57976 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 714211 T4508 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 714211 T4508 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 714239 T4508 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 714240 T4519 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@cf063fc name:ZooKeeperConnection Watcher:127.0.0.1:57976/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 714240 T4508 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 714241 T4508 oascc.SolrZkClient.makePath makePath: /collections/collection2
[junit4] 2> 714243 T4508 oascc.SolrZkClient.makePath makePath: /collections/collection2/leader_elect/slice1/election
[junit4] 2> 714246 T4508 oascc.SolrZkClient.makePath makePath: /collections/collection2/leaders/slice1
[junit4] 2> 714248 T4508 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 716253 T4508 oasc.ElectionContext.cancelElection canceling election /collections/collection2/leader_elect/slice1/election/91531669138440194-dummynode1-n_0000000000
[junit4] 2> 716254 T4519 oascc.SolrZkClient.makePath makePath: /collections/collection2/leaders/slice1
[junit4] 2> 717256 T4508 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:57976 57976
[junit4] 2> 717457 T4509 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:57976 57976
[junit4] 2> 717458 T4508 oas.SolrTestCaseJ4.tearDown ###Ending testCancelElection
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 717461 T4508 oas.SolrTestCaseJ4.setUp ###Starting testStressElection
[junit4] 2> 717461 T4508 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 717461 T4520 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 717561 T4508 oasc.ZkTestServer.run start zk server on port:51799
[junit4] 2> 717562 T4508 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 717565 T4526 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@52816477 name:ZooKeeperConnection Watcher:127.0.0.1:51799 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 717565 T4508 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 717573 T4508 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 717574 T4528 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1834b2ad name:ZooKeeperConnection Watcher:127.0.0.1:51799 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 717574 T4508 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 717575 T4508 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 717577 T4508 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 717578 T4530 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1754ddd4 name:ZooKeeperConnection Watcher:127.0.0.1:51799/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 717578 T4508 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 717579 T4508 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 717580 T4533 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@33cd3fd0 name:ZooKeeperConnection Watcher:127.0.0.1:51799/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 717580 T4508 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 717581 T4534 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 717583 T4534 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 717586 T4534 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 717588 T4534 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 719583 T4535 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 719594 T4540 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@19bb1e64 name:ZooKeeperConnection Watcher:127.0.0.1:51799/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 719595 T4535 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 719596 T4535 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 719625 T4544 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5886145f name:ZooKeeperConnection Watcher:127.0.0.1:51799/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 719626 T4535 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 719626 T4535 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 719634 T4521 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x1452f917ec40004, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2>
[junit4] 2> 719667 T4548 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6994adc7 name:ZooKeeperConnection Watcher:127.0.0.1:51799/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 719668 T4535 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 719668 T4535 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 719669 T4553 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@38a9eb10 name:ZooKeeperConnection Watcher:127.0.0.1:51799/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 719669 T4535 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 719670 T4535 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 719671 T4557 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@32305b37 name:ZooKeeperConnection Watcher:127.0.0.1:51799/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 719671 T4535 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 719672 T4535 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 719683 T4561 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3ce73708 name:ZooKeeperConnection Watcher:127.0.0.1:51799/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 719683 T4535 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 719684 T4535 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 719685 T4565 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@43aafc18 name:ZooKeeperConnection Watcher:127.0.0.1:51799/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 719685 T4535 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 719685 T4535 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 719686 T4569 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@774174 name:ZooKeeperConnection Watcher:127.0.0.1:51799/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 719687 T4535 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 719687 T4535 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 719688 T4573 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@78737411 name:ZooKeeperConnection Watcher:127.0.0.1:51799/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 719688 T4535 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 719689 T4535 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 719690 T4577 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2cbb9df2 name:ZooKeeperConnection Watcher:127.0.0.1:51799/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 719690 T4535 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 719691 T4535 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 719691 T4581 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4b5fed3a name:ZooKeeperConnection Watcher:127.0.0.1:51799/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 719691 T4535 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 719692 T4535 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 719693 T4585 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@45ed81fd name:ZooKeeperConnection Watcher:127.0.0.1:51799/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 719693 T4535 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 719694 T4535 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 719694 T4589 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@27e8c6f7 name:ZooKeeperConnection Watcher:127.0.0.1:51799/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 719694 T4535 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 719695 T4535 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 719696 T4593 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@756478b6 name:ZooKeeperConnection Watcher:127.0.0.1:51799/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 719696 T4535 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 719696 T4535 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 719697 T4597 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5109e0af name:ZooKeeperConnection Watcher:127.0.0.1:51799/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 719697 T4535 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 719697 T4535 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 719698 T4600 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7240c079 name:ZooKeeperConnection Watcher:127.0.0.1:51799/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 719698 T4535 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 719698 T4535 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 719699 T4603 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@24b7a74f name:ZooKeeperConnection Watcher:127.0.0.1:51799/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 719699 T4535 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 719699 T4535 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 719700 T4606 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2eb3ea2e name:ZooKeeperConnection Watcher:127.0.0.1:51799/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 719700 T4535 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 719717 T4545 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 719934 T4570 oasc.LeaderElectionTest$ClientThread.run ERROR setup failed org.apache.solr.common.SolrException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContextBase.<init>(ElectionContext.java:110)
[junit4] 2> at org.apache.solr.cloud.LeaderElectionTest$ClientThread.setupOnConnect(LeaderElectionTest.java:113)
[junit4] 2> at org.apache.solr.cloud.LeaderElectionTest$ClientThread.run(LeaderElectionTest.java:124)
[junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[junit4] 2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[junit4] 2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:226)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:223)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:223)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:100)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:94)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContextBase.<init>(ElectionContext.java:107)
[junit4] 2> ... 9 more
[junit4] 2>
[junit4] 2> 720184 T4521 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x1452f917ec40013, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2>
[junit4] 2> 721286 T4521 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x1452f917ec4000b, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2>
[junit4] 2> 721836 T4521 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x1452f917ec40014, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2>
[junit4] 2> 722386 T4521 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x1452f917ec4000c, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2>
[junit4] 2> 722937 T4521 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x1452f917ec40012, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2>
[junit4] 2> 723488 T4521 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x1452f917ec4000a, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2>
[junit4] 2> 723582 T4570 oasc.LeaderElectionTest$ClientThread.run ERROR setup failed org.apache.solr.common.SolrException: java.lang.InterruptedException
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContextBase.<init>(ElectionContext.java:113)
[junit4] 2> at org.apache.solr.cloud.LeaderElectionTest$ClientThread.setupOnConnect(LeaderElectionTest.java:113)
[junit4] 2> at org.apache.solr.cloud.LeaderElectionTest$ClientThread.run(LeaderElectionTest.java:124)
[junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[junit4] 2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[junit4] 2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2> Caused by: java.lang.InterruptedException
[junit4] 2> at java.lang.Object.wait(Native Method)
[junit4] 2> at java.lang.Object.wait(Object.java:502)
[junit4] 2> at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1342)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1040)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:226)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:223)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:223)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:100)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:94)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContextBase.<init>(ElectionContext.java:107)
[junit4] 2> ... 9 more
[junit4] 2>
[junit4] 2> 723582 T4558 oasc.LeaderElectionTest$ClientThread.run ERROR setup failed org.apache.solr.common.SolrException: java.lang.InterruptedException
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContextBase.<init>(ElectionContext.java:113)
[junit4] 2> at org.apache.solr.cloud.LeaderElectionTest$ClientThread.setupOnConnect(LeaderElectionTest.java:113)
[junit4] 2> at org.apache.solr.cloud.LeaderElectionTest$ClientThread.run(LeaderElectionTest.java:124)
[junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[junit4] 2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[junit4] 2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2> Caused by: java.lang.InterruptedException
[junit4] 2> at java.lang.Object.wait(Native Method)
[junit4] 2> at java.lang.Object.wait(Object.java:502)
[junit4] 2> at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1342)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1040)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:226)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:223)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:223)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:100)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:94)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContextBase.<init>(ElectionContext.java:107)
[junit4] 2> ... 9 more
[junit4] 2>
[junit4] 2> 723582 T4562 oasc.LeaderElectionTest$ClientThread.run ERROR setup failed org.apache.solr.common.SolrException: java.lang.InterruptedException
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContextBase.<init>(ElectionContext.java:113)
[junit4] 2> at org.apache.solr.cloud.LeaderElectionTest$ClientThread.setupOnConnect(LeaderElectionTest.java:113)
[junit4] 2> at org.apache.solr.cloud.LeaderElectionTest$ClientThread.run(LeaderElectionTest.java:124)
[junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[junit4] 2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[junit4] 2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2> Caused by: java.lang.InterruptedException
[junit4] 2> at java.lang.Object.wait(Native Method)
[junit4] 2> at java.lang.Object.wait(Object.java:502)
[junit4] 2> at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1342)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1040)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:226)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:223)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:223)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:100)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:94)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContextBase.<init>(ElectionContext.java:107)
[junit4] 2> ... 9 more
[junit4] 2>
[junit4] 2> 723582 T4541 oasc.LeaderElectionTest$ClientThread.run ERROR setup failed org.apache.solr.common.SolrException: java.lang.InterruptedException
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContextBase.<init>(ElectionContext.java:113)
[junit4] 2> at org.apache.solr.cloud.LeaderElectionTest$ClientThread.setupOnConnect(LeaderElectionTest.java:113)
[junit4] 2> at org.apache.solr.cloud.LeaderElectionTest$ClientThread.run(LeaderElectionTest.java:124)
[junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[junit4] 2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[junit4] 2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2> Caused by: java.lang.InterruptedException
[junit4] 2> at java.lang.Object.wait(Native Method)
[junit4] 2> at java.lang.Object.wait(Object.java:502)
[junit4] 2> at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1342)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1040)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:226)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:223)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:223)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:100)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:94)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContextBase.<init>(ElectionContext.java:107)
[junit4] 2> ... 9 more
[junit4] 2>
[junit4] 2> 723582 T4566 oasc.LeaderElectionTest$ClientThread.run ERROR setup failed org.apache.solr.common.SolrException: java.lang.InterruptedException
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContextBase.<init>(ElectionContext.java:113)
[junit4] 2> at org.apache.solr.cloud.LeaderElectionTest$ClientThread.setupOnConnect(LeaderElectionTest.java:113)
[junit4] 2> at org.apache.solr.cloud.LeaderElectionTest$ClientThread.run(LeaderElectionTest.java:124)
[junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[junit4] 2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[junit4] 2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2> Caused by: java.lang.InterruptedException
[junit4] 2> at java.lang.Object.wait(Native Method)
[junit4] 2> at java.lang.Object.wait(Object.java:502)
[junit4] 2> at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1342)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1040)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:226)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:223)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:223)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:100)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:94)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContextBase.<init>(ElectionContext.java:107)
[junit4] 2> ... 9 more
[junit4] 2>
[junit4] 2> 723582 T4586 oasc.LeaderElectionTest$ClientThread.run ERROR setup failed org.apache.solr.common.SolrException: java.lang.InterruptedException: sleep interrupted
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContextBase.<init>(ElectionContext.java:113)
[junit4] 2> at org.apache.solr.cloud.LeaderElectionTest$ClientThread.setupOnConnect(LeaderElectionTest.java:113)
[junit4] 2> at org.apache.solr.cloud.LeaderElectionTest$ClientThread.run(LeaderElectionTest.java:124)
[junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[junit4] 2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[junit4] 2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2> Caused by: java.lang.InterruptedException: sleep interrupted
[junit4] 2> at java.lang.Thread.sleep(Native Method)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryDelay(ZkCmdExecutor.java:119)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:87)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:223)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:100)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:94)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContextBase.<init>(ElectionContext.java:107)
[junit4] 2> ... 9 more
[junit4] 2>
[junit4] 2> 723582 T4554 oasc.LeaderElectionTest$ClientThread.run ERROR setup failed org.apache.solr.common.SolrException: java.lang.InterruptedException
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContextBase.<init>(ElectionContext.java:113)
[junit4] 2> at org.apache.solr.cloud.LeaderElectionTest$ClientThread.setupOnConnect(LeaderElectionTest.java:113)
[junit4] 2> at org.apache.solr.cloud.LeaderElectionTest$ClientThread.run(LeaderElectionTest.java:124)
[junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[junit4] 2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[junit4] 2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2> Caused by: java.lang.InterruptedException
[junit4] 2> at java.lang.Object.wait(Native Method)
[junit4] 2> at java.lang.Object.wait(Object.java:502)
[junit4] 2> at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1342)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1040)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:226)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:223)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:223)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:100)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:94)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContextBase.<init>(ElectionContext.java:107)
[junit4] 2> ... 9 more
[junit4] 2>
[junit4] 2> 723582 T4534 oasc.LeaderElectionTest$ClientThread.run ERROR setup failed org.apache.solr.common.SolrException: java.lang.InterruptedException
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContextBase.<init>(ElectionContext.java:113)
[junit4] 2> at org.apache.solr.cloud.LeaderElectionTest$ClientThread.setupOnConnect(LeaderElectionTest.java:113)
[junit4] 2> at org.apache.solr.cloud.LeaderElectionTest$ClientThread.run(LeaderElectionTest.java:124)
[junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[junit4] 2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[junit4] 2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2> Caused by: java.lang.InterruptedException
[junit4] 2> at java.lang.Object.wait(Native Method)
[junit4] 2> at java.lang.Object.wait(Object.java:502)
[junit4] 2> at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1342)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1040)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:226)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:223)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:223)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:100)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:94)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContextBase.<init>(ElectionContext.java:107)
[junit4] 2> ... 9 more
[junit4] 2>
[junit4] 2> 723582 T4582 oasc.LeaderElectionTest$ClientThread.run ERROR setup failed org.apache.solr.common.SolrException: java.lang.InterruptedException
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContextBase.<init>(ElectionContext.java:113)
[junit4] 2> at org.apache.solr.cloud.LeaderElectionTest$ClientThread.setupOnConnect(LeaderElectionTest.java:113)
[junit4] 2> at org.apache.solr.cloud.LeaderElectionTest$ClientThread.run(LeaderElectionTest.java:124)
[junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[junit4] 2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[junit4] 2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2> Caused by: java.lang.InterruptedException
[junit4] 2> at java.lang.Object.wait(Native Method)
[junit4] 2> at java.lang.Object.wait(Object.java:502)
[junit4] 2> at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1342)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1040)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:226)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:223)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:223)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:100)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:94)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContextBase.<init>(ElectionContext.java:107)
[junit4] 2> ... 9 more
[junit4] 2>
[junit4] 2> 723582 T4545 oasc.LeaderElectionTest$ClientThread.run ERROR setup failed org.apache.solr.common.SolrException: Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:136)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:142)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:110)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:267)
[junit4] 2> at org.apache.solr.cloud.LeaderElectionTest$ClientThread.setupOnConnect(LeaderElectionTest.java:116)
[junit4] 2> at org.apache.solr.cloud.LeaderElectionTest$ClientThread.run(LeaderElectionTest.java:124)
[junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[junit4] 2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[junit4] 2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2> Caused by: org.apache.solr.common.SolrException: java.lang.InterruptedException
[junit4] 2> at org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:40)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:123)
[junit4] 2> ... 12 more
[junit4] 2> Caused by: java.lang.InterruptedException
[junit4] 2> at java.lang.Object.wait(Native Method)
[junit4] 2> at java.lang.Object.wait(Object.java:502)
[junit4] 2> at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1342)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:781)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:432)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:429)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:386)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:373)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContextBase$1.execute(ElectionContext.java:128)
[junit4] 2> at org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:34)
[junit4] 2> ... 13 more
[junit4] 2>
[junit4] 2> 723582 T4550 oasc.LeaderElectionTest$ClientThread.run ERROR setup failed org.apache.solr.common.SolrException: java.lang.InterruptedException
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContextBase.<init>(ElectionContext.java:113)
[junit4] 2> at org.apache.solr.cloud.LeaderElectionTest$ClientThread.setupOnConnect(LeaderElectionTest.java:113)
[junit4] 2> at org.apache.solr.cloud.LeaderElectionTest$ClientThread.run(LeaderElectionTest.java:124)
[junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[junit4] 2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[junit4] 2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2> Caused by: java.lang.InterruptedException
[junit4] 2> at java.lang.Object.wait(Native Method)
[junit4] 2> at java.lang.Object.wait(Object.java:502)
[junit4] 2> at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1342)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1040)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:226)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:223)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:223)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:100)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:94)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContextBase.<init>(ElectionContext.java:107)
[junit4] 2> ... 9 more
[junit4] 2>
[junit4] 2> 723582 T4574 oasc.LeaderElectionTest$ClientThread.run ERROR setup failed org.apache.solr.common.SolrException: java.lang.InterruptedException
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContextBase.<init>(ElectionContext.java:113)
[junit4] 2> at org.apache.solr.cloud.LeaderElectionTest$ClientThread.setupOnConnect(LeaderElectionTest.java:113)
[junit4] 2> at org.apache.solr.cloud.LeaderElectionTest$ClientThread.run(LeaderElectionTest.java:124)
[junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[junit4] 2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[junit4] 2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2> Caused by: java.lang.InterruptedException
[junit4] 2> at java.lang.Object.wait(Native Method)
[junit4] 2> at java.lang.Object.wait(Object.java:502)
[junit4] 2> at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1342)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1040)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:226)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:223)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:223)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:100)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:94)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContextBase.<init>(ElectionContext.java:107)
[junit4] 2> ... 9 more
[junit4] 2>
[junit4] 2> 723582 T4590 oasc.LeaderElectionTest$ClientThread.run ERROR setup failed org.apache.solr.common.SolrException: java.lang.InterruptedException
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContextBase.<init>(ElectionContext.java:113)
[junit4] 2> at org.apache.solr.cloud.LeaderElectionTest$ClientThread.setupOnConnect(LeaderElectionTest.java:113)
[junit4] 2> at org.apache.solr.cloud.LeaderElectionTest$ClientThread.run(LeaderElectionTest.java:124)
[junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[junit4] 2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[junit4] 2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2> Caused by: java.lang.InterruptedException
[junit4] 2> at java.lang.Object.wait(Native Method)
[junit4] 2> at java.lang.Object.wait(Object.java:502)
[junit4] 2> at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1342)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1040)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:226)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:223)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:223)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:100)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:94)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContextBase.<init>(ElectionContext.java:107)
[junit4] 2> ... 9 more
[junit4] 2>
[junit4] 2> 723582 T4594 oasc.LeaderElectionTest$ClientThread.run ERROR setup failed org.apache.solr.common.SolrException: java.lang.InterruptedException
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContextBase.<init>(ElectionContext.java:113)
[junit4] 2> at org.apache.solr.cloud.LeaderElectionTest$ClientThread.setupOnConnect(LeaderElectionTest.java:113)
[junit4] 2> at org.apache.solr.cloud.LeaderElectionTest$ClientThread.run(LeaderElectionTest.java:124)
[junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[junit4] 2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[junit4] 2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2> Caused by: java.lang.InterruptedException
[junit4] 2> at java.lang.Object.wait(Native Method)
[junit4] 2> at java.lang.Object.wait(Object.java:502)
[junit4] 2> at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1342)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1040)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:226)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:223)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:223)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:100)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:94)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContextBase.<init>(ElectionContext.java:107)
[junit4] 2> ... 9 more
[junit4] 2>
[junit4] 2> 723583 T4521 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x1452f917ec40009, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2>
[junit4] 2> 723594 T4521 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x1452f917ec40007, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2>
[junit4] 2> 723594 T4521 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x1452f917ec40006, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2>
[junit4] 2> 723595 T4521 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x1452f917ec40010, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2>
[junit4] 2> 723595 T4521 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x1452f917ec40005, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2>
[junit4] 2> 723583 T4578 oasc.LeaderElectionTest$ClientThread.run ERROR setup failed org.apache.solr.common.SolrException: java.lang.InterruptedException
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContextBase.<init>(ElectionContext.java:113)
[junit4] 2> at org.apache.solr.cloud.LeaderElectionTest$ClientThread.setupOnConnect(LeaderElectionTest.java:113)
[junit4] 2> at org.apache.solr.cloud.LeaderElectionTest$ClientThread.run(LeaderElectionTest.java:124)
[junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[junit4] 2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[junit4] 2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2> Caused by: java.lang.InterruptedException
[junit4] 2> at java.lang.Object.wait(Native Method)
[junit4] 2> at java.lang.Object.wait(Object.java:502)
[junit4] 2> at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1342)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1040)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:226)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:223)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:223)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:100)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:94)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContextBase.<init>(ElectionContext.java:107)
[junit4] 2> ... 9 more
[junit4] 2>
[junit4] 2> 723595 T4521 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x1452f917ec40015, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2>
[junit4] 2> 723596 T4521 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x1452f917ec40008, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2>
[junit4] 2> 723597 T4521 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x1452f917ec4000d, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2>
[junit4] 2> 725715 T4523 oazsp.FileTxnLog.commit WARN fsync-ing the write ahead log in SyncThread:0 took 5997ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
[junit4] 2> 725716 T4523 oazs.NIOServerCnxn.sendBuffer ERROR Unexpected Exception: java.nio.channels.CancelledKeyException
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1081)
[junit4] 2> at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:404)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:169)
[junit4] 2>
[junit4] 2> 725718 T4523 oazs.NIOServerCnxn.sendBuffer ERROR Unexpected Exception: java.nio.channels.CancelledKeyException
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1081)
[junit4] 2> at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:404)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:200)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:131)
[junit4] 2>
[junit4] 2> 725719 T4523 oazs.NIOServerCnxn.sendBuffer ERROR Unexpected Exception: java.nio.channels.CancelledKeyException
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1081)
[junit4] 2> at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:404)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:200)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:131)
[junit4] 2>
[junit4] 2> 725719 T4523 oazs.NIOServerCnxn.sendBuffer ERROR Unexpected Exception: java.nio.channels.CancelledKeyException
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1081)
[junit4] 2> at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:404)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:200)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:131)
[junit4] 2>
[junit4] 2> 725719 T4523 oazs.NIOServerCnxn.sendBuffer ERROR Unexpected Exception: java.nio.channels.CancelledKeyException
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1081)
[junit4] 2> at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:404)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:200)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:131)
[junit4] 2>
[junit4] 2> 725719 T4523 oazs.NIOServerCnxn.sendBuffer ERROR Unexpected Exception: java.nio.channels.CancelledKeyException
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1081)
[junit4] 2> at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:404)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:200)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:131)
[junit4] 2>
[junit4] 2> 725720 T4523 oazs.NIOServerCnxn.sendBuffer ERROR Unexpected Exception: java.nio.channels.CancelledKeyException
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1081)
[junit4] 2> at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:404)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:200)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:131)
[junit4] 2>
[junit4] 2> 725720 T4523 oazs.NIOServerCnxn.sendBuffer ERROR Unexpected Exception: java.nio.channels.CancelledKeyException
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1081)
[junit4] 2> at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:404)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:200)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:131)
[junit4] 2>
[junit4] 2> 725720 T4523 oazs.NIOServerCnxn.sendBuffer ERROR Unexpected Exception: java.nio.channels.CancelledKeyException
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1081)
[junit4] 2> at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:404)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:200)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:131)
[junit4] 2>
[junit4] 2> 725721 T4523 oazs.NIOServerCnxn.sendBuffer ERROR Unexpected Exception: java.nio.channels.CancelledKeyException
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1081)
[junit4] 2> at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:404)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:200)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:131)
[junit4] 2>
[junit4] 2> 725721 T4523 oazs.NIOServerCnxn.sendBuffer ERROR Unexpected Exception: java.nio.channels.CancelledKeyException
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1081)
[junit4] 2> at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:404)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:200)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:131)
[junit4] 2>
[junit4] 2> 725721 T4523 oazs.NIOServerCnxn.sendBuffer ERROR Unexpected Exception: java.nio.channels.CancelledKeyException
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1081)
[junit4] 2> at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:404)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:200)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:131)
[junit4] 2>
[junit4] 2> 725721 T4523 oazs.NIOServerCnxn.sendBuffer ERROR Unexpected Exception: java.nio.channels.CancelledKeyException
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1081)
[junit4] 2> at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:404)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:200)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:131)
[junit4] 2>
[junit4] 2> 725722 T4523 oazs.NIOServerCnxn.sendBuffer ERROR Unexpected Exception: java.nio.channels.CancelledKeyException
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1081)
[junit4] 2> at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:404)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:200)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:131)
[junit4] 2>
[junit4] 2> 725722 T4523 oazs.NIOServerCnxn.sendBuffer ERROR Unexpected Exception: java.nio.channels.CancelledKeyException
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1081)
[junit4] 2> at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:404)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:200)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:131)
[junit4] 2>
[junit4] 2> 725723 T4523 oazs.NIOServerCnxn.sendBuffer ERROR Unexpected Exception: java.nio.channels.CancelledKeyException
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1081)
[junit4] 2> at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:404)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:200)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:131)
[junit4] 2>
[junit4] 2> 725723 T4523 oazs.NIOServerCnxn.sendBuffer ERROR Unexpected Exception: java.nio.channels.CancelledKeyException
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1081)
[junit4] 2> at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:404)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:200)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:131)
[junit4] 2>
[junit4] 2> 755770 T4508 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:51799 51799
[junit4] 2> 756122 T4520 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:51799 51799
[junit4] 2> 756123 T4508 oas.SolrTestCaseJ4.tearDown ###Ending testStressElection
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 1> / (2)
[junit4] 1> /overseer (1)
[junit4] 1> DATA:
[junit4] 1>
[junit4] 1> /overseer/queue (1)
[junit4] 1> /overseer/queue/qn-0000000000 (0)
[junit4] 1> DATA:
[junit4] 1> {
[junit4] 1> "operation":"leader",
[junit4] 1> "shard":"shard1",
[junit4] 1> "collection":"collection1",
[junit4] 1> "base_url":"0",
[junit4] 1> "core":"",
[junit4] 1> "state":"active"}
[junit4] 1> /collections (1)
[junit4] 1> /collections/collection1 (2)
[junit4] 1> /collections/collection1/leaders (0)
[junit4] 1> /collections/collection1/leader_elect (1)
[junit4] 1> /collections/collection1/leader_elect/shard1 (1)
[junit4] 1> /collections/collection1/leader_elect/shard1/election (0)
[junit4] 1>
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=LeaderElectionTest -Dtests.method=testStressElection -Dtests.seed=87C3BDDA0A20A364 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=fi_FI -Dtests.timezone=Europe/Prague -Dtests.file.encoding=UTF-8
[junit4] ERROR 38.7s J0 | LeaderElectionTest.testStressElection <<<
[junit4] > Throwable #1: java.lang.RuntimeException: Could not get leader props
[junit4] > at __randomizedtesting.SeedInfo.seed([87C3BDDA0A20A364:789569746D6EB277]:0)
[junit4] > at org.apache.solr.cloud.LeaderElectionTest.getLeaderUrl(LeaderElectionTest.java:225)
[junit4] > at org.apache.solr.cloud.LeaderElectionTest.getLeaderThread(LeaderElectionTest.java:325)
[junit4] > at org.apache.solr.cloud.LeaderElectionTest.testStressElection(LeaderElectionTest.java:434)
[junit4] > at java.lang.Thread.run(Thread.java:744)
[junit4] 2> 756131 T4508 oas.SolrTestCaseJ4.setUp ###Starting testBasic
[junit4] 2> 756131 T4508 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 756131 T4614 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 756222 T4593 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@756478b6 name:ZooKeeperConnection Watcher:127.0.0.1:51799/solr got event WatchedEvent state:Disconnected type:None path:null path:null type:None
[junit4] 2> 756223 T4593 oascc.ConnectionManager.process zkClient has disconnected
[junit4] 2> 756222 T4585 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@45ed81fd name:ZooKeeperConnection Watcher:127.0.0.1:51799/solr got event WatchedEvent state:Disconnected type:None path:null path:null type:None
[junit4] 2> 756223 T4585 oascc.ConnectionManager.process zkClient has disconnected
[junit4] 2> 756231 T4508 oasc.ZkTestServer.run start zk server on port:45357
[junit4] 2> 756232 T4508 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 756237 T4620 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2a8f2197 name:ZooKeeperConnection Watcher:127.0.0.1:45357 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 756237 T4508 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 756255 T4508 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 756256 T4622 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2
[...truncated too long message...]
[junit4] 2>
[junit4] 2> 781135 T4584 oazc.StaticHostProvider.next WARN Unexpected exception java.lang.InterruptedException: sleep interrupted
[junit4] 2> at java.lang.Thread.sleep(Native Method)
[junit4] 2> at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
[junit4] 2>
[junit4] 2> 781136 T4584 oaz.ClientCnxn$SendThread.run WARN Session 0x1452f917ec4000f for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
[junit4] 2> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
[junit4] 2> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
[junit4] 2> at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
[junit4] 2>
[junit4] 2> 67781 T4507 ccr.ThreadLeakControl.tryToInterruptAll SEVERE There are still zombie threads that couldn't be terminated:
[junit4] 2> 1) Thread[id=4584, name=Thread-2081-SendThread(127.0.0.1:51799), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]
[junit4] 2> at java.lang.Thread.sleep(Native Method)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
[junit4] 2> 2) Thread[id=4592, name=Thread-2081-SendThread(127.0.0.1:51799), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]
[junit4] 2> at java.lang.Thread.sleep(Native Method)
[junit4] 2> at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
[junit4] 2> NOTE: test params are: codec=Lucene46: {}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=false,coord=no): {}, locale=fi_FI, timezone=Europe/Prague
[junit4] 2> NOTE: Linux 3.8.0-38-generic amd64/Oracle Corporation 1.8.0 (64-bit)/cpus=8,threads=3,free=149420336,total=310378496
[junit4] 2> NOTE: All tests run in this JVM: [SolrInfoMBeanTest, TestDocumentBuilder, DocExpirationUpdateProcessorFactoryTest, ShowFileRequestHandlerTest, TestCollationFieldDocValues, PeerSyncTest, SolrCoreTest, TestCoreContainer, PreAnalyzedUpdateProcessorTest, CoreAdminHandlerTest, AnalysisAfterCoreReloadTest, TestSearcherReuse, TestSchemaResource, TestMaxScoreQueryParser, CoreAdminRequestStatusTest, TestLuceneMatchVersion, DistributedSpellCheckComponentTest, ExternalFileFieldSortTest, TestDistributedMissingSort, TestWriterPerf, UnloadDistributedZkTest, CopyFieldTest, TermVectorComponentTest, DistributedTermsComponentTest, TestReplicationHandler, HdfsBasicDistributedZkTest, TestComplexPhraseQParserPlugin, TestCursorMarkWithoutUniqueKey, TestAddFieldRealTimeGet, OverseerRolesTest, TestClassNameShortening, SyncSliceTest, BinaryUpdateRequestHandlerTest, SolrIndexSplitterTest, PrimitiveFieldTypeTest, DocumentBuilderTest, JsonLoaderTest, TestSolrDeletionPolicy1, TestSort, SpellPossibilityIteratorTest, TestSurroundQueryParser, TestMiniSolrCloudCluster, AnalysisErrorHandlingTest, TestExceedMaxTermLength, RegexBoostProcessorTest, BJQParserTest, MBeansHandlerTest, OverseerTest, CustomCollectionTest, TestDefaultSearchFieldResource, TestStressVersions, TestSuggestSpellingConverter, TestLRUCache, DeleteInactiveReplicaTest, BlockDirectoryTest, IndexSchemaTest, TestLMDirichletSimilarityFactory, AliasIntegrationTest, DirectUpdateHandlerOptimizeTest, OverseerStatusTest, MigrateRouteKeyTest, DocValuesMissingTest, CollectionsAPIAsyncDistributedZkTest, TestSerializedLuceneMatchVersion, BasicDistributedZkTest, TestCollapseQParserPlugin, TestAnalyzeInfixSuggestions, ClusterStateTest, TestCloudManagedSchema, UpdateRequestProcessorFactoryTest, SOLR749Test, ConnectionManagerTest, TestHighFrequencyDictionaryFactory, TestDynamicFieldResource, TestNonNRTOpen, ResourceLoaderTest, EchoParamsTest, FastVectorHighlighterTest, TestInitQParser, HdfsUnloadDistributedZkTest, TestJoin, TestSolrXmlPersistence, TestQueryUtils, TestSolrXml, TestNonDefinedSimilarityFactory, TestNoOpRegenerator, CacheHeaderTest, ResponseLogComponentTest, LeaderElectionTest]
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=LeaderElectionTest -Dtests.seed=87C3BDDA0A20A364 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=fi_FI -Dtests.timezone=Europe/Prague -Dtests.file.encoding=UTF-8
[junit4] ERROR 0.00s J0 | LeaderElectionTest (suite) <<<
[junit4] > Throwable #1: java.lang.AssertionError: ERROR: SolrZkClient opens=46 closes=44
[junit4] > at __randomizedtesting.SeedInfo.seed([87C3BDDA0A20A364]:0)
[junit4] > at org.apache.solr.SolrTestCaseJ4.endTrackingZkClients(SolrTestCaseJ4.java:469)
[junit4] > at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:195)
[junit4] > at java.lang.Thread.run(Thread.java:744)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: 4 threads leaked from SUITE scope at org.apache.solr.cloud.LeaderElectionTest:
[junit4] > 1) Thread[id=4593, name=Thread-2081-EventThread, state=WAITING, group=TGRP-LeaderElectionTest]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
[junit4] > at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
[junit4] > at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
[junit4] > 2) Thread[id=4584, name=Thread-2081-SendThread(127.0.0.1:51799), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]
[junit4] > at java.lang.Thread.sleep(Native Method)
[junit4] > at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
[junit4] > at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
[junit4] > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
[junit4] > 3) Thread[id=4585, name=Thread-2081-EventThread, state=WAITING, group=TGRP-LeaderElectionTest]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
[junit4] > at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
[junit4] > at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
[junit4] > 4) Thread[id=4592, name=Thread-2081-SendThread(127.0.0.1:51799), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]
[junit4] > at java.lang.Thread.sleep(Native Method)
[junit4] > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
[junit4] > at __randomizedtesting.SeedInfo.seed([87C3BDDA0A20A364]:0)Throwable #3: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
[junit4] > 1) Thread[id=4584, name=Thread-2081-SendThread(127.0.0.1:51799), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]
[junit4] > at java.lang.Thread.sleep(Native Method)
[junit4] > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
[junit4] > 2) Thread[id=4592, name=Thread-2081-SendThread(127.0.0.1:51799), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]
[junit4] > at java.lang.Thread.sleep(Native Method)
[junit4] > at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
[junit4] > at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
[junit4] > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
[junit4] > at __randomizedtesting.SeedInfo.seed([87C3BDDA0A20A364]:0)
[junit4] Completed on J0 in 67.82s, 4 tests, 1 failure, 3 errors <<< FAILURES!
[...truncated 595 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1276: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:908: There were test failures: 382 suites, 1611 tests, 3 suite-level errors, 1 error, 813 ignored (15 assumptions)
Total time: 43 minutes 39 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure