You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/08/23 11:44:49 UTC
[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_67) - Build #
11085 - Still Failing!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11085/
Java: 32bit/jdk1.7.0_67 -client -XX:+UseSerialGC
1 tests failed.
REGRESSION: org.apache.solr.cloud.OverseerTest.testOverseerFailure
Error Message:
Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed
Stack Trace:
org.apache.solr.common.SolrException: Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed
at __randomizedtesting.SeedInfo.seed([6001E52C352F5EF5:64096ADF278AB1D4]:0)
at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:150)
at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:163)
at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:125)
at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:155)
at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:314)
at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:221)
at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:157)
at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:662)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1
at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
at org.apache.solr.common.cloud.SolrZkClient$11.execute(SolrZkClient.java:462)
at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:74)
at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:459)
at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:416)
at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:403)
at org.apache.solr.cloud.ShardLeaderElectionContextBase$1.execute(ElectionContext.java:142)
at org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:31)
at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:137)
... 47 more
Build Log:
[...truncated 11517 lines...]
[junit4] Suite: org.apache.solr.cloud.OverseerTest
[junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.OverseerTest-6001E52C352F5EF5-001/init-core-data-001
[junit4] 2> 1281815 T4012 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (true)
[junit4] 2> 1281816 T4012 oas.SolrTestCaseJ4.initCore ####initCore
[junit4] 2> 1281816 T4012 oas.SolrTestCaseJ4.initCore ####initCore end
[junit4] 2> 1281821 T4012 oas.SolrTestCaseJ4.setUp ###Starting testReplay
[junit4] 2> 1281822 T4012 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 1281823 T4013 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 1281922 T4012 oasc.ZkTestServer.run start zk server on port:33410
[junit4] 2> 1281924 T4012 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1281926 T4019 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@14872fc name:ZooKeeperConnection Watcher:127.0.0.1:33410/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1281926 T4012 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1281927 T4012 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1281928 T4021 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@13e368a name:ZooKeeperConnection Watcher:127.0.0.1:33410 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1281928 T4012 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1281930 T4012 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1281931 T4023 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@11ad4bf name:ZooKeeperConnection Watcher:127.0.0.1:33410 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1281931 T4012 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1281931 T4012 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 1281934 T4012 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 1281935 T4012 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 1281936 T4012 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 1281937 T4012 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1281939 T4012 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 1281941 T4012 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1281942 T4025 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@61605f name:ZooKeeperConnection Watcher:127.0.0.1:33410/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1281942 T4012 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1281942 T4012 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: retry=false
[junit4] 2> 1281943 T4012 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 1281945 T4012 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 1281946 T4012 oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 1281947 T4012 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:33410_solr
[junit4] 2> 1281948 T4012 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 1281949 T4012 oasc.Overseer.start Overseer (id=92326161521115139-127.0.0.1:33410_solr-n_0000000000) starting
[junit4] 2> 1281950 T4012 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 1281952 T4012 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 1281954 T4012 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 1281956 T4012 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 1281958 T4012 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 1281960 T4012 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 1281960 T4027 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 1281962 T4026 oasc.Overseer$ClusterStateUpdater.run Replaying operations from work queue.
[junit4] 2> 1281962 T4026 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1/solr",
[junit4] 2> "node_name":"node1",
[junit4] 2> "shard":"s1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "core":"core1",
[junit4] 2> "roles":"",
[junit4] 2> "state":"recovering"}
[junit4] 2> 1281963 T4029 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 1281965 T4026 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1/solr",
[junit4] 2> "node_name":"node1",
[junit4] 2> "shard":"s1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "core":"core2",
[junit4] 2> "roles":"",
[junit4] 2> "state":"recovering"}
[junit4] 2> 1281965 T4029 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 1281966 T4026 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 1281968 T4026 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1/solr",
[junit4] 2> "node_name":"node1",
[junit4] 2> "shard":"s1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "core":"core3",
[junit4] 2> "roles":"",
[junit4] 2> "state":"recovering"}
[junit4] 2> 1281969 T4030 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1281970 T4029 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 1282064 T4012 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:33410 33410
[junit4] 2> 1282637 T4013 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:33410 33410
[junit4] 2> 1282638 T4012 oas.SolrTestCaseJ4.tearDown ###Ending testReplay
[junit4] 2> 1282639 T4012 oasc.Overseer.close Overseer (id=92326161521115139-127.0.0.1:33410_solr-n_0000000000) closing
[junit4] 2> 1282639 T4026 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:33410_solr
[junit4] 2> 1282645 T4031 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:307)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:304)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:74)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:304)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:321)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:89)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:293)
[junit4] 2>
[junit4] 2> 1282646 T4012 oas.SolrTestCaseJ4.setUp ###Starting testPlaceholders
[junit4] 2> 1282647 T4012 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 1282648 T4032 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 1282747 T4012 oasc.ZkTestServer.run start zk server on port:50393
[junit4] 2> 1282750 T4012 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1282752 T4038 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@185ca01 name:ZooKeeperConnection Watcher:127.0.0.1:50393/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1282752 T4012 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1282753 T4012 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1282754 T4040 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@a9d3e6 name:ZooKeeperConnection Watcher:127.0.0.1:50393 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1282755 T4012 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1282757 T4012 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1282760 T4042 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1be5223 name:ZooKeeperConnection Watcher:127.0.0.1:50393 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1282761 T4012 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1282761 T4012 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 1282764 T4012 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 1282767 T4012 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 1282769 T4012 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 1282770 T4012 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1282773 T4012 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1282774 T4044 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7dcdd5 name:ZooKeeperConnection Watcher:127.0.0.1:50393/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1282775 T4012 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1282776 T4012 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1282777 T4012 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
[junit4] 2> 1282787 T4012 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1282790 T4048 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b0942e name:ZooKeeperConnection Watcher:127.0.0.1:50393/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1282790 T4012 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1282790 T4012 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: retry=false
[junit4] 2> 1282792 T4012 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 1282793 T4012 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 1282795 T4012 oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 1282797 T4012 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:50393_solr
[junit4] 2> 1282797 T4012 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 1282799 T4012 oasc.Overseer.start Overseer (id=92326161575116804-127.0.0.1:50393_solr-n_0000000000) starting
[junit4] 2> 1282800 T4012 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 1282801 T4012 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 1282803 T4012 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 1282804 T4012 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 1282806 T4012 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 1282808 T4012 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 1282812 T4012 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 1282813 T4050 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 1282813 T4049 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 1282815 T4049 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=12 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"12",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 1282816 T4049 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3, shard4, shard5, shard6, shard7, shard8, shard9, shard10, shard11, shard12]
[junit4] 2> 1282816 T4049 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
[junit4] 2> 1282817 T4052 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1282818 T4046 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1282818 T4045 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1283314 T4012 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4] 2> 1283322 T4012 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
[junit4] 2> 1283327 T4052 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1283331 T4012 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:50393 50393
[junit4] 2> 1283881 T4032 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:50393 50393
[junit4] 2> 1283882 T4012 oas.SolrTestCaseJ4.tearDown ###Ending testPlaceholders
[junit4] 2> 1283882 T4012 oasc.Overseer.close Overseer (id=92326161575116804-127.0.0.1:50393_solr-n_0000000000) closing
[junit4] 2> 1283883 T4049 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:50393_solr
[junit4] 2> 1283884 T4053 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:307)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:304)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:74)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:304)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:321)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:89)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:293)
[junit4] 2>
[junit4] 2> 1283887 T4012 oas.SolrTestCaseJ4.setUp ###Starting testBadQueueItem
[junit4] 2> 1283888 T4012 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 1283888 T4054 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 1283988 T4012 oasc.ZkTestServer.run start zk server on port:38098
[junit4] 2> 1283989 T4012 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1283991 T4060 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@aa9b3d name:ZooKeeperConnection Watcher:127.0.0.1:38098 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1283991 T4012 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1283993 T4012 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1283994 T4062 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e6a6b0 name:ZooKeeperConnection Watcher:127.0.0.1:38098 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1283994 T4012 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1283994 T4012 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 1283997 T4012 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1283998 T4064 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1fd7556 name:ZooKeeperConnection Watcher:127.0.0.1:38098/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1283998 T4012 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1283998 T4012 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 1284000 T4012 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1284001 T4066 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a72b9a name:ZooKeeperConnection Watcher:127.0.0.1:38098/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1284001 T4012 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1284001 T4012 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: retry=false
[junit4] 2> 1284002 T4012 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 1284003 T4012 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 1284005 T4012 oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 1284006 T4012 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:38098_solr
[junit4] 2> 1284007 T4012 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 1284008 T4012 oasc.Overseer.start Overseer (id=92326161656381443-127.0.0.1:38098_solr-n_0000000000) starting
[junit4] 2> 1284009 T4012 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 1284010 T4012 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 1284012 T4012 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 1284014 T4012 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 1284015 T4012 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 1284017 T4012 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 1284021 T4012 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 1284021 T4068 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 1284022 T4067 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 1284022 T4012 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 1284024 T4012 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 1284028 T4012 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1284031 T4012 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1284033 T4071 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@16d1a33 name:ZooKeeperConnection Watcher:127.0.0.1:38098/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1284033 T4012 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1284034 T4012 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1284037 T4012 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1
[junit4] 2> 1284042 T4074 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1284044 T4067 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 1284044 T4067 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
[junit4] 2> 1284044 T4067 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
[junit4] 2> 1284047 T4073 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1284047 T4072 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1284542 T4012 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4] 2> 1284548 T4012 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
[junit4] 2> 1284550 T4074 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1284553 T4067 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core2",
[junit4] 2> "core_node_name":"node2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 1284553 T4067 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 1284553 T4067 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 1284554 T4074 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1284555 T4072 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1284555 T4073 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1285051 T4012 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 1285061 T4012 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4] 2> 1285067 T4074 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1285072 T4067 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core3",
[junit4] 2> "core_node_name":"node3",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 1285073 T4067 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 1285073 T4067 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1285076 T4074 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1285077 T4073 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1285077 T4072 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1285569 T4012 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1285580 T4012 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 1285586 T4074 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1285691 T4073 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1285692 T4072 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1285739 T4074 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1285743 T4067 oasc.Overseer$ClusterStateUpdater.checkKeyExistence ERROR Skipping invalid Overseer message because it has no collection specified: {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core0",
[junit4] 2> "core_node_name":"node0",
[junit4] 2> "collection":"",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 1285747 T4067 oasc.Overseer$ClusterStateUpdater.checkKeyExistence ERROR Skipping invalid Overseer message because it has no collection specified: {
[junit4] 2> "operation":"deletecore",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core0",
[junit4] 2> "core_node_name":"node0",
[junit4] 2> "collection":""}
[junit4] 2> 1285751 T4074 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1285752 T4067 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection2",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 1285752 T4067 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection2 with shards [shard1, shard2, shard3]
[junit4] 2> 1285753 T4067 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
[junit4] 2> 1285858 T4072 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1285858 T4073 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1286245 T4012 oascc.SolrZkClient.makePath makePath: /collections/collection2/leader_elect/shard3/election
[junit4] 2> 1286250 T4012 oascc.SolrZkClient.makePath makePath: /collections/collection2/leaders/shard3
[junit4] 2> 1286255 T4074 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1286258 T4067 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core2",
[junit4] 2> "core_node_name":"node2",
[junit4] 2> "collection":"collection2",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 1286258 T4067 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 1286258 T4067 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 1286259 T4074 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1286260 T4073 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1286260 T4072 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1286756 T4012 oascc.SolrZkClient.makePath makePath: /collections/collection2/leader_elect/shard2/election
[junit4] 2> 1286760 T4012 oascc.SolrZkClient.makePath makePath: /collections/collection2/leaders/shard2
[junit4] 2> 1286763 T4074 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1286765 T4067 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core3",
[junit4] 2> "core_node_name":"node3",
[junit4] 2> "collection":"collection2",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 1286766 T4067 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 1286766 T4067 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1286767 T4074 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1286768 T4072 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1286768 T4073 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1287264 T4012 oascc.SolrZkClient.makePath makePath: /collections/collection2/leader_elect/shard1/election
[junit4] 2> 1287270 T4012 oascc.SolrZkClient.makePath makePath: /collections/collection2/leaders/shard1
[junit4] 2> 1287274 T4074 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1287378 T4073 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1287378 T4072 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1287428 T4012 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:38098 38098
[junit4] 2> 1288025 T4068 oasc.OverseerCollectionProcessor.amILeader org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:307)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:304)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:74)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:304)
[junit4] 2> at org.apache.solr.cloud.OverseerCollectionProcessor.amILeader(OverseerCollectionProcessor.java:513)
[junit4] 2> at org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:271)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2>
[junit4] 2> 1288146 T4068 oasc.OverseerCollectionProcessor.amILeader According to ZK I (id=92326161656381443-127.0.0.1:38098_solr-n_0000000000) am no longer a leader.
[junit4] 2> 1288146 T4054 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:38098 38098
[junit4] 2> 1288148 T4012 oas.SolrTestCaseJ4.tearDown ###Ending testBadQueueItem
[junit4] 2> 1288149 T4012 oasc.Overseer.close Overseer (id=92326161656381443-127.0.0.1:38098_solr-n_0000000000) closing
[junit4] 2> 1288149 T4067 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:38098_solr
[junit4] 2> 1288151 T4075 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:307)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:304)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:74)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:304)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:321)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:89)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:293)
[junit4] 2>
[junit4] 2> 1288155 T4012 oas.SolrTestCaseJ4.setUp ###Starting testShardAssignment
[junit4] 2> 1288155 T4012 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 1288156 T4076 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 1288256 T4012 oasc.ZkTestServer.run start zk server on port:50654
[junit4] 2> 1288257 T4012 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1288259 T4082 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@965bd6 name:ZooKeeperConnection Watcher:127.0.0.1:50654 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1288259 T4012 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1288261 T4012 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1288262 T4084 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@b3a774 name:ZooKeeperConnection Watcher:127.0.0.1:50654 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1288266 T4012 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1288266 T4012 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 1288270 T4012 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1288272 T4086 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@150477a name:ZooKeeperConnection Watcher:127.0.0.1:50654/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1288272 T4012 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1288272 T4012 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 1288275 T4012 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1288276 T4088 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@981cae name:ZooKeeperConnection Watcher:127.0.0.1:50654/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1288276 T4012 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1288277 T4012 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: retry=false
[junit4] 2> 1288278 T4012 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 1288280 T4012 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 1288281 T4012 oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 1288283 T4012 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:50654_solr
[junit4] 2> 1288283 T4012 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 1288285 T4012 oasc.Overseer.start Overseer (id=92326161936285699-127.0.0.1:50654_solr-n_0000000000) starting
[junit4] 2> 1288286 T4012 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 1288290 T4012 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 1288292 T4012 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 1288294 T4012 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 1288296 T4012 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 1288298 T4012 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 1288302 T4012 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 1288303 T4089 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 1288303 T4090 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 1288310 T4012 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 1288314 T4012 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 1288316 T4012 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1288330 T4012 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1288330 T4093 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@181ec36 name:ZooKeeperConnection Watcher:127.0.0.1:50654/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1288331 T4012 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1288332 T4012 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1288334 T4012 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1
[junit4] 2> 1288337 T4096 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1288338 T4089 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 1288338 T4089 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
[junit4] 2> 1288339 T4089 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
[junit4] 2> 1288341 T4094 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1288341 T4095 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1288837 T4012 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4] 2> 1288842 T4012 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
[junit4] 2> 1288846 T4096 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1288849 T4089 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core2",
[junit4] 2> "core_node_name":"node2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 1288849 T4089 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 1288850 T4089 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 1288851 T4096 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1288852 T4095 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1288853 T4094 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1289347 T4012 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 1289352 T4012 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4] 2> 1289354 T4096 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1289356 T4089 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core3",
[junit4] 2> "core_node_name":"node3",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 1289357 T4089 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 1289357 T4089 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1289358 T4096 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1289358 T4094 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1289358 T4095 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1289855 T4012 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1289862 T4012 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 1289866 T4096 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1289871 T4089 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core4",
[junit4] 2> "core_node_name":"node4",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 1289872 T4089 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 1289872 T4089 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
[junit4] 2> 1289875 T4096 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1289876 T4095 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1289876 T4094 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1290368 T4012 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4] 2> 1290378 T4096 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1290380 T4089 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core5",
[junit4] 2> "core_node_name":"node5",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 1290380 T4089 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 1290381 T4089 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 1290485 T4095 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1290485 T4094 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1290878 T4012 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 1290883 T4096 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1290884 T4089 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core6",
[junit4] 2> "core_node_name":"node6",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 1290885 T4089 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 1290885 T4089 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1290988 T4094 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1290989 T4095 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1291383 T4012 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1291390 T4093 oaz.ClientCnxn$EventThread.processEvent ERROR Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@117ea69 rejected from java.util.concurrent.ThreadPoolExecutor@11eb0c7[Shutting down, pool size = 2, active threads = 2, queued tasks = 0, completed tasks = 8]
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
[junit4] 2> at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:110)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$3.process(SolrZkClient.java:206)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
[junit4] 2>
[junit4] 2> 1291390 T4097 oasc.LeaderElector$ElectionWatcher.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard3/election
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:290)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:287)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:74)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:287)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:358)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$3$1.run(SolrZkClient.java:209)
[junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2>
[junit4] 2> 1292890 T4094 oasc.LeaderElector$ElectionWatcher.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard2/election
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:290)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:287)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:74)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:287)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:358)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$3$1.run(SolrZkClient.java:209)
[junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2>
[junit4] 2> 1292894 T4012 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:50654 50654
[junit4] 2> 1293443 T4076 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:50654 50654
[junit4] 2> 1293444 T4012 oas.SolrTestCaseJ4.tearDown ###Ending testShardAssignment
[junit4] 2> 1293445 T4012 oasc.Overseer.close Overseer (id=92326161936285699-127.0.0.1:50654_solr-n_0000000000) closing
[junit4] 2> 1293445 T4089 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:50654_solr
[junit4] 2> 1293447 T4099 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:307)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:304)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:74)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:304)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:321)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:89)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:293)
[junit4] 2>
[junit4] 2> 1293451 T4012 oas.SolrTestCaseJ4.setUp ###Starting testDoubleAssignment
[junit4] 2> 1293452 T4012 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 1293452 T4100 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 1293552 T4012 oasc.ZkTestServer.run start zk server on port:51410
[junit4] 2> 1293553 T4012 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1293556 T4106 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@16a9062 name:ZooKeeperConnection Watcher:127.0.0.1:51410/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1293556 T4012 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1293557 T4012 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1293559 T4108 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@19eef99 name:ZooKeeperConnection Watcher:127.0.0.1:51410 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1293559 T4012 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1293562 T4012 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1293563 T4110 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@bbbfc5 name:ZooKeeperConnection Watcher:127.0.0.1:51410 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1293563 T4012 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1293564 T4012 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 1293569 T4012 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 1293573 T4012 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 1293576 T4012 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 1293578 T4012 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1293581 T4012 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1293583 T4112 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1eed4b name:ZooKeeperConnection Watcher:127.0.0.1:51410/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1293583 T4012 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1293585 T4012 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1293587 T4012 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
[junit4] 2> 1293589 T4012 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1293590 T4116 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@68fc3a name:ZooKeeperConnection Watcher:127.0.0.1:51410/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1293590 T4012 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1293590 T4012 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: retry=false
[junit4] 2> 1293591 T4012 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 1293593 T4012 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 1293594 T4012 oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 1293596 T4012 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:51410_solr
[junit4] 2> 1293596 T4012 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 1293597 T4012 oasc.Overseer.start Overseer (id=92326162283167748-127.0.0.1:51410_solr-n_0000000000) starting
[junit4] 2> 1293599 T4012 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 1293600 T4012 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 1293602 T4012 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 1293604 T4012 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 1293606 T4012 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 1293608 T4012 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 1293611 T4012 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 1293612 T4118 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 1293612 T4117 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 1293614 T4117 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 1293614 T4117 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1]
[junit4] 2> 1293615 T4117 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1293617 T4120 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1293617 T4113 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1293617 T4114 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1294113 T4012 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1294125 T4012 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 1294132 T4120 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1294137 T4012 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1294138 T4122 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@181bf68 name:ZooKeeperConnection Watcher:127.0.0.1:51410/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1294138 T4012 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1294140 T4012 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1294144 T4012 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
[junit4] 2> 1294152 T4120 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1294152 T4012 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1294154 T4117 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 1294154 T4117 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
[junit4] 2> 1294159 T4012 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 1294164 T4120 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1294269 T4114 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1294269 T4123 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1294274 T4012 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:51410 51410
[junit4] 2> 1294951 T4100 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:51410 51410
[junit4] 2> 1294954 T4012 oas.SolrTestCaseJ4.tearDown ###Ending testDoubleAssignment
[junit4] 2> 1294954 T4012 oasc.Overseer.close Overseer (id=92326162283167748-127.0.0.1:51410_solr-n_0000000000) closing
[junit4] 2> 1294955 T4117 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:51410_solr
[junit4] 2> 1294958 T4124 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:307)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:304)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:74)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:304)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:321)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:89)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:293)
[junit4] 2>
[junit4] 2> 1294965 T4012 oas.SolrTestCaseJ4.setUp ###Starting testShardLeaderChange
[junit4] 2> 1294966 T4012 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 1294967 T4125 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 1295066 T4012 oasc.ZkTestServer.run start zk server on port:37925
[junit4] 2> 1295067 T4012 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1295070 T4131 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2bd309 name:ZooKeeperConnection Watcher:127.0.0.1:37925/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1295071 T4012 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1295072 T4012 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1295073 T4133 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d437f5 name:ZooKeeperConnection Watcher:127.0.0.1:37925 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1295074 T4012 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1295078 T4012 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1295079 T4135 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@50b04f name:ZooKeeperConnection Watcher:127.0.0.1:37925 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1295080 T4012 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1295081 T4012 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 1295086 T4012 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 1295090 T4012 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1295090 T4136 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1295092 T4139 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@888cdf name:ZooKeeperConnection Watcher:127.0.0.1:37925/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1295092 T4140 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d8c24b name:ZooKeeperConnection Watcher:127.0.0.1:37925/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1295092 T4012 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1295093 T4136 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1295094 T4136 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: retry=false
[junit4] 2> 1295094 T4012 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 1295096 T4136 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 1295098 T4012 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 1295098 T4136 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 1295100 T4012 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1295101 T4136 oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 1295104 T4012 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
[junit4] 2> 1295104 T4136 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:37925_solr
[junit4] 2> 1295105 T4136 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 1295108 T4136 oasc.Overseer.start Overseer (id=92326162382520324-127.0.0.1:37925_solr-n_0000000000) starting
[junit4] 2> 1295108 T4012 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 1295109 T4136 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 1295112 T4136 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 1295115 T4136 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 1295118 T4136 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 1295121 T4136 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 1295123 T4136 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 1295127 T4136 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 1295128 T4143 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 1295128 T4136 oasc.OverseerTest$OverseerRestarter.run Killing overseer.
[junit4] 2> 1295129 T4142 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 1295129 T4142 oasc.Overseer$ClusterStateUpdater.amILeader org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:307)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:304)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:74)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:304)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.amILeader(Overseer.java:606)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:211)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2>
[junit4] 2> 1295130 T4136 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1295130 T4142 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=92326162382520324-127.0.0.1:37925_solr-n_0000000000) am no longer a leader.
[junit4] 2> 1295130 T4142 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:37925_solr
[junit4] 2> 1295131 T4146 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@bcd91 name:ZooKeeperConnection Watcher:127.0.0.1:37925/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1295131 T4136 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1295132 T4147 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:307)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:304)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:74)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:304)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:321)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:89)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:293)
[junit4] 2>
[junit4] 2> 1295132 T4136 oasc.Overseer.close Overseer (id=92326162382520324-127.0.0.1:37925_solr-n_0000000000) closing
[junit4] 2> 1295133 T4143 oasc.OverseerCollectionProcessor.run ERROR Unable to prioritize overseer org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /roles.json
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:256)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:253)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:74)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:253)
[junit4] 2> at org.apache.solr.cloud.OverseerCollectionProcessor.prioritizeOverseerNodes(OverseerCollectionProcessor.java:411)
[junit4] 2> at org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:258)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2>
[junit4] 2> 1295134 T4136 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: retry=false
[junit4] 2> 1295137 T4136 oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 1295140 T4136 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:37925_solr
[junit4] 2> 1295140 T4136 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 1295142 T4136 oasc.Overseer.start Overseer (id=92326162382520325-127.0.0.1:37925_solr-n_0000000001) starting
[junit4] 2> 1295156 T4136 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 1295156 T4149 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 1295157 T4148 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 1295160 T4148 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"state1",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 1295161 T4148 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1]
[junit4] 2> 1295161 T4148 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1295164 T4151 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1295165 T4141 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1295611 T4012 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1295620 T4012 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 1295626 T4151 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1295627 T4012 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1295630 T4148 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"state2",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 1295631 T4148 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
[junit4] 2> 1295633 T4012 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1295633 T4151 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1295634 T4153 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1492b92 name:ZooKeeperConnection Watcher:127.0.0.1:37925/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1295634 T4012 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 129
[...truncated too long message...]
java:783)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$11.execute(SolrZkClient.java:462)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:74)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:459)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:416)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:403)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContextBase$1.execute(ElectionContext.java:142)
[junit4] 2> at org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:31)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:137)
[junit4] 2> ... 10 more
[junit4] 2>
[junit4] 2> 1355522 T4012 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:57519 57519
[junit4] 2> 1356103 T4814 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:57519 57519
[junit4] 2> 1356105 T4012 oas.SolrTestCaseJ4.tearDown ###Ending testOverseerFailure
[junit4] 2> 1356106 T4012 oasc.Overseer.close Overseer (id=92326165173174277-127.0.0.1:57519_solr-n_0000000001) closing
[junit4] 2> 1356106 T4838 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:57519_solr
[junit4] 2> 1356107 T4843 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:307)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:304)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:74)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:304)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:321)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:89)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:293)
[junit4] 2>
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=OverseerTest -Dtests.method=testOverseerFailure -Dtests.seed=6001E52C352F5EF5 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=es_VE -Dtests.timezone=Africa/Djibouti -Dtests.file.encoding=ISO-8859-1
[junit4] ERROR 18.6s J1 | OverseerTest.testOverseerFailure <<<
[junit4] > Throwable #1: org.apache.solr.common.SolrException: Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed
[junit4] > at __randomizedtesting.SeedInfo.seed([6001E52C352F5EF5:64096ADF278AB1D4]:0)
[junit4] > at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:150)
[junit4] > at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:163)
[junit4] > at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:125)
[junit4] > at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:155)
[junit4] > at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:314)
[junit4] > at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:221)
[junit4] > at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:157)
[junit4] > at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:662)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] > Caused by: org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1
[junit4] > at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
[junit4] > at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] > at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
[junit4] > at org.apache.solr.common.cloud.SolrZkClient$11.execute(SolrZkClient.java:462)
[junit4] > at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:74)
[junit4] > at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:459)
[junit4] > at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:416)
[junit4] > at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:403)
[junit4] > at org.apache.solr.cloud.ShardLeaderElectionContextBase$1.execute(ElectionContext.java:142)
[junit4] > at org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:31)
[junit4] > at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:137)
[junit4] > ... 47 more
[junit4] 2> 1359121 T4012 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4] 2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.OverseerTest-6001E52C352F5EF5-001
[junit4] 2> NOTE: test params are: codec=Lucene410: {}, docValues:{}, sim=DefaultSimilarity, locale=es_VE, timezone=Africa/Djibouti
[junit4] 2> NOTE: Linux 3.13.0-34-generic i386/Oracle Corporation 1.7.0_67 (32-bit)/cpus=8,threads=1,free=168530072,total=400728064
[junit4] 2> NOTE: All tests run in this JVM: [TestRecovery, CoreAdminRequestStatusTest, ScriptEngineTest, SolrPluginUtilsTest, ZkSolrClientTest, TestRealTimeGet, TestHighlightDedupGrouping, DateFieldTest, TermVectorComponentTest, DistribDocExpirationUpdateProcessorTest, SolrInfoMBeanTest, FileBasedSpellCheckerTest, StatsComponentTest, TestMultiCoreConfBootstrap, SoftAutoCommitTest, TestQueryTypes, TestPostingsSolrHighlighter, TestMissingGroups, SortByFunctionTest, TestFieldSortValues, TestReversedWildcardFilterFactory, QueryParsingTest, TestCSVResponseWriter, HighlighterMaxOffsetTest, TestLFUCache, TestDefaultSearchFieldResource, CollectionsAPIDistributedZkTest, TestReRankQParserPlugin, LegacyHTMLStripCharFilterTest, SuggestComponentTest, TestIndexingPerformance, TestQueryUtils, TestUpdate, TestDistributedGrouping, TestCopyFieldCollectionResource, CoreMergeIndexesAdminHandlerTest, TermVectorComponentDistributedTest, ConvertedLegacyTest, DocValuesTest, URLClassifyProcessorTest, TestHighFrequencyDictionaryFactory, TestPHPSerializedResponseWriter, TestPivotHelperCode, CopyFieldTest, TestNonDefinedSimilarityFactory, TestPerFieldSimilarity, RequestHandlersTest, TestExtendedDismaxParser, TestSolrQueryParser, ChaosMonkeySafeLeaderTest, TestManagedResourceStorage, TestSolrQueryParserResource, DistributedDebugComponentTest, TestCollationFieldDocValues, TestSolrQueryParserDefaultOperatorResource, UniqFieldsUpdateProcessorFactoryTest, TestLeaderElectionZkExpiry, CurrencyFieldXmlFileTest, CoreContainerCoreInitFailuresTest, TestValueSourceCache, TestOrdValues, TestDocBasedVersionConstraints, TestCollationField, TestStandardQParsers, TestWriterPerf, PreAnalyzedFieldTest, FunctionTest, MigrateRouteKeyTest, DistanceFunctionTest, SolrCoreTest, TestAnalyzeInfixSuggestions, TestMergePolicyConfig, CustomCollectionTest, AnalysisAfterCoreReloadTest, AddSchemaFieldsUpdateProcessorFactoryTest, SchemaVersionSpecificBehaviorTest, DebugComponentTest, TestNoOpRegenerator, DistributedFacetPivotSmallTest, AnalysisErrorHandlingTest, TestLazyCores, SolrRequestParserTest, TestOverriddenPrefixQueryForCustomFieldType, TestUtils, HdfsWriteToMultipleCollectionsTest, AliasIntegrationTest, TestSweetSpotSimilarityFactory, MultiTermTest, TestCloudInspectUtil, QueryElevationComponentTest, TestCSVLoader, TestCustomSort, TestAtomicUpdateErrorCases, ChangedSchemaMergeTest, ShardRoutingCustomTest, SearchHandlerTest, AutoCommitTest, TestInitQParser, HdfsBasicDistributedZkTest, TestZkChroot, TestReplicationHandlerBackup, DirectUpdateHandlerOptimizeTest, TestAddFieldRealTimeGet, BasicDistributedZk2Test, SyncSliceTest, OpenCloseCoreStressTest, OverseerTest]
[junit4] Completed on J1 in 77.34s, 9 tests, 1 error <<< FAILURES!
[...truncated 739 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:485: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:465: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:48: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:494: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1342: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:957: There were test failures: 430 suites, 1783 tests, 1 error, 40 ignored (19 assumptions)
Total time: 96 minutes 51 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 32bit/jdk1.7.0_67 -client -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_67) - Build #
11088 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11088/
Java: 32bit/jdk1.7.0_67 -server -XX:+UseG1GC
3 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.rest.TestManagedResourceStorage
Error Message:
SolrCore.getOpenCount()==2
Stack Trace:
java.lang.RuntimeException: SolrCore.getOpenCount()==2
at __randomizedtesting.SeedInfo.seed([DE98127AF886701B]:0)
at org.apache.solr.util.TestHarness.close(TestHarness.java:332)
at org.apache.solr.SolrTestCaseJ4.deleteCore(SolrTestCaseJ4.java:633)
at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:184)
at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:790)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at java.lang.Thread.run(Thread.java:745)
FAILED: junit.framework.TestSuite.org.apache.solr.rest.TestManagedResourceStorage
Error Message:
5 threads leaked from SUITE scope at org.apache.solr.rest.TestManagedResourceStorage: 1) Thread[id=2411, name=coreZkRegister-1550-thread-1, state=WAITING, group=TGRP-TestManagedResourceStorage] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 2) Thread[id=2405, name=OverseerHdfsCoreFailoverThread-92328293758402563-188.138.97.18:_-n_0000000000, state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.] at java.lang.Thread.sleep(Native Method) at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:136) at java.lang.Thread.run(Thread.java:745) 3) Thread[id=2403, name=OverseerStateUpdate-92328293758402563-188.138.97.18:_-n_0000000000, state=TIMED_WAITING, group=Overseer state updater.] at java.lang.Object.wait(Native Method) at org.apache.solr.cloud.DistributedQueue$LatchChildWatcher.await(DistributedQueue.java:266) at org.apache.solr.cloud.DistributedQueue.getChildren(DistributedQueue.java:309) at org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:582) at org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:560) at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:221) at java.lang.Thread.run(Thread.java:745) 4) Thread[id=2408, name=searcherExecutor-1556-thread-1, state=WAITING, group=TGRP-TestManagedResourceStorage] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 5) Thread[id=2409, name=Thread-862, state=WAITING, group=TGRP-TestManagedResourceStorage] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at org.apache.solr.core.CloserThread.run(CoreContainer.java:905)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 5 threads leaked from SUITE scope at org.apache.solr.rest.TestManagedResourceStorage:
1) Thread[id=2411, name=coreZkRegister-1550-thread-1, state=WAITING, group=TGRP-TestManagedResourceStorage]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2) Thread[id=2405, name=OverseerHdfsCoreFailoverThread-92328293758402563-188.138.97.18:_-n_0000000000, state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.]
at java.lang.Thread.sleep(Native Method)
at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:136)
at java.lang.Thread.run(Thread.java:745)
3) Thread[id=2403, name=OverseerStateUpdate-92328293758402563-188.138.97.18:_-n_0000000000, state=TIMED_WAITING, group=Overseer state updater.]
at java.lang.Object.wait(Native Method)
at org.apache.solr.cloud.DistributedQueue$LatchChildWatcher.await(DistributedQueue.java:266)
at org.apache.solr.cloud.DistributedQueue.getChildren(DistributedQueue.java:309)
at org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:582)
at org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:560)
at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:221)
at java.lang.Thread.run(Thread.java:745)
4) Thread[id=2408, name=searcherExecutor-1556-thread-1, state=WAITING, group=TGRP-TestManagedResourceStorage]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
5) Thread[id=2409, name=Thread-862, state=WAITING, group=TGRP-TestManagedResourceStorage]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:503)
at org.apache.solr.core.CloserThread.run(CoreContainer.java:905)
at __randomizedtesting.SeedInfo.seed([DE98127AF886701B]:0)
FAILED: junit.framework.TestSuite.org.apache.solr.rest.TestManagedResourceStorage
Error Message:
There are still zombie threads that couldn't be terminated: 1) Thread[id=2409, name=Thread-862, state=WAITING, group=TGRP-TestManagedResourceStorage] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at org.apache.solr.core.CloserThread.run(CoreContainer.java:905) 2) Thread[id=2411, name=coreZkRegister-1550-thread-1, state=WAITING, group=TGRP-TestManagedResourceStorage] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 3) Thread[id=2405, name=OverseerHdfsCoreFailoverThread-92328293758402563-188.138.97.18:_-n_0000000000, state=RUNNABLE, group=Overseer Hdfs SolrCore Failover Thread.] at java.lang.Thread.sleep(Native Method) at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:136) at java.lang.Thread.run(Thread.java:745) 4) Thread[id=2408, name=searcherExecutor-1556-thread-1, state=WAITING, group=TGRP-TestManagedResourceStorage] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
1) Thread[id=2409, name=Thread-862, state=WAITING, group=TGRP-TestManagedResourceStorage]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:503)
at org.apache.solr.core.CloserThread.run(CoreContainer.java:905)
2) Thread[id=2411, name=coreZkRegister-1550-thread-1, state=WAITING, group=TGRP-TestManagedResourceStorage]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
3) Thread[id=2405, name=OverseerHdfsCoreFailoverThread-92328293758402563-188.138.97.18:_-n_0000000000, state=RUNNABLE, group=Overseer Hdfs SolrCore Failover Thread.]
at java.lang.Thread.sleep(Native Method)
at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:136)
at java.lang.Thread.run(Thread.java:745)
4) Thread[id=2408, name=searcherExecutor-1556-thread-1, state=WAITING, group=TGRP-TestManagedResourceStorage]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
at __randomizedtesting.SeedInfo.seed([DE98127AF886701B]:0)
Build Log:
[...truncated 11115 lines...]
[junit4] Suite: org.apache.solr.rest.TestManagedResourceStorage
[junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.rest.TestManagedResourceStorage-DE98127AF886701B-001/init-core-data-001
[junit4] 2> 813258 T2389 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
[junit4] 2> 813258 T2389 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 813259 T2390 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 813359 T2389 oasc.ZkTestServer.run start zk server on port:45355
[junit4] 2> 813360 T2389 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 813362 T2396 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@133ff2c name:ZooKeeperConnection Watcher:127.0.0.1:45355 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 813362 T2389 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 813363 T2389 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 813366 T2389 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 813367 T2398 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6944d1 name:ZooKeeperConnection Watcher:127.0.0.1:45355/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 813367 T2389 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 813368 T2389 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 813370 T2389 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 813372 T2389 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 813373 T2389 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 813374 T2389 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 813374 T2389 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 813377 T2389 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4] 2> 813377 T2389 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 813379 T2389 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 813379 T2389 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 813380 T2389 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 813381 T2389 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 813382 T2389 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 813382 T2389 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 813385 T2389 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 813385 T2389 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 813388 T2389 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 813389 T2389 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
[junit4] 2> 813392 T2389 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 813392 T2389 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 813396 T2389 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 813396 T2389 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 813399 T2389 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 813399 T2389 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 813401 T2389 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 813401 T2389 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 813403 T2389 oas.SolrTestCaseJ4.initCore ####initCore
[junit4] 2> 813404 T2389 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/'
[junit4] 2> 813404 T2389 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/.svn/' to classloader
[junit4] 2> 813404 T2389 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/classes/' to classloader
[junit4] 2> 813405 T2389 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/README' to classloader
[junit4] 2> 813446 T2389 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
[junit4] 2> 813483 T2389 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 813484 T2389 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 813490 T2389 oass.IndexSchema.readSchema [null] Schema name=test
[junit4] 2> 813725 T2389 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 813729 T2389 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 813731 T2389 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 813736 T2389 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 813738 T2389 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 813740 T2389 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 813741 T2389 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 813741 T2389 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 813741 T2389 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 813741 T2389 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 813742 T2389 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 813742 T2389 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 813742 T2389 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr
[junit4] 2> 813742 T2389 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/'
[junit4] 2> 813775 T2389 oasc.CoreContainer.<init> New CoreContainer 22436957
[junit4] 2> 813775 T2389 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/]
[junit4] 2> 813776 T2389 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4] 2> 813776 T2389 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
[junit4] 2> 813777 T2389 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4] 2> 813777 T2389 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 813777 T2389 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 813777 T2389 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 813777 T2389 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 813778 T2389 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 813778 T2389 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 813778 T2389 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=30000&connTimeout=30000&retry=false
[junit4] 2> 813778 T2389 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 813779 T2389 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 813779 T2389 oasc.CoreContainer.load Host Name:
[junit4] 2> 813779 T2389 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:45355/solr
[junit4] 2> 813779 T2389 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 813780 T2389 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 813781 T2400 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@c9e89c name:ZooKeeperConnection Watcher:127.0.0.1:45355 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 813781 T2389 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 813785 T2389 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 813786 T2402 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@173b439 name:ZooKeeperConnection Watcher:127.0.0.1:45355/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 813786 T2389 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 813787 T2389 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 813789 T2389 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 813791 T2389 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 813792 T2389 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 813794 T2389 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 813795 T2389 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 813796 T2389 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/188.138.97.18:_
[junit4] 2> 813797 T2389 oascc.SolrZkClient.makePath makePath: /live_nodes/188.138.97.18:_
[junit4] 2> 813798 T2389 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 813799 T2389 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 813800 T2389 oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 813801 T2389 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 188.138.97.18:_
[junit4] 2> 813802 T2389 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 813803 T2389 oasc.Overseer.start Overseer (id=92328293758402563-188.138.97.18:_-n_0000000000) starting
[junit4] 2> 813804 T2389 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 813809 T2389 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 813809 T2404 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 813810 T2389 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 813811 T2389 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 813812 T2389 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 813813 T2403 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 813816 T2406 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 813816 T2406 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 813817 T2406 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 813817 T2407 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 813818 T2403 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4] 2> "state":"down",
[junit4] 2> "operation":"state",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core_node_name":"188.138.97.18:__collection1",
[junit4] 2> "node_name":"188.138.97.18:_",
[junit4] 2> "roles":null,
[junit4] 2> "core":"collection1",
[junit4] 2> "base_url":"http://188.138.97.18:",
[junit4] 2> "collection":"collection1"}
[junit4] 2> 813820 T2407 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 814817 T2406 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 814818 T2406 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 814818 T2406 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 814819 T2406 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 814819 T2406 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/'
[junit4] 2> 814820 T2406 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/.svn/' to classloader
[junit4] 2> 814820 T2406 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/classes/' to classloader
[junit4] 2> 814820 T2406 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/README' to classloader
[junit4] 2> 814869 T2406 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
[junit4] 2> 814922 T2406 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 814923 T2406 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 814929 T2406 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 815231 T2406 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 815237 T2406 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 815240 T2406 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 815250 T2406 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 815254 T2406 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 815257 T2406 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 815258 T2406 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 815258 T2406 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 815258 T2406 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 815259 T2406 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 815259 T2406 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 815260 T2406 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 815260 T2406 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 815260 T2406 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.rest.TestManagedResourceStorage-DE98127AF886701B-001/init-core-data-001/
[junit4] 2> 815261 T2406 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@89dbeb
[junit4] 2> 815261 T2406 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.rest.TestManagedResourceStorage-DE98127AF886701B-001/init-core-data-001
[junit4] 2> 815262 T2406 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.rest.TestManagedResourceStorage-DE98127AF886701B-001/init-core-data-001/index/
[junit4] 2> 815262 T2406 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.rest.TestManagedResourceStorage-DE98127AF886701B-001/init-core-data-001/index' doesn't exist. Creating new index...
[junit4] 2> 815262 T2406 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.rest.TestManagedResourceStorage-DE98127AF886701B-001/init-core-data-001/index
[junit4] 2> 815263 T2406 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=16, maxMergeAtOnceExplicit=33, maxMergedSegmentMB=58.4873046875, floorSegmentMB=1.666015625, forceMergeDeletesPctAllowed=8.89732897402962, segmentsPerTier=18.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 815264 T2406 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1b4badb lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1f05ce5)),segFN=segments_1,generation=1}
[junit4] 2> 815264 T2406 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 815264 T2406 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
[junit4] 2> 815269 T2406 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 815269 T2406 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 815269 T2406 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe-allfields"
[junit4] 2> 815270 T2406 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
[junit4] 2> 815270 T2406 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 815270 T2406 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 815271 T2406 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "uniq-fields"
[junit4] 2> 815271 T2406 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
[junit4] 2> 815271 T2406 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 815272 T2406 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 815272 T2406 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 815272 T2406 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 815273 T2406 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 815273 T2406 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 815274 T2406 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 815274 T2406 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 815274 T2406 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 815275 T2406 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 815275 T2406 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 815276 T2406 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 815276 T2406 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
[junit4] 2> 815277 T2406 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
[junit4] 2> 815277 T2406 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 815277 T2406 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 815278 T2406 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 815278 T2406 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
[junit4] 2> 815278 T2406 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
[junit4] 2> 815279 T2406 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
[junit4] 2> 815279 T2406 oasc.RequestHandlers.initHandlersFromConfig WARN Multiple requestHandler registered to the same name: /update ignoring: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 815280 T2406 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 815280 T2406 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
[junit4] 2> 815280 T2406 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
[junit4] 2> 815281 T2406 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
[junit4] 2> 815281 T2406 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
[junit4] 2> 815282 T2406 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
[junit4] 2> 815282 T2406 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
[junit4] 2> 815283 T2406 oasc.RequestHandlers.initHandlersFromConfig created mltrh: org.apache.solr.handler.component.SearchHandler
[junit4] 2> 815283 T2406 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
[junit4] 2> 815284 T2406 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
[junit4] 2> 815284 T2406 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
[junit4] 2> 815289 T2406 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 815290 T2406 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 815292 T2406 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 815293 T2406 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 815295 T2406 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 815296 T2406 oasc.SolrCore.initDeprecatedSupport WARN solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4] 2> 815298 T2406 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 815298 T2406 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 815299 T2406 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=39, maxMergeAtOnceExplicit=43, maxMergedSegmentMB=85.1123046875, floorSegmentMB=0.6669921875, forceMergeDeletesPctAllowed=22.11236875098133, segmentsPerTier=35.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6087837040547225
[junit4] 2> 815300 T2406 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1b4badb lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1f05ce5)),segFN=segments_1,generation=1}
[junit4] 2> 815300 T2406 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 815300 T2406 oass.SolrIndexSearcher.<init> Opening Searcher@c26f0a[collection1] main
[junit4] 2> 815301 T2406 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 815302 T2406 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 815302 T2406 oasr.RestManager.init Initializing RestManager with initArgs: {storageIO=org.apache.solr.rest.ManagedResourceStorage$InMemoryStorageIO}
[junit4] 2> 815302 T2406 oasr.ManagedResourceStorage.load Reading _rest_managed.json using InMemoryStorage
[junit4] 2> 815302 T2406 oasr.ManagedResource.reloadFromStorage WARN No stored data found for /rest/managed
[junit4] 2> 815303 T2406 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using InMemoryStorage
[junit4] 2> 815303 T2406 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 815303 T2406 oashc.SpellCheckComponent.inform Initializing spell checkers
[junit4] 2> 815392 T2406 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4] 2> 815410 T2408 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@c26f0a[collection1] main{UninvertingDirectoryReader()}
[junit4] 2> 815414 T2406 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 815414 T2411 oasc.ZkController.register Register replica - core:collection1 address:http://188.138.97.18: collection:collection1 shard:shard1
[junit4] 2> 815415 T2411 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 815416 T2389 oas.SolrTestCaseJ4.initCore ####initCore end
[junit4] 2> 815420 T2411 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 815424 T2407 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 815424 T2411 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 815424 T2411 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 815424 T2389 oas.SolrTestCaseJ4.setUp ###Starting testFileBasedJsonStorage
[junit4] 2> 815424 T2411 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://188.138.97.18:/collection1/ shard1
[junit4] 2> 815425 T2411 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 815425 T2389 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.rest.TestManagedResourceStorage-DE98127AF886701B-001/json-storage-001/'
[junit4] 2> 815426 T2407 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 815429 T2407 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 815430 T2407 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 815439 T2389 oasr.ManagedResourceStorage$FileStorageIO.configure File-based storage initialized to use dir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.rest.TestManagedResourceStorage-DE98127AF886701B-001/json-storage-001/managed
[junit4] 2> 815440 T2389 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _test_foo.json using file:dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.rest.TestManagedResourceStorage-DE98127AF886701B-001/json-storage-001/managed
[junit4] 2> 815440 T2389 oasr.ManagedResourceStorage.load Reading _test_foo.json using file:dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.rest.TestManagedResourceStorage-DE98127AF886701B-001/json-storage-001/managed
[junit4] 2> 815440 T2389 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _test_foo.json using file:dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.rest.TestManagedResourceStorage-DE98127AF886701B-001/json-storage-001/managed
[junit4] 2> 815441 T2389 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _test_foo.json using file:dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.rest.TestManagedResourceStorage-DE98127AF886701B-001/json-storage-001/managed
[junit4] 2> 815441 T2389 oasr.ManagedResourceStorage.load Reading _test_foo.json using file:dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.rest.TestManagedResourceStorage-DE98127AF886701B-001/json-storage-001/managed
[junit4] 2> 815441 T2389 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _test_foo.json using file:dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.rest.TestManagedResourceStorage-DE98127AF886701B-001/json-storage-001/managed
[junit4] 2> 815442 T2389 oas.SolrTestCaseJ4.tearDown ###Ending testFileBasedJsonStorage
[junit4] 2> 815449 T2389 oas.SolrTestCaseJ4.setUp ###Starting testZkBasedJsonStorage
[junit4] 2> 815450 T2389 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './'
[junit4] 2> 815460 T2389 oascc.SolrZkClient.makePath makePath: /test
[junit4] 2> 815462 T2389 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /test
[junit4] 2> 815462 T2389 oascc.SolrZkClient.makePath makePath: /test/_test_foo.json
[junit4] 2> 815464 T2389 oasr.ManagedResourceStorage$ZooKeeperStorageIO$1.close Wrote 136 bytes to new znode /test/_test_foo.json
[junit4] 2> 815464 T2389 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _test_foo.json using ZooKeeperStorageIO:path=/test
[junit4] 2> 815464 T2389 oasr.ManagedResourceStorage.load Reading _test_foo.json using ZooKeeperStorageIO:path=/test
[junit4] 2> 815465 T2389 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 136 bytes from znode /test/_test_foo.json
[junit4] 2> 815465 T2389 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _test_foo.json using ZooKeeperStorageIO:path=/test
[junit4] 2> 815466 T2389 oasr.ManagedResourceStorage$ZooKeeperStorageIO$1.close Wrote 175 bytes to existing znode /test/_test_foo.json
[junit4] 2> 815466 T2389 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _test_foo.json using ZooKeeperStorageIO:path=/test
[junit4] 2> 815466 T2389 oasr.ManagedResourceStorage.load Reading _test_foo.json using ZooKeeperStorageIO:path=/test
[junit4] 2> 815467 T2389 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 175 bytes from znode /test/_test_foo.json
[junit4] 2> 815467 T2389 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _test_foo.json using ZooKeeperStorageIO:path=/test
[junit4] 2> 815468 T2407 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 815469 T2389 oas.SolrTestCaseJ4.tearDown ###Ending testZkBasedJsonStorage
[junit4] 2> 815470 T2389 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:45355 45355
[junit4] 2> 815478 T2411 oasc.ZkController.register We are http://188.138.97.18:/collection1/ and leader is http://188.138.97.18:/collection1/
[junit4] 2> 815811 T2404 oasc.OverseerCollectionProcessor.amILeader org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:307)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:304)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:74)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:304)
[junit4] 2> at org.apache.solr.cloud.OverseerCollectionProcessor.amILeader(OverseerCollectionProcessor.java:513)
[junit4] 2> at org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:271)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2>
[junit4] 2> 816151 T2390 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:45355 45355
[junit4] 2> 816148 T2411 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://188.138.97.18:
[junit4] 2> 816151 T2411 oasc.ZkController.checkRecovery WARN Skipping recovery according to sys prop solrcloud.skip.autorecovery
[junit4] 2> 816152 T2389 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4] 2> 816151 T2404 oasc.OverseerCollectionProcessor.amILeader According to ZK I (id=92328293758402563-188.138.97.18:_-n_0000000000) am no longer a leader.
[junit4] 2> 816152 T2411 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 816153 T2411 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 816153 T2411 oasc.ZkController.getLeaderInitiatedRecoveryState WARN Unable to read /collections/collection1/leader_initiated_recovery/shard1/188.138.97.18:__collection1 due to: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_initiated_recovery/shard1/188.138.97.18:__collection1
[junit4] 2> 816153 T2411 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 816154 T2411 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 816154 T2411 oasc.ZkContainer$2.run ERROR org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer/queue/qn-
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:273)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:270)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:74)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.create(SolrZkClient.java:270)
[junit4] 2> at org.apache.solr.cloud.DistributedQueue.createData(DistributedQueue.java:369)
[junit4] 2> at org.apache.solr.cloud.DistributedQueue.offer(DistributedQueue.java:353)
[junit4] 2> at org.apache.solr.cloud.ZkController.publish(ZkController.java:1120)
[junit4] 2> at org.apache.solr.cloud.ZkController.publish(ZkController.java:1038)
[junit4] 2> at org.apache.solr.cloud.ZkController.publish(ZkController.java:1034)
[junit4] 2> at org.apache.solr.core.ZkContainer$2.run(ZkContainer.java:207)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2>
[junit4] 2> 816155 T2411 oasc.SolrException.log ERROR :org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer/queue/qn-
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:273)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:270)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:74)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.create(SolrZkClient.java:270)
[junit4] 2> at org.apache.solr.cloud.DistributedQueue.createData(DistributedQueue.java:369)
[junit4] 2> at org.apache.solr.cloud.DistributedQueue.offer(DistributedQueue.java:353)
[junit4] 2> at org.apache.solr.cloud.ZkController.publish(ZkController.java:1120)
[junit4] 2> at org.apache.solr.cloud.ZkController.publish(ZkController.java:1038)
[junit4] 2> at org.apache.solr.cloud.ZkController.publish(ZkController.java:1034)
[junit4] 2> at org.apache.solr.cloud.ZkController.register(ZkController.java:849)
[junit4] 2> at org.apache.solr.cloud.ZkController.register(ZkController.java:763)
[junit4] 2> at org.apache.solr.core.ZkContainer$2.run(ZkContainer.java:200)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2>
[junit4] 2> 3211 T2388 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 5 leaked thread(s).
[junit4] 2> 23437 T2388 ccr.ThreadLeakControl.checkThreadLeaks SEVERE 5 threads leaked from SUITE scope at org.apache.solr.rest.TestManagedResourceStorage:
[junit4] 2> 1) Thread[id=2411, name=coreZkRegister-1550-thread-1, state=WAITING, group=TGRP-TestManagedResourceStorage]
[junit4] 2> at sun.misc.Unsafe.park(Native Method)
[junit4] 2> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
[junit4] 2> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 2) Thread[id=2405, name=OverseerHdfsCoreFailoverThread-92328293758402563-188.138.97.18:_-n_0000000000, state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.]
[junit4] 2> at java.lang.Thread.sleep(Native Method)
[junit4] 2> at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:136)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 3) Thread[id=2403, name=OverseerStateUpdate-92328293758402563-188.138.97.18:_-n_0000000000, state=TIMED_WAITING, group=Overseer state updater.]
[junit4] 2> at java.lang.Object.wait(Native Method)
[junit4] 2> at org.apache.solr.cloud.DistributedQueue$LatchChildWatcher.await(DistributedQueue.java:266)
[junit4] 2> at org.apache.solr.cloud.DistributedQueue.getChildren(DistributedQueue.java:309)
[junit4] 2> at org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:582)
[junit4] 2> at org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:560)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:221)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 4) Thread[id=2408, name=searcherExecutor-1556-thread-1, state=WAITING, group=TGRP-TestManagedResourceStorage]
[junit4] 2> at sun.misc.Unsafe.park(Native Method)
[junit4] 2> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
[junit4] 2> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 5) Thread[id=2409, name=Thread-862, state=WAITING, group=TGRP-TestManagedResourceStorage]
[junit4] 2> at java.lang.Object.wait(Native Method)
[junit4] 2> at java.lang.Object.wait(Object.java:503)
[junit4] 2> at org.apache.solr.core.CloserThread.run(CoreContainer.java:905)
[junit4] 2> 23439 T2388 ccr.ThreadLeakControl.tryToInterruptAll Starting to interrupt leaked threads:
[junit4] 2> 1) Thread[id=2411, name=coreZkRegister-1550-thread-1, state=WAITING, group=TGRP-TestManagedResourceStorage]
[junit4] 2> 2) Thread[id=2405, name=OverseerHdfsCoreFailoverThread-92328293758402563-188.138.97.18:_-n_0000000000, state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.]
[junit4] 2> 3) Thread[id=2403, name=OverseerStateUpdate-92328293758402563-188.138.97.18:_-n_0000000000, state=TIMED_WAITING, group=Overseer state updater.]
[junit4] 2> 4) Thread[id=2408, name=searcherExecutor-1556-thread-1, state=WAITING, group=TGRP-TestManagedResourceStorage]
[junit4] 2> 5) Thread[id=2409, name=Thread-862, state=WAITING, group=TGRP-TestManagedResourceStorage]
[junit4] 2> 836697 T2403 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 188.138.97.18:_
[junit4] 2> 836715 T2412 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:307)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:304)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:74)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:304)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:321)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:89)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:293)
[junit4] 2>
[junit4] 2> 26470 T2388 ccr.ThreadLeakControl.tryToInterruptAll SEVERE There are still zombie threads that couldn't be terminated:
[junit4] 2> 1) Thread[id=2409, name=Thread-862, state=WAITING, group=TGRP-TestManagedResourceStorage]
[junit4] 2> at java.lang.Object.wait(Native Method)
[junit4] 2> at java.lang.Object.wait(Object.java:503)
[junit4] 2> at org.apache.solr.core.CloserThread.run(CoreContainer.java:905)
[junit4] 2> 2) Thread[id=2411, name=coreZkRegister-1550-thread-1, state=WAITING, group=TGRP-TestManagedResourceStorage]
[junit4] 2> at sun.misc.Unsafe.park(Native Method)
[junit4] 2> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
[junit4] 2> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 3) Thread[id=2405, name=OverseerHdfsCoreFailoverThread-92328293758402563-188.138.97.18:_-n_0000000000, state=RUNNABLE, group=Overseer Hdfs SolrCore Failover Thread.]
[junit4] 2> at java.lang.Thread.sleep(Native Method)
[junit4] 2> at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:136)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 4) Thread[id=2408, name=searcherExecutor-1556-thread-1, state=WAITING, group=TGRP-TestManagedResourceStorage]
[junit4] 2> at sun.misc.Unsafe.park(Native Method)
[junit4] 2> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
[junit4] 2> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> NOTE: test params are: codec=Lucene410: {}, docValues:{}, sim=DefaultSimilarity, locale=ar_SD, timezone=Asia/Aqtau
[junit4] 2> NOTE: Linux 3.13.0-34-generic i386/Oracle Corporation 1.7.0_67 (32-bit)/cpus=8,threads=5,free=167968336,total=463470592
[junit4] 2> NOTE: All tests run in this JVM: [TestCoreContainer, TestDocBasedVersionConstraints, TestCollationFieldDocValues, PolyFieldTest, MultiThreadedOCPTest, TestIBSimilarityFactory, HdfsBasicDistributedZk2Test, TestUpdate, CoreAdminRequestStatusTest, CoreMergeIndexesAdminHandlerTest, MigrateRouteKeyTest, TestHighlightDedupGrouping, OutputWriterTest, DateFieldTest, TermVectorComponentTest, TestLeaderElectionZkExpiry, FunctionTest, ChaosMonkeySafeLeaderTest, CursorMarkTest, NumericFieldsTest, StatsComponentTest, DistributedQueryElevationComponentTest, CSVRequestHandlerTest, TestDistribDocBasedVersion, TestQueryUtils, TestCSVResponseWriter, DirectUpdateHandlerOptimizeTest, TestHighFrequencyDictionaryFactory, TestReloadAndDeleteDocs, TestSchemaVersionResource, TestReplicationHandlerBackup, SchemaVersionSpecificBehaviorTest, TestSolrQueryParser, AddSchemaFieldsUpdateProcessorFactoryTest, ConnectionManagerTest, HdfsBasicDistributedZkTest, TestCustomSort, AnalysisAfterCoreReloadTest, FileBasedSpellCheckerTest, TestInitQParser, IndexSchemaRuntimeFieldTest, HighlighterMaxOffsetTest, TestRequestStatusCollectionAPI, SpatialFilterTest, AbstractAnalyticsStatsTest, SearchHandlerTest, TestXIncludeConfig, CircularListTest, TestQuerySenderNoQuery, SolrIndexConfigTest, SuggesterFSTTest, UpdateRequestProcessorFactoryTest, SolrRequestParserTest, TestPostingsSolrHighlighter, TestBadConfig, TestFileDictionaryLookup, TestMergePolicyConfig, TestPhraseSuggestions, TestIndexSearcher, LeaderElectionIntegrationTest, TestAtomicUpdateErrorCases, RemoteQueryErrorTest, DistributedQueryComponentCustomSortTest, DistributedFacetPivotLongTailTest, TestMissingGroups, TestNonDefinedSimilarityFactory, TestSolrQueryParserResource, HdfsSyncSliceTest, TestManagedResourceStorage]
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestManagedResourceStorage -Dtests.seed=DE98127AF886701B -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ar_SD -Dtests.timezone=Asia/Aqtau -Dtests.file.encoding=UTF-8
[junit4] ERROR 0.00s J1 | TestManagedResourceStorage (suite) <<<
[junit4] > Throwable #1: java.lang.RuntimeException: SolrCore.getOpenCount()==2
[junit4] > at __randomizedtesting.SeedInfo.seed([DE98127AF886701B]:0)
[junit4] > at org.apache.solr.util.TestHarness.close(TestHarness.java:332)
[junit4] > at org.apache.solr.SolrTestCaseJ4.deleteCore(SolrTestCaseJ4.java:633)
[junit4] > at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:184)
[junit4] > at java.lang.Thread.run(Thread.java:745)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: 5 threads leaked from SUITE scope at org.apache.solr.rest.TestManagedResourceStorage:
[junit4] > 1) Thread[id=2411, name=coreZkRegister-1550-thread-1, state=WAITING, group=TGRP-TestManagedResourceStorage]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
[junit4] > at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] > 2) Thread[id=2405, name=OverseerHdfsCoreFailoverThread-92328293758402563-188.138.97.18:_-n_0000000000, state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.]
[junit4] > at java.lang.Thread.sleep(Native Method)
[junit4] > at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:136)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] > 3) Thread[id=2403, name=OverseerStateUpdate-92328293758402563-188.138.97.18:_-n_0000000000, state=TIMED_WAITING, group=Overseer state updater.]
[junit4] > at java.lang.Object.wait(Native Method)
[junit4] > at org.apache.solr.cloud.DistributedQueue$LatchChildWatcher.await(DistributedQueue.java:266)
[junit4] > at org.apache.solr.cloud.DistributedQueue.getChildren(DistributedQueue.java:309)
[junit4] > at org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:582)
[junit4] > at org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:560)
[junit4] > at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:221)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] > 4) Thread[id=2408, name=searcherExecutor-1556-thread-1, state=WAITING, group=TGRP-TestManagedResourceStorage]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
[junit4] > at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] > 5) Thread[id=2409, name=Thread-862, state=WAITING, group=TGRP-TestManagedResourceStorage]
[junit4] > at java.lang.Object.wait(Native Method)
[junit4] > at java.lang.Object.wait(Object.java:503)
[junit4] > at org.apache.solr.core.CloserThread.run(CoreContainer.java:905)
[junit4] > at __randomizedtesting.SeedInfo.seed([DE98127AF886701B]:0)Throwable #3: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
[junit4] > 1) Thread[id=2409, name=Thread-862, state=WAITING, group=TGRP-TestManagedResourceStorage]
[junit4] > at java.lang.Object.wait(Native Method)
[junit4] > at java.lang.Object.wait(Object.java:503)
[junit4] > at org.apache.solr.core.CloserThread.run(CoreContainer.java:905)
[junit4] > 2) Thread[id=2411, name=coreZkRegister-1550-thread-1, state=WAITING, group=TGRP-TestManagedResourceStorage]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
[junit4] > at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] > 3) Thread[id=2405, name=OverseerHdfsCoreFailoverThread-92328293758402563-188.138.97.18:_-n_0000000000, state=RUNNABLE, group=Overseer Hdfs SolrCore Failover Thread.]
[junit4] > at java.lang.Thread.sleep(Native Method)
[junit4] > at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:136)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] > 4) Thread[id=2408, name=searcherExecutor-1556-thread-1, state=WAITING, group=TGRP-TestManagedResourceStorage]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
[junit4] > at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] > at __randomizedtesting.SeedInfo.seed([DE98127AF886701B]:0)
[junit4] Completed on J1 in 26.51s, 2 tests, 3 errors <<< FAILURES!
[...truncated 891 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:485: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:465: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:48: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:494: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1342: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:957: There were test failures: 430 suites, 1783 tests, 3 suite-level errors, 1007 ignored (12 assumptions)
Total time: 71 minutes 17 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 32bit/jdk1.7.0_67 -server -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.8.0_20) - Build #
11087 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11087/
Java: 32bit/jdk1.8.0_20 -client -XX:+UseParallelGC
1 tests failed.
REGRESSION: org.apache.lucene.analysis.core.TestRandomChains.testRandomChainsWithLargeStrings
Error Message:
startOffset must be non-negative, and endOffset must be >= startOffset, startOffset=8,endOffset=7
Stack Trace:
java.lang.IllegalArgumentException: startOffset must be non-negative, and endOffset must be >= startOffset, startOffset=8,endOffset=7
at __randomizedtesting.SeedInfo.seed([B2BF235981891C13:D8E49C48D8C73CE0]:0)
at org.apache.lucene.analysis.tokenattributes.PackedTokenAttributeImpl.setOffset(PackedTokenAttributeImpl.java:107)
at org.apache.lucene.analysis.shingle.ShingleFilter.incrementToken(ShingleFilter.java:345)
at org.apache.lucene.analysis.ValidatingTokenFilter.incrementToken(ValidatingTokenFilter.java:68)
at org.apache.lucene.analysis.hu.HungarianLightStemFilter.incrementToken(HungarianLightStemFilter.java:48)
at org.apache.lucene.analysis.ValidatingTokenFilter.incrementToken(ValidatingTokenFilter.java:68)
at org.apache.lucene.analysis.miscellaneous.CapitalizationFilter.incrementToken(CapitalizationFilter.java:97)
at org.apache.lucene.analysis.ValidatingTokenFilter.incrementToken(ValidatingTokenFilter.java:68)
at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkAnalysisConsistency(BaseTokenStreamTestCase.java:704)
at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:615)
at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:513)
at org.apache.lucene.analysis.core.TestRandomChains.testRandomChainsWithLargeStrings(TestRandomChains.java:925)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at java.lang.Thread.run(Thread.java:745)
Build Log:
[...truncated 5740 lines...]
[junit4] Suite: org.apache.lucene.analysis.core.TestRandomChains
[junit4] 2> TEST FAIL: useCharFilter=false text='\u18f0\u18e9\u18c7\u18db\u18be\u18ef\u18d7\u18f6\u18cb\u18d2\u18fd\u18f7\u18f9 \ud802\udc0f\ud802\udc2a\ud802\udc21 \u1ddc\ud80e\udccf\u000e\u000c\u0205\uda57\uddfd\u00b9\ud94f\uddd7\ud99e\udc97\u9078 \u30f4\u30b2\u30aa\u30c6\u30ea\u30b9\u30c6\u30d5\u30c7\u30f9\u30d7\u30a4 \u0b7e\u0b0d\u0b14\u0b11\u0b45'
[junit4] 2> Exception from random analyzer:
[junit4] 2> charfilters=
[junit4] 2> org.apache.lucene.analysis.charfilter.MappingCharFilter(org.apache.lucene.analysis.charfilter.NormalizeCharMap@5d5690, java.io.StringReader@11790e1)
[junit4] 2> tokenizer=
[junit4] 2> org.apache.lucene.analysis.ngram.EdgeNGramTokenizer(org.apache.lucene.util.AttributeFactory$1@e31f645, 9, 16)
[junit4] 2> filters=
[junit4] 2> org.apache.lucene.analysis.miscellaneous.WordDelimiterFilter(ValidatingTokenFilter@14dcec8 term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,flags=0,payload=null, [-25, -19, 12, 120, 57, 2, 9, 109, -115, 100, 58, 28, 103, 111, 56, 86, 21, -58, -64, 73, 110, -4, 29, 62, 14, 28, -118, -75, -100, 82, 6, -118, -8, 77, -56, 121, 85, -79, 19, 105, -83, 80, -128, 20, -46, 54, -65, -105, 14, 112, -10, 91, 89, 35, -125, -110, -14, -115, -43, -23, -68, -69, -4, -52, 120, 47, -7, -113, 43, 49, 55, -88, 21, -6, 109, -75, -109, -73, -34, -117, 36, -28, -125, 81, 29, -119, 96, -98, -64, 104, -123, 43, -109, 76, 91, 60, 45, -107, -98, -119, -80, -3, -60, 37, 25, 86, 67, -99, -88, 97, 118, -90, -127, 21, -90, 108, 107, 15, -73, -86, -1, 13, 101, -43, 105, 114, -85, -16, -60, -58, 30], -23, [az, fzhhr, vecwhjelw, qwzkvtx, n, g])
[junit4] 2> org.apache.lucene.analysis.shingle.ShingleFilter(ValidatingTokenFilter@13f7f55 term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,flags=0,payload=null, 10, 14)
[junit4] 2> org.apache.lucene.analysis.hu.HungarianLightStemFilter(ValidatingTokenFilter@a68481 term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,flags=0,payload=null,keyword=false)
[junit4] 2> org.apache.lucene.analysis.miscellaneous.CapitalizationFilter(ValidatingTokenFilter@ff765c term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,flags=0,payload=null,keyword=false)
[junit4] 2> offsetsAreCorrect=false
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestRandomChains -Dtests.method=testRandomChainsWithLargeStrings -Dtests.seed=B2BF235981891C13 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=en -Dtests.timezone=Atlantic/St_Helena -Dtests.file.encoding=UTF-8
[junit4] ERROR 13.9s J1 | TestRandomChains.testRandomChainsWithLargeStrings <<<
[junit4] > Throwable #1: java.lang.IllegalArgumentException: startOffset must be non-negative, and endOffset must be >= startOffset, startOffset=8,endOffset=7
[junit4] > at __randomizedtesting.SeedInfo.seed([B2BF235981891C13:D8E49C48D8C73CE0]:0)
[junit4] > at org.apache.lucene.analysis.tokenattributes.PackedTokenAttributeImpl.setOffset(PackedTokenAttributeImpl.java:107)
[junit4] > at org.apache.lucene.analysis.shingle.ShingleFilter.incrementToken(ShingleFilter.java:345)
[junit4] > at org.apache.lucene.analysis.ValidatingTokenFilter.incrementToken(ValidatingTokenFilter.java:68)
[junit4] > at org.apache.lucene.analysis.hu.HungarianLightStemFilter.incrementToken(HungarianLightStemFilter.java:48)
[junit4] > at org.apache.lucene.analysis.ValidatingTokenFilter.incrementToken(ValidatingTokenFilter.java:68)
[junit4] > at org.apache.lucene.analysis.miscellaneous.CapitalizationFilter.incrementToken(CapitalizationFilter.java:97)
[junit4] > at org.apache.lucene.analysis.ValidatingTokenFilter.incrementToken(ValidatingTokenFilter.java:68)
[junit4] > at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkAnalysisConsistency(BaseTokenStreamTestCase.java:704)
[junit4] > at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:615)
[junit4] > at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:513)
[junit4] > at org.apache.lucene.analysis.core.TestRandomChains.testRandomChainsWithLargeStrings(TestRandomChains.java:925)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] 2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/build/analysis/common/test/J1/./temp/lucene.analysis.core.TestRandomChains-B2BF235981891C13-001
[junit4] 2> NOTE: test params are: codec=Lucene46, sim=RandomSimilarityProvider(queryNorm=false,coord=no): {}, locale=en, timezone=Atlantic/St_Helena
[junit4] 2> NOTE: Linux 3.13.0-34-generic i386/Oracle Corporation 1.8.0_20 (32-bit)/cpus=8,threads=1,free=137562680,total=234356736
[junit4] 2> NOTE: All tests run in this JVM: [TestCzechStemmer, TestRollingCharBuffer, TestStopFilter, TestKeepCase, TestReverseStringFilter, TestArabicAnalyzer, TestGreekStemmer, TestStrangeOvergeneration, TestCharTokenizers, TestCollationDocValuesField, TestSnowballPorterFilterFactory, TestTypeTokenFilter, TestFilesystemResourceLoader, TestTurkishAnalyzer, TestBulgarianStemFilterFactory, TestFrenchAnalyzer, TestTwoFold, TestCompoundWordTokenFilter, TestFinnishAnalyzer, TestSpanishAnalyzer, TestStemmer, TestPortugueseStemFilter, TestSynonymMapFilter, TestTrimFilterFactory, TestGermanAnalyzer, TestHindiAnalyzer, TestSegmentingTokenizerBase, TestZeroAffix, TestAnalysisSPILoader, TestScandinavianFoldingFilterFactory, TestArabicFilters, TestGalicianStemFilterFactory, TestSwedishLightStemFilter, TestGalicianStemFilter, TestLatvianStemFilterFactory, TestEscaped, TestMorphData, TestClassicAnalyzer, EdgeNGramTokenizerTest, TestPortugueseMinimalStemFilter, TestCJKWidthFilter, TestAnalyzers, TestMappingCharFilterFactory, TestZeroAffix2, TestCollationKeyAnalyzer, TestPersianCharFilter, TestCJKBigramFilterFactory, TestApostropheFilterFactory, TestCommonGramsFilterFactory, QueryAutoStopWordAnalyzerTest, TestCapitalizationFilter, TestGreekStemFilterFactory, DateRecognizerSinkTokenizerTest, TestFullStrip, TokenRangeSinkTokenizerTest, TestCaseSensitive, TestTypeTokenFilterFactory, TestPersianAnalyzer, TestConv, TestFrenchMinimalStemFilterFactory, TestCodepointCountFilter, TestPortugueseLightStemFilterFactory, TestTurkishLowerCaseFilter, TestCharArrayIterator, TestFlagNum, TestEnglishAnalyzer, TestFactories, TestSnowball, TestRandomChains]
[junit4] Completed on J1 in 42.99s, 2 tests, 1 error <<< FAILURES!
[...truncated 394 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:485: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:465: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:48: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/build.xml:544: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:2100: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/analysis/build.xml:106: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/analysis/build.xml:38: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/module-build.xml:60: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1342: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:957: There were test failures: 268 suites, 1360 tests, 1 error, 1 ignored
Total time: 30 minutes 56 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 32bit/jdk1.8.0_20 -client -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.7.0_67) - Build #
11086 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11086/
Java: 64bit/jdk1.7.0_67 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC
1 tests failed.
FAILED: org.apache.solr.cloud.OverseerTest.testOverseerFailure
Error Message:
Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed
Stack Trace:
org.apache.solr.common.SolrException: Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed
at __randomizedtesting.SeedInfo.seed([20505546D23B741:60D8AA77F865860]:0)
at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:150)
at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:163)
at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:125)
at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:155)
at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:314)
at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:221)
at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:157)
at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:662)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1
at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
at org.apache.solr.common.cloud.SolrZkClient$11.execute(SolrZkClient.java:462)
at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:74)
at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:459)
at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:416)
at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:403)
at org.apache.solr.cloud.ShardLeaderElectionContextBase$1.execute(ElectionContext.java:142)
at org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:31)
at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:137)
... 47 more
Build Log:
[...truncated 11454 lines...]
[junit4] Suite: org.apache.solr.cloud.OverseerTest
[junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.OverseerTest-20505546D23B741-001/init-core-data-001
[junit4] 2> 1588232 T4315 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (true)
[junit4] 2> 1588233 T4315 oas.SolrTestCaseJ4.initCore ####initCore
[junit4] 2> 1588233 T4315 oas.SolrTestCaseJ4.initCore ####initCore end
[junit4] 2> 1588236 T4315 oas.SolrTestCaseJ4.setUp ###Starting testShardAssignment
[junit4] 2> 1588237 T4315 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 1588238 T4316 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 1588337 T4315 oasc.ZkTestServer.run start zk server on port:48876
[junit4] 2> 1588338 T4315 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1588341 T4322 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ffcce28 name:ZooKeeperConnection Watcher:127.0.0.1:48876 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1588341 T4315 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1588344 T4315 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1588345 T4324 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7e13c12d name:ZooKeeperConnection Watcher:127.0.0.1:48876 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1588345 T4315 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1588346 T4315 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 1588353 T4315 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1588353 T4326 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@70fe9f9f name:ZooKeeperConnection Watcher:127.0.0.1:48876/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1588354 T4315 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1588354 T4315 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 1588358 T4315 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1588360 T4328 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@102f3a5d name:ZooKeeperConnection Watcher:127.0.0.1:48876/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1588360 T4315 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1588361 T4315 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: retry=false
[junit4] 2> 1588362 T4315 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 1588364 T4315 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 1588366 T4315 oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 1588367 T4315 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:48876_solr
[junit4] 2> 1588368 T4315 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 1588369 T4315 oasc.Overseer.start Overseer (id=92326958919188483-127.0.0.1:48876_solr-n_0000000000) starting
[junit4] 2> 1588371 T4315 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 1588375 T4315 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 1588377 T4315 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 1588379 T4315 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 1588380 T4315 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 1588383 T4315 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 1588389 T4315 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 1588390 T4330 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 1588391 T4315 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 1588392 T4329 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 1588395 T4315 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 1588396 T4315 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1588413 T4315 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1588415 T4333 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@fce9cd4 name:ZooKeeperConnection Watcher:127.0.0.1:48876/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1588415 T4315 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1588417 T4315 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1588418 T4315 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1
[junit4] 2> 1588429 T4336 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1588431 T4329 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 1588431 T4329 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
[junit4] 2> 1588432 T4329 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
[junit4] 2> 1588434 T4334 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1588434 T4335 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1588926 T4315 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4] 2> 1588931 T4315 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
[junit4] 2> 1588934 T4336 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1588937 T4329 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core2",
[junit4] 2> "core_node_name":"node2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 1588937 T4329 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 1588937 T4329 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1588939 T4336 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1588940 T4335 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1588945 T4334 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1589435 T4315 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1589439 T4315 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 1589443 T4336 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1589447 T4329 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core3",
[junit4] 2> "core_node_name":"node3",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 1589448 T4329 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 1589448 T4329 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 1589449 T4336 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1589556 T4335 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1589556 T4334 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1589945 T4315 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 1589950 T4315 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4] 2> 1589953 T4336 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1589956 T4329 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core4",
[junit4] 2> "core_node_name":"node4",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 1589957 T4329 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 1589957 T4329 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
[junit4] 2> 1589958 T4336 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1590060 T4334 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1590060 T4335 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1590455 T4315 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4] 2> 1590460 T4336 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1590462 T4329 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core5",
[junit4] 2> "core_node_name":"node5",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 1590462 T4329 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 1590462 T4329 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1590565 T4335 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1590565 T4334 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1590961 T4315 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1590966 T4336 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1590968 T4329 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core6",
[junit4] 2> "core_node_name":"node6",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 1590968 T4329 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 1590968 T4329 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 1591071 T4335 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1591071 T4334 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1591466 T4315 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 1591476 T4333 oaz.ClientCnxn$EventThread.processEvent ERROR Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@6a5204ca rejected from java.util.concurrent.ThreadPoolExecutor@2eb8a04e[Shutting down, pool size = 2, active threads = 2, queued tasks = 0, completed tasks = 8]
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
[junit4] 2> at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:110)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$3.process(SolrZkClient.java:206)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
[junit4] 2>
[junit4] 2> 1591477 T4337 oasc.LeaderElector$ElectionWatcher.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard3/election
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:290)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:287)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:74)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:287)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:358)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$3$1.run(SolrZkClient.java:209)
[junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2>
[junit4] 2> 1592976 T4335 oasc.LeaderElector$ElectionWatcher.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:290)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:287)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:74)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:287)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:358)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$3$1.run(SolrZkClient.java:209)
[junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2>
[junit4] 2> 1592979 T4315 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:48876 48876
[junit4] 2> 1593675 T4316 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:48876 48876
[junit4] 2> 1593676 T4315 oas.SolrTestCaseJ4.tearDown ###Ending testShardAssignment
[junit4] 2> 1593677 T4315 oasc.Overseer.close Overseer (id=92326958919188483-127.0.0.1:48876_solr-n_0000000000) closing
[junit4] 2> 1593678 T4329 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:48876_solr
[junit4] 2> 1593679 T4339 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:307)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:304)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:74)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:304)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:321)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:89)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:293)
[junit4] 2>
[junit4] 2> 1593683 T4315 oas.SolrTestCaseJ4.setUp ###Starting testReplay
[junit4] 2> 1593684 T4315 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 1593685 T4340 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 1593784 T4315 oasc.ZkTestServer.run start zk server on port:53420
[junit4] 2> 1593785 T4315 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1593788 T4346 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@55cdaed8 name:ZooKeeperConnection Watcher:127.0.0.1:53420/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1593788 T4315 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1593790 T4315 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1593791 T4348 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@402c21ef name:ZooKeeperConnection Watcher:127.0.0.1:53420 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1593791 T4315 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1593793 T4315 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1593795 T4350 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@113c54d4 name:ZooKeeperConnection Watcher:127.0.0.1:53420 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1593795 T4315 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1593795 T4315 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 1593798 T4315 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 1593800 T4315 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 1593802 T4315 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 1593804 T4315 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1593806 T4315 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 1593809 T4315 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1593811 T4352 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@248db0b3 name:ZooKeeperConnection Watcher:127.0.0.1:53420/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1593811 T4315 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1593812 T4315 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: retry=false
[junit4] 2> 1593815 T4315 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 1593818 T4315 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 1593820 T4315 oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 1593821 T4315 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:53420_solr
[junit4] 2> 1593821 T4315 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 1593823 T4315 oasc.Overseer.start Overseer (id=92326959276097539-127.0.0.1:53420_solr-n_0000000000) starting
[junit4] 2> 1593825 T4315 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 1593829 T4315 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 1593832 T4315 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 1593835 T4315 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 1593837 T4315 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 1593851 T4315 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 1593852 T4354 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 1593856 T4353 oasc.Overseer$ClusterStateUpdater.run Replaying operations from work queue.
[junit4] 2> 1593857 T4353 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1/solr",
[junit4] 2> "node_name":"node1",
[junit4] 2> "shard":"s1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "core":"core1",
[junit4] 2> "roles":"",
[junit4] 2> "state":"recovering"}
[junit4] 2> 1593858 T4356 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 1593859 T4353 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1/solr",
[junit4] 2> "node_name":"node1",
[junit4] 2> "shard":"s1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "core":"core2",
[junit4] 2> "roles":"",
[junit4] 2> "state":"recovering"}
[junit4] 2> 1593860 T4356 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 1593861 T4353 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 1593862 T4353 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1/solr",
[junit4] 2> "node_name":"node1",
[junit4] 2> "shard":"s1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "core":"core3",
[junit4] 2> "roles":"",
[junit4] 2> "state":"recovering"}
[junit4] 2> 1593863 T4357 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1593864 T4356 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 1593957 T4315 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:53420 53420
[junit4] 2> 1594506 T4340 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:53420 53420
[junit4] 2> 1594507 T4315 oas.SolrTestCaseJ4.tearDown ###Ending testReplay
[junit4] 2> 1594508 T4315 oasc.Overseer.close Overseer (id=92326959276097539-127.0.0.1:53420_solr-n_0000000000) closing
[junit4] 2> 1594508 T4353 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:53420_solr
[junit4] 2> 1594515 T4358 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:307)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:304)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:74)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:304)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:321)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:89)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:293)
[junit4] 2>
[junit4] 2> 1594515 T4315 oas.SolrTestCaseJ4.setUp ###Starting testBadQueueItem
[junit4] 2> 1594516 T4315 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 1594517 T4359 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 1594617 T4315 oasc.ZkTestServer.run start zk server on port:52292
[junit4] 2> 1594618 T4315 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1594621 T4365 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3d4603e4 name:ZooKeeperConnection Watcher:127.0.0.1:52292 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1594621 T4315 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1594626 T4315 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1594627 T4367 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4f94df2 name:ZooKeeperConnection Watcher:127.0.0.1:52292 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1594627 T4315 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1594628 T4315 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 1594631 T4315 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1594632 T4369 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@55e25fd7 name:ZooKeeperConnection Watcher:127.0.0.1:52292/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1594633 T4315 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1594633 T4315 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 1594637 T4315 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1594637 T4371 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5b9a6c6a name:ZooKeeperConnection Watcher:127.0.0.1:52292/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1594638 T4315 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1594638 T4315 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: retry=false
[junit4] 2> 1594641 T4315 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 1594643 T4315 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 1594645 T4315 oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 1594646 T4315 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:52292_solr
[junit4] 2> 1594646 T4315 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 1594647 T4315 oasc.Overseer.start Overseer (id=92326959330623491-127.0.0.1:52292_solr-n_0000000000) starting
[junit4] 2> 1594650 T4315 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 1594654 T4315 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 1594657 T4315 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 1594666 T4315 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 1594669 T4315 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 1594672 T4315 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 1594682 T4315 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 1594683 T4373 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 1594683 T4315 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 1594684 T4372 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 1594685 T4315 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 1594686 T4315 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1594688 T4315 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1594690 T4376 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7d425f56 name:ZooKeeperConnection Watcher:127.0.0.1:52292/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1594690 T4315 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1594691 T4315 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1594693 T4315 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1
[junit4] 2> 1594699 T4379 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1594701 T4372 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 1594702 T4372 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
[junit4] 2> 1594702 T4372 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
[junit4] 2> 1594704 T4377 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1594704 T4378 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1595199 T4315 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4] 2> 1595205 T4315 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
[junit4] 2> 1595209 T4379 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1595213 T4372 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core2",
[junit4] 2> "core_node_name":"node2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 1595213 T4372 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 1595213 T4372 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1595215 T4379 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1595216 T4378 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1595217 T4377 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1595710 T4315 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1595715 T4315 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 1595718 T4379 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1595722 T4372 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core3",
[junit4] 2> "core_node_name":"node3",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 1595723 T4372 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 1595723 T4372 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 1595724 T4379 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1595826 T4377 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1595826 T4378 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1596224 T4315 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 1596231 T4315 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4] 2> 1596233 T4379 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1596337 T4378 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1596337 T4377 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1596385 T4379 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1596386 T4372 oasc.Overseer$ClusterStateUpdater.checkKeyExistence ERROR Skipping invalid Overseer message because it has no collection specified: {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core0",
[junit4] 2> "core_node_name":"node0",
[junit4] 2> "collection":"",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 1596388 T4372 oasc.Overseer$ClusterStateUpdater.checkKeyExistence ERROR Skipping invalid Overseer message because it has no collection specified: {
[junit4] 2> "operation":"deletecore",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core0",
[junit4] 2> "core_node_name":"node0",
[junit4] 2> "collection":""}
[junit4] 2> 1596389 T4379 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1596390 T4372 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection2",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 1596390 T4372 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection2 with shards [shard1, shard2, shard3]
[junit4] 2> 1596390 T4372 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
[junit4] 2> 1596492 T4377 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1596492 T4378 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1596886 T4315 oascc.SolrZkClient.makePath makePath: /collections/collection2/leader_elect/shard3/election
[junit4] 2> 1596891 T4315 oascc.SolrZkClient.makePath makePath: /collections/collection2/leaders/shard3
[junit4] 2> 1596894 T4379 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1596898 T4372 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core2",
[junit4] 2> "core_node_name":"node2",
[junit4] 2> "collection":"collection2",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 1596898 T4372 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 1596898 T4372 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1596899 T4379 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1596900 T4378 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1596900 T4377 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1597396 T4315 oascc.SolrZkClient.makePath makePath: /collections/collection2/leader_elect/shard1/election
[junit4] 2> 1597400 T4315 oascc.SolrZkClient.makePath makePath: /collections/collection2/leaders/shard1
[junit4] 2> 1597403 T4379 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1597406 T4372 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core3",
[junit4] 2> "core_node_name":"node3",
[junit4] 2> "collection":"collection2",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 1597406 T4372 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 1597406 T4372 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 1597408 T4379 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1597408 T4377 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1597408 T4378 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1597904 T4315 oascc.SolrZkClient.makePath makePath: /collections/collection2/leader_elect/shard2/election
[junit4] 2> 1597910 T4315 oascc.SolrZkClient.makePath makePath: /collections/collection2/leaders/shard2
[junit4] 2> 1597912 T4379 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1598016 T4378 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1598016 T4377 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1598067 T4315 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:52292 52292
[junit4] 2> 1598639 T4359 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:52292 52292
[junit4] 2> 1598641 T4315 oas.SolrTestCaseJ4.tearDown ###Ending testBadQueueItem
[junit4] 2> 1598641 T4315 oasc.Overseer.close Overseer (id=92326959330623491-127.0.0.1:52292_solr-n_0000000000) closing
[junit4] 2> 1598642 T4372 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:52292_solr
[junit4] 2> 1598645 T4380 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:307)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:304)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:74)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:304)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:321)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:89)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:293)
[junit4] 2>
[junit4] 2> 1598650 T4315 oas.SolrTestCaseJ4.setUp ###Starting testShardLeaderChange
[junit4] 2> 1598651 T4315 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 1598652 T4381 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 1598752 T4315 oasc.ZkTestServer.run start zk server on port:41539
[junit4] 2> 1598753 T4315 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1598755 T4387 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2d92ced name:ZooKeeperConnection Watcher:127.0.0.1:41539/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1598755 T4315 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1598756 T4315 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1598757 T4389 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3583c567 name:ZooKeeperConnection Watcher:127.0.0.1:41539 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1598757 T4315 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1598758 T4315 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1598760 T4391 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@51d9c781 name:ZooKeeperConnection Watcher:127.0.0.1:41539 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1598760 T4315 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1598760 T4315 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 1598761 T4315 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 1598764 T4315 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1598764 T4392 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1598765 T4395 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@47113641 name:ZooKeeperConnection Watcher:127.0.0.1:41539/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1598765 T4392 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1598765 T4396 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5d3cddf9 name:ZooKeeperConnection Watcher:127.0.0.1:41539/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1598765 T4392 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: retry=false
[junit4] 2> 1598765 T4315 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1598766 T4315 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 1598766 T4392 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 1598767 T4315 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 1598768 T4392 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 1598768 T4315 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1598769 T4392 oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 1598770 T4315 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
[junit4] 2> 1598770 T4392 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:41539_solr
[junit4] 2> 1598770 T4392 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 1598771 T4392 oasc.Overseer.start Overseer (id=92326959601680387-127.0.0.1:41539_solr-n_0000000000) starting
[junit4] 2> 1598772 T4315 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 1598772 T4392 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 1598774 T4392 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 1598775 T4392 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 1598777 T4392 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 1598778 T4392 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 1598780 T4392 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 1598783 T4392 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 1598783 T4399 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 1598783 T4392 oasc.OverseerTest$OverseerRestarter.run Killing overseer.
[junit4] 2> 1598784 T4398 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 1598784 T4398 oasc.Overseer$ClusterStateUpdater.amILeader org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:307)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:304)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:74)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:304)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.amILeader(Overseer.java:606)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:211)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2>
[junit4] 2> 1598785 T4398 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=92326959601680387-127.0.0.1:41539_solr-n_0000000000) am no longer a leader.
[junit4] 2> 1598785 T4392 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1598785 T4398 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:41539_solr
[junit4] 2> 1598786 T4403 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:307)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:304)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:74)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:304)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:321)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:89)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:293)
[junit4] 2>
[junit4] 2> 1598789 T4402 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7b534180 name:ZooKeeperConnection Watcher:127.0.0.1:41539/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1598789 T4392 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1598790 T4392 oasc.Overseer.close Overseer (id=92326959601680387-127.0.0.1:41539_solr-n_0000000000) closing
[junit4] 2> 1598790 T4392 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: retry=false
[junit4] 2> 1598791 T4399 oasc.OverseerCollectionProcessor.run ERROR Unable to prioritize overseer org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /roles.json
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:256)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:253)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:74)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:253)
[junit4] 2> at org.apache.solr.cloud.OverseerCollectionProcessor.prioritizeOverseerNodes(OverseerCollectionProcessor.java:411)
[junit4] 2> at org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:258)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2>
[junit4] 2> 1598793 T4392 oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 1598794 T4392 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:41539_solr
[junit4] 2> 1598794 T4392 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 1598796 T4392 oasc.Overseer.start Overseer (id=92326959601680389-127.0.0.1:41539_solr-n_0000000001) starting
[junit4] 2> 1598801 T4392 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 1598801 T4405 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 1598802 T4404 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 1598803 T4404 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"state1",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 1598803 T4404 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1]
[junit4] 2> 1598803 T4404 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1598805 T4407 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1598805 T4397 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1599274 T4315 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1599278 T4315 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 1599280 T4407 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1599282 T4315 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1599283 T4404 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"state2",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 1599283 T4404 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
[junit4] 2> 1599284 T4407 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1599284 T4315 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1599285 T4397 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1599285 T4409 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@149ee47a name:ZooKeeperConnection Watcher:127.0.0.1:41539/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1599285 T4315 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1599286 T4315 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1599287 T4315 oascc.SolrZkClient.makePath makePath: /live_nodes/node2
[junit4] 2> 1599291 T4407 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1599291 T4315 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1599293 T4404 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"state1",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 1599293 T4404 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
[junit4] 2> 1599297 T4407 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1599298 T4404 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"state2",
[junit4] 2> "node_name":"node2",
[junit4] 2> "core":"core4",
[junit4] 2> "core_node_name":"node2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node2/solr/"}
[junit4] 2> 1599299 T4404 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4] 2> 1599299 T4404 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1599403 T4410 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1599403 T4397 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1599798 T4315 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1599809 T4411 oasc.LeaderElector$ElectionWatcher.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:290)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:287)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:74)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:287)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:358)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$3$1.run(SolrZkClient.java:209)
[junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2>
[junit4] 2> 1599810 T4410 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 1599815 T4407 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1599921 T4410 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1601309 T4397 oasc.LeaderElector$ElectionWatcher.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:290)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:287)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:74)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:287)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:358)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$3$1.run(SolrZkClient.java:209)
[junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2>
[junit4] 2> 1601311 T4315 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1601312 T4413 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3ecde5d name:ZooKeeperConnection Watcher:127.0.0.1:41539/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1601312 T4315 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1601313 T4315 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1601315 T4315 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
[junit4] 2> 1601318 T4407 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1601318 T4315 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1601319 T4404 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"state1",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 1601319 T4404 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
[junit4] 2> 1601322 T4414 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 1601322 T4407 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1601322 T4315 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1601323 T4404 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"state2",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 1601323 T4404 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
[junit4] 2> 1601325 T4407 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1601326 T4315 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1601327 T4416 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3a346c72 name:ZooKeeperConnection Watcher:127.0.0.1:41539/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1601327 T4315 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1601328 T4315 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1601329 T4315 oascc.SolrZkClient.makePath makePath: /live_nodes/node2
[junit4] 2> 1601330 T4407 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1601331 T4315 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1601331 T4404 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"state1",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 1601331 T4404 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
[junit4] 2> 1601404 T4392 oasc.OverseerTest$OverseerRestarter.run Killing overseer.
[junit4] 2> 1601406 T4392 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1601408 T4419 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@474011af name:ZooKeeperConnection Watcher:127.0.0.1:41539/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1601408 T4392 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1601408 T4392 oasc.Overseer.close Overseer (id=92326959601680389-127.0.0.1:41539_solr-n_0000000001) closing
[junit4] 2> 1601408 T4404 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:41539_solr
[junit4] 2> 1601408 T4392 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: retry=false
[junit4] 2> 1601410 T4420 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:307)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:304)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:74)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:304)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:321)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:89)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:293)
[junit4] 2>
[junit4] 2> 1601411 T4392 oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 1601413 T4392 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:41539_solr
[junit4] 2> 1601413 T4392 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 1601414 T4392 oasc.Overseer.start Overseer (id=92326959601680393-127.0.0.1:41539_solr-n_0000000002) starting
[junit4] 2> 1601419 T4392 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 1601419 T4422 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 1601421 T4421 oasc.Overseer$ClusterStateUpdater.run Replaying operations from work queue.
[junit4] 2> 1601421 T4421 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"state1",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 1601421 T4421 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
[junit4] 2> 1601422 T4417 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1601422 T4414 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1601423 T4421 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"state2",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 1601423 T4421 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
[junit4] 2> 1601423 T4414 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1601423 T4417 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1601425 T4417 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1601425 T4414 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1601426 T4421 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"state1",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 1601426 T4421 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
[junit4] 2> 1601427 T4417 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1601427 T4414 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1601427 T4421 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 1601537 T4315 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1601537 T4424 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1601539 T4421 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"state2",
[junit4] 2> "node_name":"node2",
[junit4] 2> "core":"core4",
[junit4] 2> "core_node_name":"node2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node2/solr/"}
[junit4] 2> 1601539 T4421 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
[junit4] 2> 1601542 T4413 oaz.ClientCnxn$EventThread.processEvent ERROR Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3aebb20f rejected from java.util.concurrent.ThreadPoolExecutor@47f71501[Shutting down, pool size = 2, active threads = 2, queued tasks = 0, completed tasks = 8]
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
[junit4] 2> at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:110)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$3.process(SolrZkClient.java:206)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
[junit4] 2>
[junit4] 2> 1601542 T4425 oasc.LeaderElector$ElectionWatcher.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:290)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:287)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:74)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:287)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:358)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$3$1.run(SolrZkClient.java:209)
[junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] 2> at
[...truncated too long message...]
ForConnected Client is connected to ZooKeeper
[junit4] 2> 1664053 T4315 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: retry=false
[junit4] 2> 1664056 T4315 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 1664059 T4315 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 1664062 T4315 oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 1664064 T4315 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:46213_solr
[junit4] 2> 1664065 T4315 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 1664068 T4315 oasc.Overseer.start Overseer (id=92326963879215107-127.0.0.1:46213_solr-n_0000000000) starting
[junit4] 2> 1664070 T4315 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 1664074 T4315 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 1664078 T4315 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 1664082 T4315 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 1664086 T4315 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 1664089 T4315 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 1664095 T4315 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 1664096 T5087 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 1664098 T5086 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 1664101 T5086 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1/solr",
[junit4] 2> "node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "core":"core1",
[junit4] 2> "roles":"",
[junit4] 2> "state":"recovering"}
[junit4] 2> 1664101 T5086 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1664105 T5089 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1664105 T5090 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 1664153 T5089 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1664156 T5086 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1/solr",
[junit4] 2> "node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "core":"core1",
[junit4] 2> "roles":"",
[junit4] 2> "state":"active"}
[junit4] 2> 1664156 T5086 oasc.Overseer$ClusterStateUpdater.updateState node=core_node1 is already registered
[junit4] 2> 1664156 T5086 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
[junit4] 2> 1664261 T5090 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 1664307 T4315 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:46213 46213
[junit4] 2> 1664891 T5073 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:46213 46213
[junit4] 2> 1664893 T4315 oas.SolrTestCaseJ4.tearDown ###Ending testStateChange
[junit4] 2> 1664894 T4315 oasc.Overseer.close Overseer (id=92326963879215107-127.0.0.1:46213_solr-n_0000000000) closing
[junit4] 2> 1664894 T5086 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:46213_solr
[junit4] 2> 1664895 T5091 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:307)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:304)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:74)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:304)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:321)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:89)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:293)
[junit4] 2>
[junit4] 2> 1667897 T4315 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4] 2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.OverseerTest-20505546D23B741-001
[junit4] 2> NOTE: test params are: codec=CheapBastard, sim=DefaultSimilarity, locale=fr_CH, timezone=America/New_York
[junit4] 2> NOTE: Linux 3.13.0-34-generic amd64/Oracle Corporation 1.7.0_67 (64-bit)/cpus=8,threads=1,free=97132456,total=330399744
[junit4] 2> NOTE: All tests run in this JVM: [TriLevelCompositeIdRoutingTest, TestDistributedMissingSort, DirectSolrSpellCheckerTest, TestRTGBase, ZkControllerTest, TestRangeQuery, TestSolrDeletionPolicy2, TestCodecSupport, TestFieldTypeResource, DistributedExpandComponentTest, SharedFSAutoReplicaFailoverUtilsTest, TestShortCircuitedRequests, ZkNodePropsTest, NoCacheHeaderTest, TestAnalyzedSuggestions, HighlighterTest, AddBlockUpdateTest, EnumFieldTest, TestSolr4Spatial, JSONWriterTest, FieldFacetExtrasTest, TestTrie, EchoParamsTest, MinimalSchemaTest, TermsComponentTest, OpenExchangeRatesOrgProviderTest, TestArbitraryIndexDir, TestStressVersions, ModifyConfFileTest, FileUtilsTest, DeleteShardTest, TestFastWriter, SpellingQueryConverterTest, DistribCursorPagingTest, MBeansHandlerTest, PathHierarchyTokenizerFactoryTest, SpellPossibilityIteratorTest, HighlighterConfigTest, XmlUpdateRequestHandlerTest, ZkCLITest, ShardRoutingTest, TestSurroundQueryParser, OverseerCollectionProcessorTest, SolrIndexSplitterTest, TestRandomDVFaceting, ResponseHeaderTest, DocValuesMultiTest, PeerSyncTest, SpellCheckCollatorTest, TestFastLRUCache, TestExpandComponent, ExternalFileFieldSortTest, CoreAdminHandlerTest, XsltUpdateRequestHandlerTest, TestBlendedInfixSuggestions, TestLRUCache, DocValuesMissingTest, DateRangeFieldTest, TestJmxIntegration, HdfsDirectoryTest, ResponseLogComponentTest, ChaosMonkeyNothingIsSafeTest, TestLuceneMatchVersion, AsyncMigrateRouteKeyTest, TestSystemIdResolver, SharedFSAutoReplicaFailoverTest, TestHashPartitioner, AtomicUpdatesTest, QueryFacetTest, TestModifyConfFiles, TestDocumentBuilder, TestSolrCoreProperties, PingRequestHandlerTest, TestRestManager, RecoveryZkTest, TestCloudPivotFacet, StatelessScriptUpdateProcessorFactoryTest, InfoHandlerTest, TestSearchPerf, SolrCmdDistributorTest, HdfsLockFactoryTest, TestFoldingMultitermQuery, NotRequiredUniqueKeyTest, TestManagedSchemaFieldResource, SignatureUpdateProcessorFactoryTest, TestMiniSolrCloudCluster, RegexBoostProcessorTest, ShowFileRequestHandlerTest, StandardRequestHandlerTest, TestRemoteStreaming, TestCursorMarkWithoutUniqueKey, HdfsRecoveryZkTest, TestWordDelimiterFilterFactory, TestMaxScoreQueryParser, TestShardHandlerFactory, TestSolr4Spatial2, TestSolrIndexConfig, SSLMigrationTest, TestSolrDeletionPolicy1, DistributedQueryComponentOptimizationTest, TestChildDocTransformer, IndexBasedSpellCheckerTest, SuggesterTSTTest, BasicDistributedZkTest, SyncSliceTest, OverseerTest]
[junit4] Completed on J0 in 79.69s, 9 tests, 1 error <<< FAILURES!
[...truncated 647 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:485: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:465: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:48: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:494: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1342: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:957: There were test failures: 430 suites, 1783 tests, 1 error, 40 ignored (19 assumptions)
Total time: 85 minutes 3 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.7.0_67 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any