You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2015/11/06 14:50:26 UTC
[JENKINS] Lucene-Solr-trunk-Windows (64bit/jdk1.8.0_66) - Build #
5377 - Still Failing!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Windows/5377/
Java: 64bit/jdk1.8.0_66 -XX:-UseCompressedOops -XX:+UseSerialGC
1 tests failed.
FAILED: org.apache.solr.cloud.OverseerTest.testOverseerStatsReset
Error Message:
expected:<1> but was:<2>
Stack Trace:
java.lang.AssertionError: expected:<1> but was:<2>
at __randomizedtesting.SeedInfo.seed([7771E2C402B621D6:DC2505F8976C82D8]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at org.apache.solr.cloud.OverseerTest.testOverseerStatsReset(OverseerTest.java:722)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1660)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:866)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:902)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:916)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:875)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:777)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:811)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:822)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at java.lang.Thread.run(Thread.java:745)
Build Log:
[...truncated 9973 lines...]
[junit4] Suite: org.apache.solr.cloud.OverseerTest
[junit4] 2> Creating dataDir: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.OverseerTest_7771E2C402B621D6-001\init-core-data-001
[junit4] 2> 1150373 INFO (SUITE-OverseerTest-seed#[7771E2C402B621D6]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false)
[junit4] 2> 1150373 INFO (SUITE-OverseerTest-seed#[7771E2C402B621D6]-worker) [ ] o.a.s.SolrTestCaseJ4 ####initCore
[junit4] 2> 1150373 INFO (SUITE-OverseerTest-seed#[7771E2C402B621D6]-worker) [ ] o.a.s.SolrTestCaseJ4 ####initCore end
[junit4] 2> 1150374 INFO (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[7771E2C402B621D6]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testExternalClusterStateChangeBehavior
[junit4] 2> 1150377 INFO (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[7771E2C402B621D6]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 1150377 INFO (Thread-3103) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 1150377 INFO (Thread-3103) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 1150478 INFO (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[7771E2C402B621D6]) [ ] o.a.s.c.ZkTestServer start zk server on port:59118
[junit4] 2> 1150478 INFO (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 1150479 INFO (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 1150482 INFO (zkCallback-909-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@6d6a099b name:ZooKeeperConnection Watcher:127.0.0.1:59118/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1150482 INFO (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 1150482 INFO (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 1150482 INFO (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 1150483 INFO (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 1150484 INFO (zkCallback-910-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@2e0c09b3 name:ZooKeeperConnection Watcher:127.0.0.1:59118 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1150484 INFO (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 1150485 INFO (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 1150487 INFO (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 1150487 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x150dcf28af40001, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 1150488 INFO (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 1150489 INFO (zkCallback-911-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@736f423 name:ZooKeeperConnection Watcher:127.0.0.1:59118 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1150489 INFO (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 1150489 INFO (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 1150490 INFO (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient makePath: /solr
[junit4] 2> 1150492 INFO (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
[junit4] 2> 1150493 INFO (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections
[junit4] 2> 1150494 INFO (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
[junit4] 2> 1150495 INFO (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
[junit4] 2> 1150495 INFO (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient makePath: /security.json
[junit4] 2> 1150496 INFO (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 1150498 INFO (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 1150499 INFO (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 1150500 INFO (zkCallback-912-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@5338480e name:ZooKeeperConnection Watcher:127.0.0.1:59118/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1150500 INFO (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 1150500 INFO (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 1150500 INFO (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[7771E2C402B621D6]) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=600000&connTimeout=60000&retry=true
[junit4] 2> 1150504 INFO (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
[junit4] 2> 1150505 INFO (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
[junit4] 2> 1150506 INFO (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[7771E2C402B621D6]) [ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 1150506 INFO (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[7771E2C402B621D6]) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/94818526446223363-127.0.0.1:59118_solr-n_0000000000
[junit4] 2> 1150506 INFO (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[7771E2C402B621D6]) [ ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:59118_solr
[junit4] 2> 1150507 INFO (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
[junit4] 2> 1150507 INFO (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[7771E2C402B621D6]) [ ] o.a.s.c.Overseer Overseer (id=94818526446223363-127.0.0.1:59118_solr-n_0000000000) starting
[junit4] 2> 1150508 INFO (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
[junit4] 2> 1150509 INFO (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
[junit4] 2> 1150511 INFO (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
[junit4] 2> 1150512 INFO (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
[junit4] 2> 1150526 INFO (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
[junit4] 2> 1150530 INFO (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
[junit4] 2> 1150532 INFO (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[7771E2C402B621D6]) [ ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 1150533 INFO (OverseerCollectionConfigSetProcessor-94818526446223363-127.0.0.1:59118_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
[junit4] 2> 1150534 INFO (OverseerStateUpdate-94818526446223363-127.0.0.1:59118_solr-n_0000000000) [ ] o.a.s.c.Overseer Starting to work on the main queue
[junit4] 2> 1150535 INFO (OverseerStateUpdate-94818526446223363-127.0.0.1:59118_solr-n_0000000000) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 1150538 INFO (OverseerStateUpdate-94818526446223363-127.0.0.1:59118_solr-n_0000000000) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1/solr",
[junit4] 2> "node_name":"node1",
[junit4] 2> "collection":"c1",
[junit4] 2> "core":"core1",
[junit4] 2> "roles":"",
[junit4] 2> "state":"down"} current state version: 0
[junit4] 2> 1150539 INFO (OverseerStateUpdate-94818526446223363-127.0.0.1:59118_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=null message={
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1/solr",
[junit4] 2> "node_name":"node1",
[junit4] 2> "collection":"c1",
[junit4] 2> "core":"core1",
[junit4] 2> "roles":"",
[junit4] 2> "state":"down"}
[junit4] 2> 1150539 INFO (OverseerStateUpdate-94818526446223363-127.0.0.1:59118_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
[junit4] 2> 1150540 INFO (zkCallback-909-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 1150540 INFO (zkCallback-912-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 1150591 INFO (OverseerStateUpdate-94818526446223363-127.0.0.1:59118_solr-n_0000000000) [ ] o.a.s.c.Overseer processMessage: queueSize: 2, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1/solr",
[junit4] 2> "node_name":"node1",
[junit4] 2> "collection":"c1",
[junit4] 2> "core":"core1",
[junit4] 2> "roles":"",
[junit4] 2> "state":"recovering"} current state version: 1
[junit4] 2> 1150591 INFO (zkCallback-909-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 1150591 INFO (zkCallback-912-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 1150591 INFO (OverseerStateUpdate-94818526446223363-127.0.0.1:59118_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=null message={
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1/solr",
[junit4] 2> "node_name":"node1",
[junit4] 2> "collection":"c1",
[junit4] 2> "core":"core1",
[junit4] 2> "roles":"",
[junit4] 2> "state":"recovering"}
[junit4] 2> 1150591 INFO (OverseerStateUpdate-94818526446223363-127.0.0.1:59118_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator node=core_node1 is already registered
[junit4] 2> 1150591 INFO (OverseerStateUpdate-94818526446223363-127.0.0.1:59118_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator shard=shard1 is already registered
[junit4] 2> 1150593 INFO (OverseerStateUpdate-94818526446223363-127.0.0.1:59118_solr-n_0000000000) [ ] o.a.s.c.Overseer processMessage: queueSize: 2, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1/solr",
[junit4] 2> "node_name":"node1",
[junit4] 2> "collection":"c1",
[junit4] 2> "core":"core1",
[junit4] 2> "roles":"",
[junit4] 2> "state":"active"} current state version: 1
[junit4] 2> 1150593 INFO (OverseerStateUpdate-94818526446223363-127.0.0.1:59118_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=null message={
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1/solr",
[junit4] 2> "node_name":"node1",
[junit4] 2> "collection":"c1",
[junit4] 2> "core":"core1",
[junit4] 2> "roles":"",
[junit4] 2> "state":"active"}
[junit4] 2> 1150594 INFO (OverseerStateUpdate-94818526446223363-127.0.0.1:59118_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator node=core_node1 is already registered
[junit4] 2> 1150594 INFO (OverseerStateUpdate-94818526446223363-127.0.0.1:59118_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator shard=shard1 is already registered
[junit4] 2> 1150596 INFO (OverseerStateUpdate-94818526446223363-127.0.0.1:59118_solr-n_0000000000) [ ] o.a.s.c.Overseer processMessage: queueSize: 3, message = {
[junit4] 2> "operation":"create",
[junit4] 2> "name":"test",
[junit4] 2> "numShards":"1",
[junit4] 2> "replicationFactor":"1",
[junit4] 2> "stateFormat":"2"} current state version: 1
[junit4] 2> 1150596 INFO (OverseerStateUpdate-94818526446223363-127.0.0.1:59118_solr-n_0000000000) [ ] o.a.s.c.o.ClusterStateMutator building a new cName: test
[junit4] 2> 1150596 WARN (OverseerStateUpdate-94818526446223363-127.0.0.1:59118_solr-n_0000000000) [ ] o.a.s.c.Overseer Bad version writing to ZK using compare-and-set, will force refresh cluster state
[junit4] 2> org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode = BadVersion for /clusterstate.json
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:115)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1270)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:370)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:367)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.setData(SolrZkClient.java:367)
[junit4] 2> at org.apache.solr.cloud.overseer.ZkStateWriter.writePendingUpdates(ZkStateWriter.java:251)
[junit4] 2> at org.apache.solr.cloud.overseer.ZkStateWriter.enqueueUpdate(ZkStateWriter.java:118)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.processQueueItem(Overseer.java:290)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:214)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 1150599 INFO (OverseerStateUpdate-94818526446223363-127.0.0.1:59118_solr-n_0000000000) [ ] o.a.s.c.Overseer processMessage: workQueueSize: 2, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1/solr",
[junit4] 2> "node_name":"node1",
[junit4] 2> "collection":"c1",
[junit4] 2> "core":"core1",
[junit4] 2> "roles":"",
[junit4] 2> "state":"recovering"}
[junit4] 2> 1150599 INFO (OverseerStateUpdate-94818526446223363-127.0.0.1:59118_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=null message={
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1/solr",
[junit4] 2> "node_name":"node1",
[junit4] 2> "collection":"c1",
[junit4] 2> "core":"core1",
[junit4] 2> "roles":"",
[junit4] 2> "state":"recovering"}
[junit4] 2> 1150599 INFO (OverseerStateUpdate-94818526446223363-127.0.0.1:59118_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator node=core_node1 is already registered
[junit4] 2> 1150599 INFO (OverseerStateUpdate-94818526446223363-127.0.0.1:59118_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator shard=shard1 is already registered
[junit4] 2> 1150600 INFO (zkCallback-912-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 1150600 INFO (zkCallback-909-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 1150601 INFO (OverseerStateUpdate-94818526446223363-127.0.0.1:59118_solr-n_0000000000) [ ] o.a.s.c.Overseer processMessage: workQueueSize: 2, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1/solr",
[junit4] 2> "node_name":"node1",
[junit4] 2> "collection":"c1",
[junit4] 2> "core":"core1",
[junit4] 2> "roles":"",
[junit4] 2> "state":"active"}
[junit4] 2> 1150602 INFO (OverseerStateUpdate-94818526446223363-127.0.0.1:59118_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=null message={
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1/solr",
[junit4] 2> "node_name":"node1",
[junit4] 2> "collection":"c1",
[junit4] 2> "core":"core1",
[junit4] 2> "roles":"",
[junit4] 2> "state":"active"}
[junit4] 2> 1150602 INFO (OverseerStateUpdate-94818526446223363-127.0.0.1:59118_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator node=core_node1 is already registered
[junit4] 2> 1150602 INFO (OverseerStateUpdate-94818526446223363-127.0.0.1:59118_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator shard=shard1 is already registered
[junit4] 2> 1150602 INFO (zkCallback-912-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 1150602 INFO (zkCallback-909-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 1150604 INFO (OverseerStateUpdate-94818526446223363-127.0.0.1:59118_solr-n_0000000000) [ ] o.a.s.c.Overseer processMessage: queueSize: 0, message = {
[junit4] 2> "operation":"create",
[junit4] 2> "name":"test",
[junit4] 2> "numShards":"1",
[junit4] 2> "replicationFactor":"1",
[junit4] 2> "stateFormat":"2"} current state version: 4
[junit4] 2> 1150604 INFO (OverseerStateUpdate-94818526446223363-127.0.0.1:59118_solr-n_0000000000) [ ] o.a.s.c.o.ClusterStateMutator building a new cName: test
[junit4] 2> 1150606 INFO (OverseerStateUpdate-94818526446223363-127.0.0.1:59118_solr-n_0000000000) [ ] o.a.s.c.Overseer processMessage: queueSize: 0, message = {
[junit4] 2> "operation":"createshard",
[junit4] 2> "collection":"test",
[junit4] 2> "shard":"x",
[junit4] 2> "replicationFactor":"1"} current state version: 4
[junit4] 2> 1150607 INFO (OverseerStateUpdate-94818526446223363-127.0.0.1:59118_solr-n_0000000000) [ ] o.a.s.c.Overseer processMessage: queueSize: 0, message = {
[junit4] 2> "operation":"addreplica",
[junit4] 2> "collection":"test",
[junit4] 2> "shard":"x",
[junit4] 2> "base_url":"http://127.0.0.1/solr",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "state":"down"} current state version: 4
[junit4] 2> 1150607 INFO (OverseerStateUpdate-94818526446223363-127.0.0.1:59118_solr-n_0000000000) [ ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"addreplica",
[junit4] 2> "collection":"test",
[junit4] 2> "shard":"x",
[junit4] 2> "base_url":"http://127.0.0.1/solr",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "state":"down"}
[junit4] 2> 1150710 INFO (OverseerStateUpdate-94818526446223363-127.0.0.1:59118_solr-n_0000000000) [ ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/test/state.json
[junit4] 2> 1150756 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x150dcf28af40003, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 1150756 INFO (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[7771E2C402B621D6]) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:59118 59118
[junit4] 2> 1152536 INFO (OverseerCollectionConfigSetProcessor-94818526446223363-127.0.0.1:59118_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor
[junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:353)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:350)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:350)
[junit4] 2> at org.apache.solr.cloud.OverseerTaskProcessor.amILeader(OverseerTaskProcessor.java:355)
[junit4] 2> at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:172)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 1152536 INFO (OverseerCollectionConfigSetProcessor-94818526446223363-127.0.0.1:59118_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor According to ZK I (id=94818526446223363-127.0.0.1:59118_solr-n_0000000000) am no longer a leader.
[junit4] 2> 1153008 INFO (Thread-3103) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:59118 59118
[junit4] 2> 1153010 WARN (Thread-3103) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 2 /solr/aliases.json
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 2 /solr/clusterstate.json
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 2 /solr/live_nodes
[junit4] 2> 2 /solr/collections
[junit4] 2>
[junit4] 2> 1153010 INFO (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[7771E2C402B621D6]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testExternalClusterStateChangeBehavior
[junit4] 2> 1153010 INFO (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[7771E2C402B621D6]) [ ] o.a.s.c.Overseer Overseer (id=94818526446223363-127.0.0.1:59118_solr-n_0000000000) closing
[junit4] 2> 1153010 INFO (OverseerStateUpdate-94818526446223363-127.0.0.1:59118_solr-n_0000000000) [ ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:59118_solr
[junit4] 2> 1153013 ERROR (OverseerExitThread) [ ] o.a.s.c.Overseer could not read the data
[junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:353)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:350)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:350)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:304)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:87)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$2.run(Overseer.java:265)
[junit4] 2> 1153014 INFO (TEST-OverseerTest.testOverseerStatsReset-seed#[7771E2C402B621D6]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testOverseerStatsReset
[junit4] 2> 1153015 INFO (TEST-OverseerTest.testOverseerStatsReset-seed#[7771E2C402B621D6]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 1153015 INFO (Thread-3104) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 1153015 INFO (Thread-3104) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 1153115 INFO (TEST-OverseerTest.testOverseerStatsReset-seed#[7771E2C402B621D6]) [ ] o.a.s.c.ZkTestServer start zk server on port:59151
[junit4] 2> 1153115 INFO (TEST-OverseerTest.testOverseerStatsReset-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 1153116 INFO (TEST-OverseerTest.testOverseerStatsReset-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 1153119 INFO (zkCallback-914-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@f2de27e name:ZooKeeperConnection Watcher:127.0.0.1:59151 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1153119 INFO (TEST-OverseerTest.testOverseerStatsReset-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 1153120 INFO (TEST-OverseerTest.testOverseerStatsReset-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 1153121 INFO (TEST-OverseerTest.testOverseerStatsReset-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 1153122 INFO (TEST-OverseerTest.testOverseerStatsReset-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 1153123 INFO (zkCallback-915-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@322835b0 name:ZooKeeperConnection Watcher:127.0.0.1:59151 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1153123 INFO (TEST-OverseerTest.testOverseerStatsReset-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 1153123 INFO (TEST-OverseerTest.testOverseerStatsReset-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 1153123 INFO (TEST-OverseerTest.testOverseerStatsReset-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient makePath: /solr
[junit4] 2> 1153125 INFO (TEST-OverseerTest.testOverseerStatsReset-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 1153126 INFO (TEST-OverseerTest.testOverseerStatsReset-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 1153128 INFO (zkCallback-916-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@568f7762 name:ZooKeeperConnection Watcher:127.0.0.1:59151/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1153128 INFO (TEST-OverseerTest.testOverseerStatsReset-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 1153129 INFO (TEST-OverseerTest.testOverseerStatsReset-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 1153129 INFO (TEST-OverseerTest.testOverseerStatsReset-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
[junit4] 2> 1153130 INFO (TEST-OverseerTest.testOverseerStatsReset-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections
[junit4] 2> 1153131 INFO (TEST-OverseerTest.testOverseerStatsReset-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
[junit4] 2> 1153133 INFO (TEST-OverseerTest.testOverseerStatsReset-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
[junit4] 2> 1153134 INFO (TEST-OverseerTest.testOverseerStatsReset-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient makePath: /security.json
[junit4] 2> 1153135 INFO (TEST-OverseerTest.testOverseerStatsReset-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 1153137 INFO (TEST-OverseerTest.testOverseerStatsReset-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 1153138 INFO (TEST-OverseerTest.testOverseerStatsReset-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 1153139 INFO (zkCallback-917-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@70d39551 name:ZooKeeperConnection Watcher:127.0.0.1:59151/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1153139 INFO (TEST-OverseerTest.testOverseerStatsReset-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 1153140 INFO (TEST-OverseerTest.testOverseerStatsReset-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 1153141 INFO (TEST-OverseerTest.testOverseerStatsReset-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 1153143 INFO (TEST-OverseerTest.testOverseerStatsReset-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/node1
[junit4] 2> 1153145 INFO (zkCallback-916-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 0)
[junit4] 2> 1153145 INFO (zkCallback-917-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 0)
[junit4] 2> 1153145 INFO (TEST-OverseerTest.testOverseerStatsReset-seed#[7771E2C402B621D6]) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=600000&connTimeout=60000&retry=true
[junit4] 2> 1153146 INFO (TEST-OverseerTest.testOverseerStatsReset-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
[junit4] 2> 1153148 INFO (TEST-OverseerTest.testOverseerStatsReset-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
[junit4] 2> 1153150 INFO (TEST-OverseerTest.testOverseerStatsReset-seed#[7771E2C402B621D6]) [ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 1153150 INFO (TEST-OverseerTest.testOverseerStatsReset-seed#[7771E2C402B621D6]) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/94818526619107330-127.0.0.1:59151_solr-n_0000000000
[junit4] 2> 1153151 INFO (TEST-OverseerTest.testOverseerStatsReset-seed#[7771E2C402B621D6]) [ ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:59151_solr
[junit4] 2> 1153151 INFO (TEST-OverseerTest.testOverseerStatsReset-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
[junit4] 2> 1153153 INFO (TEST-OverseerTest.testOverseerStatsReset-seed#[7771E2C402B621D6]) [ ] o.a.s.c.Overseer Overseer (id=94818526619107330-127.0.0.1:59151_solr-n_0000000000) starting
[junit4] 2> 1153156 INFO (TEST-OverseerTest.testOverseerStatsReset-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
[junit4] 2> 1153158 INFO (TEST-OverseerTest.testOverseerStatsReset-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
[junit4] 2> 1153160 INFO (TEST-OverseerTest.testOverseerStatsReset-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
[junit4] 2> 1153162 INFO (TEST-OverseerTest.testOverseerStatsReset-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
[junit4] 2> 1153165 INFO (TEST-OverseerTest.testOverseerStatsReset-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
[junit4] 2> 1153167 INFO (TEST-OverseerTest.testOverseerStatsReset-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
[junit4] 2> 1153171 INFO (TEST-OverseerTest.testOverseerStatsReset-seed#[7771E2C402B621D6]) [ ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 1153171 INFO (OverseerCollectionConfigSetProcessor-94818526619107330-127.0.0.1:59151_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
[junit4] 2> 1153171 INFO (OverseerStateUpdate-94818526619107330-127.0.0.1:59151_solr-n_0000000000) [ ] o.a.s.c.Overseer Starting to work on the main queue
[junit4] 2> 1153174 INFO (OverseerStateUpdate-94818526619107330-127.0.0.1:59151_solr-n_0000000000) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"} current state version: 0
[junit4] 2> 1153175 INFO (OverseerStateUpdate-94818526619107330-127.0.0.1:59151_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 1153175 INFO (OverseerStateUpdate-94818526619107330-127.0.0.1:59151_solr-n_0000000000) [ ] o.a.s.c.o.ClusterStateMutator building a new cName: collection1
[junit4] 2> 1153175 INFO (OverseerStateUpdate-94818526619107330-127.0.0.1:59151_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
[junit4] 2> 1153176 INFO (zkCallback-917-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1153176 INFO (zkCallback-916-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1153673 INFO (TEST-OverseerTest.testOverseerStatsReset-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1153674 INFO (zkCallback-917-thread-1) [ ] o.a.s.c.c.ZkStateReader A collections change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections, has occurred - updating...
[junit4] 2> 1153674 INFO (zkCallback-916-thread-1) [ ] o.a.s.c.c.ZkStateReader A collections change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections, has occurred - updating...
[junit4] 2> 1153678 INFO (TEST-OverseerTest.testOverseerStatsReset-seed#[7771E2C402B621D6]) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /collections/collection1/leader_elect/shard1/election/94818526619107331-node1_core1-n_0000000000
[junit4] 2> 1153679 INFO (TEST-OverseerTest.testOverseerStatsReset-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leaders/shard1
[junit4] 2> 1153682 INFO (TEST-OverseerTest.testOverseerStatsReset-seed#[7771E2C402B621D6]) [ ] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node
[junit4] 2> 1153685 INFO (OverseerStateUpdate-94818526619107330-127.0.0.1:59151_solr-n_0000000000) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "base_url":"http://node1/solr/",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "core":"core1"} current state version: 1
[junit4] 2> 1153685 INFO (OverseerStateUpdate-94818526619107330-127.0.0.1:59151_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=null message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "base_url":"http://node1/solr/",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "core":"core1"}
[junit4] 2> 1153686 INFO (zkCallback-916-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1153686 INFO (zkCallback-917-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1153688 INFO (OverseerStateUpdate-94818526619107330-127.0.0.1:59151_solr-n_0000000000) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "base_url":"http://node1/solr/",
[junit4] 2> "core":"core1"} current state version: 1
[junit4] 2> 1153790 INFO (zkCallback-916-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 1155186 WARN (zkCallback-917-thread-1) [ ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 1155188 INFO (TEST-OverseerTest.testOverseerStatsReset-seed#[7771E2C402B621D6]) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:59151 59151
[junit4] 2> 1157173 INFO (OverseerCollectionConfigSetProcessor-94818526619107330-127.0.0.1:59151_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor
[junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:353)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:350)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:350)
[junit4] 2> at org.apache.solr.cloud.OverseerTaskProcessor.amILeader(OverseerTaskProcessor.java:355)
[junit4] 2> at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:172)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 1157173 INFO (OverseerCollectionConfigSetProcessor-94818526619107330-127.0.0.1:59151_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor According to ZK I (id=94818526619107330-127.0.0.1:59151_solr-n_0000000000) am no longer a leader.
[junit4] 2> 1157439 INFO (Thread-3104) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:59151 59151
[junit4] 2> 1157441 WARN (Thread-3104) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 2 /solr/aliases.json
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 2 /solr/clusterstate.json
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 2 /solr/live_nodes
[junit4] 2> 2 /solr/collections
[junit4] 2>
[junit4] 2> 1157441 INFO (TEST-OverseerTest.testOverseerStatsReset-seed#[7771E2C402B621D6]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testOverseerStatsReset
[junit4] 2> 1157441 INFO (TEST-OverseerTest.testOverseerStatsReset-seed#[7771E2C402B621D6]) [ ] o.a.s.c.Overseer Overseer (id=94818526619107330-127.0.0.1:59151_solr-n_0000000000) closing
[junit4] 2> 1157441 INFO (OverseerStateUpdate-94818526619107330-127.0.0.1:59151_solr-n_0000000000) [ ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:59151_solr
[junit4] 2> 1157443 ERROR (OverseerExitThread) [ ] o.a.s.c.Overseer could not read the data
[junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:353)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:350)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:350)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:304)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:87)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$2.run(Overseer.java:265)
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=OverseerTest -Dtests.method=testOverseerStatsReset -Dtests.seed=7771E2C402B621D6 -Dtests.slow=true -Dtests.locale=en_AU -Dtests.timezone=Brazil/DeNoronha -Dtests.asserts=true -Dtests.file.encoding=Cp1252
[junit4] FAILURE 4.43s J1 | OverseerTest.testOverseerStatsReset <<<
[junit4] > Throwable #1: java.lang.AssertionError: expected:<1> but was:<2>
[junit4] > at __randomizedtesting.SeedInfo.seed([7771E2C402B621D6:DC2505F8976C82D8]:0)
[junit4] > at org.apache.solr.cloud.OverseerTest.testOverseerStatsReset(OverseerTest.java:722)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 1157448 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testBadQueueItem
[junit4] 2> 1157449 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 1157449 INFO (Thread-3105) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 1157449 INFO (Thread-3105) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 1157550 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.ZkTestServer start zk server on port:59187
[junit4] 2> 1157550 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 1157551 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 1157554 INFO (zkCallback-919-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@75df52dd name:ZooKeeperConnection Watcher:127.0.0.1:59187 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1157554 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 1157554 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 1157555 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 1157557 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 1157558 INFO (zkCallback-920-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@5237e3f1 name:ZooKeeperConnection Watcher:127.0.0.1:59187 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1157558 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 1157558 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 1157558 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient makePath: /solr
[junit4] 2> 1157562 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 1157563 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 1157563 INFO (zkCallback-921-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@4eb73b6c name:ZooKeeperConnection Watcher:127.0.0.1:59187/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1157563 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 1157563 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 1157564 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
[junit4] 2> 1157565 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections
[junit4] 2> 1157566 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
[junit4] 2> 1157568 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
[junit4] 2> 1157569 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient makePath: /security.json
[junit4] 2> 1157570 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 1157571 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 1157572 INFO (zkCallback-922-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@3466fc0f name:ZooKeeperConnection Watcher:127.0.0.1:59187/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1157572 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 1157572 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 1157572 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=600000&connTimeout=60000&retry=true
[junit4] 2> 1157574 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
[junit4] 2> 1157575 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
[junit4] 2> 1157576 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 1157577 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/94818526909628419-127.0.0.1:59187_solr-n_0000000000
[junit4] 2> 1157577 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:59187_solr
[junit4] 2> 1157577 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
[junit4] 2> 1157579 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.Overseer Overseer (id=94818526909628419-127.0.0.1:59187_solr-n_0000000000) starting
[junit4] 2> 1157580 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
[junit4] 2> 1157582 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
[junit4] 2> 1157584 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
[junit4] 2> 1157586 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
[junit4] 2> 1157588 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
[junit4] 2> 1157590 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
[junit4] 2> 1157593 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 1157594 INFO (OverseerCollectionConfigSetProcessor-94818526909628419-127.0.0.1:59187_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
[junit4] 2> 1157594 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 1157594 INFO (OverseerStateUpdate-94818526909628419-127.0.0.1:59187_solr-n_0000000000) [ ] o.a.s.c.Overseer Starting to work on the main queue
[junit4] 2> 1157595 INFO (OverseerStateUpdate-94818526909628419-127.0.0.1:59187_solr-n_0000000000) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 1157595 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 1157597 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 1157598 INFO (zkCallback-924-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@4c671eec name:ZooKeeperConnection Watcher:127.0.0.1:59187/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1157598 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 1157598 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 1157600 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 1157602 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1
[junit4] 2> 1157603 INFO (zkCallback-921-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 0)
[junit4] 2> 1157603 INFO (zkCallback-924-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 0)
[junit4] 2> 1157603 INFO (zkCallback-922-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 0)
[junit4] 2> 1157606 INFO (OverseerStateUpdate-94818526909628419-127.0.0.1:59187_solr-n_0000000000) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"} current state version: 0
[junit4] 2> 1157606 INFO (OverseerStateUpdate-94818526909628419-127.0.0.1:59187_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 1157606 INFO (OverseerStateUpdate-94818526909628419-127.0.0.1:59187_solr-n_0000000000) [ ] o.a.s.c.o.ClusterStateMutator building a new cName: collection1
[junit4] 2> 1157606 INFO (OverseerStateUpdate-94818526909628419-127.0.0.1:59187_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard2
[junit4] 2> 1157607 INFO (zkCallback-921-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1157607 INFO (zkCallback-924-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1157607 INFO (zkCallback-922-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1158105 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 1158106 INFO (zkCallback-924-thread-1) [ ] o.a.s.c.c.ZkStateReader A collections change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections, has occurred - updating...
[junit4] 2> 1158106 INFO (zkCallback-921-thread-1) [ ] o.a.s.c.c.ZkStateReader A collections change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections, has occurred - updating...
[junit4] 2> 1158106 INFO (zkCallback-922-thread-1) [ ] o.a.s.c.c.ZkStateReader A collections change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections, has occurred - updating...
[junit4] 2> 1158111 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /collections/collection1/leader_elect/shard2/election/94818526909628420-127.0.0.1_core1-n_0000000000
[junit4] 2> 1158112 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leaders/shard2
[junit4] 2> 1158115 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node
[junit4] 2> 1158119 INFO (OverseerStateUpdate-94818526909628419-127.0.0.1:59187_solr-n_0000000000) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "base_url":"http://127.0.0.1/solr/",
[junit4] 2> "core":"core1"} current state version: 1
[junit4] 2> 1158123 INFO (OverseerStateUpdate-94818526909628419-127.0.0.1:59187_solr-n_0000000000) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core2",
[junit4] 2> "core_node_name":"node2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"} current state version: 1
[junit4] 2> 1158124 INFO (OverseerStateUpdate-94818526909628419-127.0.0.1:59187_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core2",
[junit4] 2> "core_node_name":"node2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 1158124 INFO (OverseerStateUpdate-94818526909628419-127.0.0.1:59187_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator Collection already exists with numShards=3
[junit4] 2> 1158124 INFO (OverseerStateUpdate-94818526909628419-127.0.0.1:59187_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard3
[junit4] 2> 1158229 INFO (zkCallback-921-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1158229 INFO (zkCallback-924-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1158229 INFO (zkCallback-922-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1158621 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leader_elect/shard3/election
[junit4] 2> 1158626 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /collections/collection1/leader_elect/shard3/election/94818526909628420-127.0.0.1_core2-n_0000000000
[junit4] 2> 1158627 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leaders/shard3
[junit4] 2> 1158630 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node
[junit4] 2> 1158633 INFO (OverseerStateUpdate-94818526909628419-127.0.0.1:59187_solr-n_0000000000) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard3",
[junit4] 2> "collection":"collection1",
[junit4] 2> "base_url":"http://127.0.0.1/solr/",
[junit4] 2> "core":"core2"} current state version: 2
[junit4] 2> 1158635 INFO (OverseerStateUpdate-94818526909628419-127.0.0.1:59187_solr-n_0000000000) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core3",
[junit4] 2> "core_node_name":"node3",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"} current state version: 2
[junit4] 2> 1158636 INFO (OverseerStateUpdate-94818526909628419-127.0.0.1:59187_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core3",
[junit4] 2> "core_node_name":"node3",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 1158636 INFO (OverseerStateUpdate-94818526909628419-127.0.0.1:59187_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator Collection already exists with numShards=3
[junit4] 2> 1158636 INFO (OverseerStateUpdate-94818526909628419-127.0.0.1:59187_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
[junit4] 2> 1158738 INFO (zkCallback-921-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1158739 INFO (zkCallback-924-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1158739 INFO (zkCallback-922-thread-2) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1159134 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1159138 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /collections/collection1/leader_elect/shard1/election/94818526909628420-127.0.0.1_core3-n_0000000000
[junit4] 2> 1159140 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leaders/shard1
[junit4] 2> 1159142 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node
[junit4] 2> 1159145 INFO (OverseerStateUpdate-94818526909628419-127.0.0.1:59187_solr-n_0000000000) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "base_url":"http://127.0.0.1/solr/",
[junit4] 2> "core":"core3"} current state version: 3
[junit4] 2> 1159247 INFO (zkCallback-921-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1159247 INFO (zkCallback-924-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1159248 INFO (zkCallback-922-thread-2) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1159298 INFO (OverseerStateUpdate-94818526909628419-127.0.0.1:59187_solr-n_0000000000) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core0",
[junit4] 2> "core_node_name":"node0",
[junit4] 2> "collection":"",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"} current state version: 4
[junit4] 2> 1159298 ERROR (OverseerStateUpdate-94818526909628419-127.0.0.1:59187_solr-n_0000000000) [ ] o.a.s.c.o.CollectionMutator Skipping invalid Overseer message because it has no collection specified: {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core0",
[junit4] 2> "core_node_name":"node0",
[junit4] 2> "collection":"",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 1159301 INFO (OverseerStateUpdate-94818526909628419-127.0.0.1:59187_solr-n_0000000000) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"deletecore",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core0",
[junit4] 2> "core_node_name":"node0",
[junit4] 2> "collection":""} current state version: 4
[junit4] 2> 1159301 ERROR (OverseerStateUpdate-94818526909628419-127.0.0.1:59187_solr-n_0000000000) [ ] o.a.s.c.o.CollectionMutator Skipping invalid Overseer message because it has no collection specified: {
[junit4] 2> "operation":"deletecore",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core0",
[junit4] 2> "core_node_name":"node0",
[junit4] 2> "collection":""}
[junit4] 2> 1159302 INFO (OverseerStateUpdate-94818526909628419-127.0.0.1:59187_solr-n_0000000000) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection2",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"} current state version: 4
[junit4] 2> 1159302 INFO (OverseerStateUpdate-94818526909628419-127.0.0.1:59187_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection2",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 1159302 INFO (OverseerStateUpdate-94818526909628419-127.0.0.1:59187_solr-n_0000000000) [ ] o.a.s.c.o.ClusterStateMutator building a new cName: collection2
[junit4] 2> 1159302 INFO (OverseerStateUpdate-94818526909628419-127.0.0.1:59187_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard2
[junit4] 2> 1159407 INFO (zkCallback-921-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1159407 INFO (zkCallback-922-thread-2) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1159407 INFO (zkCallback-924-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1159801 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection2/leader_elect/shard2/election
[junit4] 2> 1159802 INFO (zkCallback-921-thread-1) [ ] o.a.s.c.c.ZkStateReader A collections change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections, has occurred - updating...
[junit4] 2> 1159802 INFO (zkCallback-922-thread-2) [ ] o.a.s.c.c.ZkStateReader A collections change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections, has occurred - updating...
[junit4] 2> 1159802 INFO (zkCallback-924-thread-1) [ ] o.a.s.c.c.ZkStateReader A collections change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections, has occurred - updating...
[junit4] 2> 1159806 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /collections/collection2/leader_elect/shard2/election/94818526909628420-127.0.0.1_core1-n_0000000000
[junit4] 2> 1159807 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection2/leaders/shard2
[junit4] 2> 1159809 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node
[junit4] 2> 1159812 INFO (OverseerStateUpdate-94818526909628419-127.0.0.1:59187_solr-n_0000000000) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard2",
[junit4] 2> "collection":"collection2",
[junit4] 2> "base_url":"http://127.0.0.1/solr/",
[junit4] 2> "core":"core1"} current state version: 5
[junit4] 2> 1159813 INFO (OverseerStateUpdate-94818526909628419-127.0.0.1:59187_solr-n_0000000000) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core2",
[junit4] 2> "core_node_name":"node2",
[junit4] 2> "collection":"collection2",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"} current state version: 5
[junit4] 2> 1159813 INFO (OverseerStateUpdate-94818526909628419-127.0.0.1:59187_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core2",
[junit4] 2> "core_node_name":"node2",
[junit4] 2> "collection":"collection2",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 1159813 INFO (OverseerStateUpdate-94818526909628419-127.0.0.1:59187_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator Collection already exists with numShards=3
[junit4] 2> 1159813 INFO (OverseerStateUpdate-94818526909628419-127.0.0.1:59187_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard3
[junit4] 2> 1159916 INFO (zkCallback-921-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1159917 INFO (zkCallback-924-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1159917 INFO (zkCallback-922-thread-2) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1160312 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection2/leader_elect/shard3/election
[junit4] 2> 1160313 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /collections/collection2/leader_elect/shard3/election/94818526909628420-127.0.0.1_core2-n_0000000000
[junit4] 2> 1160314 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection2/leaders/shard3
[junit4] 2> 1160317 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node
[junit4] 2> 1160319 INFO (OverseerStateUpdate-94818526909628419-127.0.0.1:59187_solr-n_0000000000) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard3",
[junit4] 2> "collection":"collection2",
[junit4] 2> "base_url":"http://127.0.0.1/solr/",
[junit4] 2> "core":"core2"} current state version: 6
[junit4] 2> 1160321 INFO (OverseerStateUpdate-94818526909628419-127.0.0.1:59187_solr-n_0000000000) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core3",
[junit4] 2> "core_node_name":"node3",
[junit4] 2> "collection":"collection2",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"} current state version: 6
[junit4] 2> 1160321 INFO (OverseerStateUpdate-94818526909628419-127.0.0.1:59187_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core3",
[junit4] 2> "core_node_name":"node3",
[junit4] 2> "collection":"collection2",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 1160321 INFO (OverseerStateUpdate-94818526909628419-127.0.0.1:59187_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator Collection already exists with numShards=3
[junit4] 2> 1160321 INFO (OverseerStateUpdate-94818526909628419-127.0.0.1:59187_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
[junit4] 2> 1160425 INFO (zkCallback-924-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1160425 INFO (zkCallback-921-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1160425 INFO (zkCallback-922-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1160819 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection2/leader_elect/shard1/election
[junit4] 2> 1160823 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /collections/collection2/leader_elect/shard1/election/94818526909628420-127.0.0.1_core3-n_0000000000
[junit4] 2> 1160824 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection2/leaders/shard1
[junit4] 2> 1160826 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node
[junit4] 2> 1160829 INFO (OverseerStateUpdate-94818526909628419-127.0.0.1:59187_solr-n_0000000000) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection2",
[junit4] 2> "base_url":"http://127.0.0.1/solr/",
[junit4] 2> "core":"core3"} current state version: 7
[junit4] 2> 1160932 INFO (zkCallback-921-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1160932 INFO (zkCallback-924-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1160932 INFO (zkCallback-922-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1160981 INFO (zkCallback-924-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1160981 INFO (zkCallback-922-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1162481 WARN (zkCallback-924-thread-1) [ ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 1162483 INFO (TEST-OverseerTest.testBadQueueItem-seed#[7771E2C402B621D6]) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:59187 59187
[junit4] 2> 1163597 INFO (OverseerCollectionConfigSetProcessor-94818526909628419-127.0.0.1:59187_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor
[junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:353)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:350)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:350)
[junit4] 2> at org.apache.solr.cloud.OverseerTaskProcessor.amILeader(OverseerTaskProcessor.java:355)
[junit4] 2> at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:172)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 1163597 INFO (OverseerCollectionConfigSetProcessor-94818526909628419-127.0.0.1:59187_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor According to ZK I (id=94818526909628419-127.0.0.1:59187_solr-n_0000000000) am no longer a leader.
[junit4] 2> 1164735 INFO (Thread-3105) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:59187 59187
[junit4] 2> 1164737 WARN (Thread-3105) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[j
[...truncated too long message...]
perException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer/queue
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:336)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:333)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:333)
[junit4] 2> at org.apache.solr.cloud.DistributedQueue.fetchZkChildren(DistributedQueue.java:310)
[junit4] 2> at org.apache.solr.cloud.DistributedQueue.firstChild(DistributedQueue.java:289)
[junit4] 2> at org.apache.solr.cloud.DistributedQueue.firstElement(DistributedQueue.java:358)
[junit4] 2> at org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:165)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:233)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 1232961 INFO (OverseerStateUpdate-94818531705356292-127.0.0.1:59632_solr-n_0000000000) [ ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:59632_solr
[junit4] 2> 1232962 WARN (zkCallback-1008-thread-1) [ ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 1232965 ERROR (OverseerExitThread) [ ] o.a.s.c.Overseer could not read the data
[junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:353)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:350)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:350)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:304)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:87)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$2.run(Overseer.java:265)
[junit4] 2> 1232967 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x150dcf3c46c0000, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 1232968 INFO (TEST-OverseerTest.testPlaceholders-seed#[7771E2C402B621D6]) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:59632 59632
[junit4] 2> 1235396 INFO (Thread-3114) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:59632 59632
[junit4] 2> 1235606 WARN (Thread-3114) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 3 /solr/aliases.json
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 3 /solr/clusterstate.json
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 3 /solr/live_nodes
[junit4] 2> 3 /solr/collections
[junit4] 2>
[junit4] 2> 1235606 INFO (TEST-OverseerTest.testPlaceholders-seed#[7771E2C402B621D6]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testPlaceholders
[junit4] 2> 1235606 INFO (TEST-OverseerTest.testPlaceholders-seed#[7771E2C402B621D6]) [ ] o.a.s.c.Overseer Overseer (id=94818531705356292-127.0.0.1:59632_solr-n_0000000000) closing
[junit4] 2> 1238841 INFO (SUITE-OverseerTest-seed#[7771E2C402B621D6]-worker) [ ] o.a.s.SolrTestCaseJ4 ###deleteCore
[junit4] 2> NOTE: leaving temporary files on disk at: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.OverseerTest_7771E2C402B621D6-001
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene60): {}, docValues:{}, sim=ClassicSimilarity, locale=en_AU, timezone=Brazil/DeNoronha
[junit4] 2> NOTE: Windows 7 6.1 amd64/Oracle Corporation 1.8.0_66 (64-bit)/cpus=3,threads=1,free=140450024,total=518979584
[junit4] 2> NOTE: All tests run in this JVM: [BaseCdcrDistributedZkTest, TestRangeQuery, BadCopyFieldTest, DirectUpdateHandlerOptimizeTest, CurrencyFieldOpenExchangeTest, ParsingFieldUpdateProcessorsTest, TestOmitPositions, MinimalSchemaTest, PKIAuthenticationIntegrationTest, TestConfigOverlay, AsyncCallRequestStatusResponseTest, CollectionsAPIDistributedZkTest, TestRawTransformer, MultiThreadedOCPTest, TestWriterPerf, SpellCheckCollatorTest, TestConfigSetImmutable, TestPerFieldSimilarity, TestInfoStreamLogging, TestManagedSchemaFieldResource, TestMinMaxOnMultiValuedField, TestTrie, TestCSVResponseWriter, DirectSolrConnectionTest, DefaultValueUpdateProcessorTest, TestPHPSerializedResponseWriter, TestMacros, EchoParamsTest, CdcrReplicationDistributedZkTest, SolrCoreCheckLockOnStartupTest, GraphQueryTest, TestBinaryField, TestHighFrequencyDictionaryFactory, LeaderInitiatedRecoveryOnCommitTest, TestTolerantSearch, AliasIntegrationTest, ShardRoutingCustomTest, SpellPossibilityIteratorTest, HighlighterMaxOffsetTest, OpenCloseCoreStressTest, EnumFieldTest, DistributedQueueTest, TestCustomDocTransformer, ResponseLogComponentTest, TestQueryTypes, DateRangeFieldTest, BlockCacheTest, CleanupOldIndexTest, CSVRequestHandlerTest, TestPKIAuthenticationPlugin, TestObjectReleaseTracker, TestSolrCloudWithKerberosAlt, PrimUtilsTest, ZkStateWriterTest, TriLevelCompositeIdRoutingTest, JavabinLoaderTest, TestQueryUtils, TestSuggestSpellingConverter, TestSchemaSimilarityResource, SolrCloudExampleTest, UUIDUpdateProcessorFallbackTest, TestDistributedGrouping, TestUniqueKeyFieldResource, BadIndexSchemaTest, TestSolrQueryParserResource, TestRandomFaceting, TestCollationField, TestSimpleTrackingShardHandler, BasicAuthIntegrationTest, TermVectorComponentDistributedTest, TestReversedWildcardFilterFactory, OpenExchangeRatesOrgProviderTest, TestLMJelinekMercerSimilarityFactory, FieldMutatingUpdateProcessorTest, DistributedFacetPivotLargeTest, SolrIndexSplitterTest, TestJoin, SpatialRPTFieldTypeTest, DateMathParserTest, SolrInfoMBeanTest, TestSha256AuthenticationProvider, TestSolrDeletionPolicy2, TestRealTimeGet, TestValueSourceCache, NotRequiredUniqueKeyTest, SignatureUpdateProcessorFactoryTest, TestShortCircuitedRequests, TestZkChroot, WordBreakSolrSpellCheckerTest, TestStressUserVersions, TestManagedStopFilterFactory, VersionInfoTest, TestClusterStateMutator, CdcrVersionReplicationTest, CdcrUpdateLogTest, LoggingHandlerTest, TestDefaultStatsCache, TestFoldingMultitermQuery, TestSortByMinMaxFunction, TestOverriddenPrefixQueryForCustomFieldType, SolrXmlInZkTest, HighlighterConfigTest, TestSolrDynamicMBean, BlockDirectoryTest, LukeRequestHandlerTest, AnalyticsQueryTest, TestReload, TestCopyFieldCollectionResource, TestFieldCollectionResource, TestSolrQueryParserDefaultOperatorResource, TestDynamicFieldCollectionResource, SpellCheckComponentTest, TestFilteredDocIdSet, TestConfigSetsAPI, TestSolr4Spatial, TestExpandComponent, SortByFunctionTest, OverseerStatusTest, TestBM25SimilarityFactory, BasicZkTest, TestBulkSchemaAPI, TestFreeTextSuggestions, CloneFieldUpdateProcessorFactoryTest, DataDrivenBlockJoinTest, StatsComponentTest, TestCoreDiscovery, StressHdfsTest, DeleteShardTest, TestIndexingPerformance, NumericFieldsTest, ZkSolrClientTest, TestReRankQParserPlugin, BitVectorTest, BasicDistributedZkTest, ChaosMonkeySafeLeaderTest, BasicDistributedZk2Test, OverseerTest]
[junit4] Completed [228/555] on J1 in 88.49s, 12 tests, 1 failure, 1 skipped <<< FAILURES!
[...truncated 1050 lines...]
BUILD FAILED
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:775: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:719: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:59: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build.xml:233: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\common-build.xml:516: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:1432: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:987: There were test failures: 555 suites (8 ignored), 2185 tests, 1 failure, 86 ignored (48 assumptions) [seed: 7771E2C402B621D6]
Total time: 69 minutes 11 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Lucene-Solr-trunk-Windows (64bit/jdk1.8.0_66) - Build #
5378 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Windows/5378/
Java: 64bit/jdk1.8.0_66 -XX:+UseCompressedOops -XX:+UseSerialGC
1 tests failed.
FAILED: org.apache.solr.cloud.OverseerTest.testOverseerStatsReset
Error Message:
expected:<1> but was:<2>
Stack Trace:
java.lang.AssertionError: expected:<1> but was:<2>
at __randomizedtesting.SeedInfo.seed([A8631044D0ECD97:A1D2D638D8D46E99]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at org.apache.solr.cloud.OverseerTest.testOverseerStatsReset(OverseerTest.java:722)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1660)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:866)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:902)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:916)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:875)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:777)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:811)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:822)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at java.lang.Thread.run(Thread.java:745)
Build Log:
[...truncated 9977 lines...]
[junit4] Suite: org.apache.solr.cloud.OverseerTest
[junit4] 2> Creating dataDir: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.OverseerTest_A8631044D0ECD97-001\init-core-data-001
[junit4] 2> 860007 INFO (SUITE-OverseerTest-seed#[A8631044D0ECD97]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false)
[junit4] 2> 860007 INFO (SUITE-OverseerTest-seed#[A8631044D0ECD97]-worker) [ ] o.a.s.SolrTestCaseJ4 ####initCore
[junit4] 2> 860009 INFO (SUITE-OverseerTest-seed#[A8631044D0ECD97]-worker) [ ] o.a.s.SolrTestCaseJ4 ####initCore end
[junit4] 2> 860010 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testDoubleAssignment
[junit4] 2> 860011 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 860011 INFO (Thread-1483) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 860011 INFO (Thread-1483) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 860131 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.ZkTestServer start zk server on port:58816
[junit4] 2> 860131 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 860133 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 860137 INFO (zkCallback-1355-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@5856f1cd name:ZooKeeperConnection Watcher:127.0.0.1:58816/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 860137 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 860137 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 860137 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 860141 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 860143 INFO (zkCallback-1356-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@71aff226 name:ZooKeeperConnection Watcher:127.0.0.1:58816 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 860143 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 860143 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 860147 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x150df0417c90001, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 860147 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 860148 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 860150 INFO (zkCallback-1357-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@3ba89e94 name:ZooKeeperConnection Watcher:127.0.0.1:58816 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 860150 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 860150 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 860157 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /solr
[junit4] 2> 860165 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
[junit4] 2> 860165 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections
[junit4] 2> 860167 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
[junit4] 2> 860168 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
[junit4] 2> 860170 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /security.json
[junit4] 2> 860171 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 860173 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 860175 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 860176 INFO (zkCallback-1358-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@28a724dc name:ZooKeeperConnection Watcher:127.0.0.1:58816/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 860176 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 860176 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 860178 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 860180 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/node1
[junit4] 2> 860182 INFO (zkCallback-1355-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 0)
[junit4] 2> 860182 INFO (zkCallback-1358-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 0)
[junit4] 2> 860183 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 860185 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 860186 INFO (zkCallback-1359-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@4634f45f name:ZooKeeperConnection Watcher:127.0.0.1:58816/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 860187 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 860187 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 860187 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=600000&connTimeout=60000&retry=true
[junit4] 2> 860198 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
[junit4] 2> 860199 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
[junit4] 2> 860200 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 860200 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/94820800846692356-127.0.0.1:58816_solr-n_0000000000
[junit4] 2> 860200 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:58816_solr
[junit4] 2> 860200 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
[junit4] 2> 860201 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.Overseer Overseer (id=94820800846692356-127.0.0.1:58816_solr-n_0000000000) starting
[junit4] 2> 860201 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
[junit4] 2> 860204 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
[junit4] 2> 860205 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
[junit4] 2> 860206 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
[junit4] 2> 860208 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
[junit4] 2> 860209 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
[junit4] 2> 860214 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 860214 INFO (OverseerStateUpdate-94820800846692356-127.0.0.1:58816_solr-n_0000000000) [ ] o.a.s.c.Overseer Starting to work on the main queue
[junit4] 2> 860215 INFO (OverseerStateUpdate-94820800846692356-127.0.0.1:58816_solr-n_0000000000) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 860215 INFO (OverseerCollectionConfigSetProcessor-94820800846692356-127.0.0.1:58816_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
[junit4] 2> 860219 INFO (OverseerStateUpdate-94820800846692356-127.0.0.1:58816_solr-n_0000000000) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"} current state version: 0
[junit4] 2> 860219 INFO (OverseerStateUpdate-94820800846692356-127.0.0.1:58816_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 860219 INFO (OverseerStateUpdate-94820800846692356-127.0.0.1:58816_solr-n_0000000000) [ ] o.a.s.c.o.ClusterStateMutator building a new cName: collection1
[junit4] 2> 860219 INFO (OverseerStateUpdate-94820800846692356-127.0.0.1:58816_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
[junit4] 2> 860220 INFO (zkCallback-1359-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 860220 INFO (zkCallback-1355-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 860220 INFO (zkCallback-1358-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 860811 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 860812 INFO (zkCallback-1359-thread-1) [ ] o.a.s.c.c.ZkStateReader A collections change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections, has occurred - updating...
[junit4] 2> 860812 INFO (zkCallback-1355-thread-1) [ ] o.a.s.c.c.ZkStateReader A collections change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections, has occurred - updating...
[junit4] 2> 860812 INFO (zkCallback-1358-thread-1) [ ] o.a.s.c.c.ZkStateReader A collections change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections, has occurred - updating...
[junit4] 2> 860817 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /collections/collection1/leader_elect/shard1/election/94820800846692355-node1_core1-n_0000000000
[junit4] 2> 860818 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leaders/shard1
[junit4] 2> 860820 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node
[junit4] 2> 860823 INFO (OverseerStateUpdate-94820800846692356-127.0.0.1:58816_solr-n_0000000000) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "base_url":"http://node1/solr/",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "core":"core1"} current state version: 1
[junit4] 2> 860824 INFO (OverseerStateUpdate-94820800846692356-127.0.0.1:58816_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=null message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "base_url":"http://node1/solr/",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "core":"core1"}
[junit4] 2> 860826 INFO (zkCallback-1359-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 860826 INFO (zkCallback-1355-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 860826 INFO (zkCallback-1358-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 860827 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x150df0417c90003, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 860827 INFO (OverseerStateUpdate-94820800846692356-127.0.0.1:58816_solr-n_0000000000) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "base_url":"http://node1/solr/",
[junit4] 2> "core":"core1"} current state version: 1
[junit4] 2> 860949 INFO (zkCallback-1359-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 860949 INFO (zkCallback-1355-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 862610 WARN (zkCallback-1358-thread-1) [ ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 862610 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 862611 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 862613 INFO (zkCallback-1361-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@577ee554 name:ZooKeeperConnection Watcher:127.0.0.1:58816/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 862613 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 862613 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 862617 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 862620 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/node1
[junit4] 2> 862622 INFO (zkCallback-1359-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 0)
[junit4] 2> 862622 INFO (zkCallback-1361-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 0)
[junit4] 2> 862622 INFO (zkCallback-1355-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 0)
[junit4] 2> 862624 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 862625 INFO (OverseerStateUpdate-94820800846692356-127.0.0.1:58816_solr-n_0000000000) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"} current state version: 2
[junit4] 2> 862625 INFO (OverseerStateUpdate-94820800846692356-127.0.0.1:58816_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 862625 INFO (OverseerStateUpdate-94820800846692356-127.0.0.1:58816_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator shard=shard1 is already registered
[junit4] 2> 862627 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /collections/collection1/leader_elect/shard1/election/94820800846692357-node1_core1-n_0000000001
[junit4] 2> 862628 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node
[junit4] 2> 862632 INFO (OverseerStateUpdate-94820800846692356-127.0.0.1:58816_solr-n_0000000000) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "base_url":"http://node1/solr/",
[junit4] 2> "core":"core1"} current state version: 2
[junit4] 2> 862754 INFO (zkCallback-1359-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 862754 INFO (zkCallback-1361-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 862754 INFO (zkCallback-1355-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 862757 INFO (zkCallback-1361-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 862759 INFO (zkCallback-1355-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 864588 WARN (OverseerStateUpdate-94820800846692356-127.0.0.1:58816_solr-n_0000000000) [ ] o.a.s.c.Overseer Solr cannot talk to ZK, exiting Overseer main queue loop
[junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer/queue-work
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:336)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:333)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:333)
[junit4] 2> at org.apache.solr.cloud.DistributedQueue.fetchZkChildren(DistributedQueue.java:310)
[junit4] 2> at org.apache.solr.cloud.DistributedQueue.firstChild(DistributedQueue.java:289)
[junit4] 2> at org.apache.solr.cloud.DistributedQueue.removeFirst(DistributedQueue.java:378)
[junit4] 2> at org.apache.solr.cloud.DistributedQueue.poll(DistributedQueue.java:187)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:239)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 864588 INFO (OverseerStateUpdate-94820800846692356-127.0.0.1:58816_solr-n_0000000000) [ ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:58816_solr
[junit4] 2> 864589 ERROR (OverseerExitThread) [ ] o.a.s.c.Overseer could not read the data
[junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:353)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:350)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:350)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:304)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:87)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$2.run(Overseer.java:265)
[junit4] 2> 864590 WARN (zkCallback-1361-thread-1) [ ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 864594 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:58816 58816
[junit4] 2> 865066 INFO (OverseerCollectionConfigSetProcessor-94820800846692356-127.0.0.1:58816_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor
[junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:353)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:350)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:350)
[junit4] 2> at org.apache.solr.cloud.OverseerTaskProcessor.amILeader(OverseerTaskProcessor.java:355)
[junit4] 2> at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:172)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 865066 INFO (OverseerCollectionConfigSetProcessor-94820800846692356-127.0.0.1:58816_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor According to ZK I (id=94820800846692356-127.0.0.1:58816_solr-n_0000000000) am no longer a leader.
[junit4] 2> 867539 INFO (Thread-1483) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:58816 58816
[junit4] 2> 867541 WARN (Thread-1483) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 4 /solr/aliases.json
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 4 /solr/clusterstate.json
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 4 /solr/collections
[junit4] 2> 3 /solr/live_nodes
[junit4] 2>
[junit4] 2> 867541 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testDoubleAssignment
[junit4] 2> 867541 INFO (TEST-OverseerTest.testDoubleAssignment-seed#[A8631044D0ECD97]) [ ] o.a.s.c.Overseer Overseer (id=94820800846692356-127.0.0.1:58816_solr-n_0000000000) closing
[junit4] 2> 867545 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testOverseerFailure
[junit4] 2> 867550 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 867550 INFO (Thread-1484) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 867550 INFO (Thread-1484) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 867682 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.ZkTestServer start zk server on port:58837
[junit4] 2> 867682 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 867683 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 867688 INFO (zkCallback-1362-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@460b500e name:ZooKeeperConnection Watcher:127.0.0.1:58837 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 867688 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 867688 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 867691 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 867693 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 867695 INFO (zkCallback-1363-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@700fb258 name:ZooKeeperConnection Watcher:127.0.0.1:58837 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 867695 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 867695 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 867695 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /solr
[junit4] 2> 867700 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 867701 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 867702 INFO (zkCallback-1364-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@48835df9 name:ZooKeeperConnection Watcher:127.0.0.1:58837/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 867702 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 867702 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 867704 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
[junit4] 2> 867706 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections
[junit4] 2> 867708 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
[junit4] 2> 867710 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
[junit4] 2> 867711 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /security.json
[junit4] 2> 867714 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 867716 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 867718 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 867719 INFO (zkCallback-1365-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@62f08480 name:ZooKeeperConnection Watcher:127.0.0.1:58837/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 867719 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 867719 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 867722 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 867725 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/node1
[junit4] 2> 867727 INFO (zkCallback-1365-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 0)
[junit4] 2> 867727 INFO (zkCallback-1364-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 0)
[junit4] 2> 867728 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 867731 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 867732 INFO (zkCallback-1366-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@4fcf925 name:ZooKeeperConnection Watcher:127.0.0.1:58837/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 867732 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 867732 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 867732 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=600000&connTimeout=60000&retry=true
[junit4] 2> 867736 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
[junit4] 2> 867738 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
[junit4] 2> 867741 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 867741 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/94820801340833796-127.0.0.1:58837_solr-n_0000000000
[junit4] 2> 867742 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:58837_solr
[junit4] 2> 867742 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
[junit4] 2> 867743 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.Overseer Overseer (id=94820801340833796-127.0.0.1:58837_solr-n_0000000000) starting
[junit4] 2> 867745 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
[junit4] 2> 867748 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
[junit4] 2> 867751 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
[junit4] 2> 867753 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
[junit4] 2> 867756 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
[junit4] 2> 867759 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
[junit4] 2> 867764 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 867765 INFO (OverseerCollectionConfigSetProcessor-94820801340833796-127.0.0.1:58837_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
[junit4] 2> 867765 INFO (OverseerStateUpdate-94820801340833796-127.0.0.1:58837_solr-n_0000000000) [ ] o.a.s.c.Overseer Starting to work on the main queue
[junit4] 2> 867766 INFO (OverseerStateUpdate-94820801340833796-127.0.0.1:58837_solr-n_0000000000) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 869076 INFO (OverseerStateUpdate-94820801340833796-127.0.0.1:58837_solr-n_0000000000) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"} current state version: 0
[junit4] 2> 869077 INFO (OverseerStateUpdate-94820801340833796-127.0.0.1:58837_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 869077 INFO (OverseerStateUpdate-94820801340833796-127.0.0.1:58837_solr-n_0000000000) [ ] o.a.s.c.o.ClusterStateMutator building a new cName: collection1
[junit4] 2> 869077 INFO (OverseerStateUpdate-94820801340833796-127.0.0.1:58837_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
[junit4] 2> 869078 INFO (zkCallback-1365-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 869078 INFO (zkCallback-1366-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 869078 INFO (zkCallback-1364-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 869729 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 869730 INFO (zkCallback-1365-thread-1) [ ] o.a.s.c.c.ZkStateReader A collections change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections, has occurred - updating...
[junit4] 2> 869730 INFO (zkCallback-1366-thread-1) [ ] o.a.s.c.c.ZkStateReader A collections change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections, has occurred - updating...
[junit4] 2> 869730 INFO (zkCallback-1364-thread-1) [ ] o.a.s.c.c.ZkStateReader A collections change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections, has occurred - updating...
[junit4] 2> 869736 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /collections/collection1/leader_elect/shard1/election/94820801340833795-node1_core1-n_0000000000
[junit4] 2> 869737 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leaders/shard1
[junit4] 2> 869740 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node
[junit4] 2> 869744 INFO (OverseerStateUpdate-94820801340833796-127.0.0.1:58837_solr-n_0000000000) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "base_url":"http://node1/solr/",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "core":"core1"} current state version: 1
[junit4] 2> 869744 INFO (OverseerStateUpdate-94820801340833796-127.0.0.1:58837_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=null message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "base_url":"http://node1/solr/",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "core":"core1"}
[junit4] 2> 869745 INFO (OverseerStateUpdate-94820801340833796-127.0.0.1:58837_solr-n_0000000000) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "base_url":"http://node1/solr/",
[junit4] 2> "core":"core1"} current state version: 1
[junit4] 2> 869746 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 869746 INFO (OverseerStateUpdate-94820801340833796-127.0.0.1:58837_solr-n_0000000000) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"} current state version: 1
[junit4] 2> 869748 INFO (OverseerStateUpdate-94820801340833796-127.0.0.1:58837_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 869748 INFO (OverseerStateUpdate-94820801340833796-127.0.0.1:58837_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator shard=shard1 is already registered
[junit4] 2> 869750 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /collections/collection1/leader_elect/shard1/election/94820801340833795-node1_core1-n_0000000001
[junit4] 2> 869750 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.LeaderElector was going to be leader /collections/collection1/leader_elect/shard1/election/94820801340833795-node1_core1-n_0000000001 , seq(0) /collections/collection1/leader_elect/shard1/election/94820801340833795-node1_core1-n_0000000000
[junit4] 2> 869882 INFO (zkCallback-1364-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 869882 INFO (zkCallback-1366-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 869882 INFO (zkCallback-1365-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 870384 INFO (OverseerCollectionConfigSetProcessor-94820801340833796-127.0.0.1:58837_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor
[junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:353)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:350)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:350)
[junit4] 2> at org.apache.solr.cloud.OverseerTaskProcessor.amILeader(OverseerTaskProcessor.java:355)
[junit4] 2> at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:172)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 870384 INFO (OverseerCollectionConfigSetProcessor-94820801340833796-127.0.0.1:58837_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor According to ZK I (id=94820801340833796-127.0.0.1:58837_solr-n_0000000000) am no longer a leader.
[junit4] 2> 871258 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 871263 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /collections/collection1/leader_elect/shard1/election/94820801340833795-node1_core1-n_0000000002
[junit4] 2> 871263 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.LeaderElector was going to be leader /collections/collection1/leader_elect/shard1/election/94820801340833795-node1_core1-n_0000000002 , seq(0) /collections/collection1/leader_elect/shard1/election/94820801340833795-node1_core1-n_0000000000
[junit4] 2> 871264 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 871265 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 871267 INFO (zkCallback-1368-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@3087bc29 name:ZooKeeperConnection Watcher:127.0.0.1:58837/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 871267 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 871268 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 871268 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.Overseer Overseer (id=94820801340833796-127.0.0.1:58837_solr-n_0000000000) closing
[junit4] 2> 871268 INFO (OverseerStateUpdate-94820801340833796-127.0.0.1:58837_solr-n_0000000000) [ ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:58837_solr
[junit4] 2> 871268 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=600000&connTimeout=60000&retry=true
[junit4] 2> 871269 ERROR (OverseerExitThread) [ ] o.a.s.c.Overseer could not read the data
[junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:353)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:350)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:350)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:304)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:87)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$2.run(Overseer.java:265)
[junit4] 2> 871274 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 871276 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/94820801340833797-127.0.0.1:58837_solr-n_0000000001
[junit4] 2> 871276 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:58837_solr
[junit4] 2> 871276 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
[junit4] 2> 871278 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.Overseer Overseer (id=94820801340833797-127.0.0.1:58837_solr-n_0000000001) starting
[junit4] 2> 871288 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 871288 INFO (OverseerCollectionConfigSetProcessor-94820801340833797-127.0.0.1:58837_solr-n_0000000001) [ ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
[junit4] 2> 871288 INFO (OverseerStateUpdate-94820801340833797-127.0.0.1:58837_solr-n_0000000001) [ ] o.a.s.c.Overseer Starting to work on the main queue
[junit4] 2> 871290 INFO (OverseerStateUpdate-94820801340833797-127.0.0.1:58837_solr-n_0000000001) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 871293 INFO (OverseerStateUpdate-94820801340833797-127.0.0.1:58837_solr-n_0000000001) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"} current state version: 2
[junit4] 2> 871293 INFO (OverseerStateUpdate-94820801340833797-127.0.0.1:58837_solr-n_0000000001) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 871293 INFO (OverseerStateUpdate-94820801340833797-127.0.0.1:58837_solr-n_0000000001) [ ] o.a.s.c.o.ReplicaMutator shard=shard1 is already registered
[junit4] 2> 871295 INFO (zkCallback-1365-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 871295 INFO (zkCallback-1364-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 871295 INFO (zkCallback-1368-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 871365 INFO (OverseerStateUpdate-94820801340833797-127.0.0.1:58837_solr-n_0000000001) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"deletecore",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "collection":"collection1"} current state version: 3
[junit4] 2> 871502 INFO (OverseerStateUpdate-94820801340833797-127.0.0.1:58837_solr-n_0000000001) [ ] o.a.s.c.o.ZkStateWriter going to delete_collection /collections/collection1/state.json
[junit4] 2> 871509 INFO (zkCallback-1365-thread-1) [ ] o.a.s.c.c.ZkStateReader A collections change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections, has occurred - updating...
[junit4] 2> 871509 INFO (zkCallback-1368-thread-1) [ ] o.a.s.c.c.ZkStateReader A collections change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections, has occurred - updating...
[junit4] 2> 871509 INFO (zkCallback-1364-thread-1) [ ] o.a.s.c.c.ZkStateReader A collections change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections, has occurred - updating...
[junit4] 2> 871509 INFO (zkCallback-1368-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 871509 INFO (zkCallback-1365-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 871509 INFO (zkCallback-1364-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 872163 INFO (zkCallback-1365-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 872163 INFO (zkCallback-1368-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 872163 INFO (zkCallback-1364-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 874126 WARN (zkCallback-1365-thread-1) [ ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 874128 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:58837 58837
[junit4] 2> 876521 INFO (OverseerCollectionConfigSetProcessor-94820801340833797-127.0.0.1:58837_solr-n_0000000001) [ ] o.a.s.c.OverseerTaskProcessor
[junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:353)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:350)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:350)
[junit4] 2> at org.apache.solr.cloud.OverseerTaskProcessor.amILeader(OverseerTaskProcessor.java:355)
[junit4] 2> at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:172)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 876521 INFO (OverseerCollectionConfigSetProcessor-94820801340833797-127.0.0.1:58837_solr-n_0000000001) [ ] o.a.s.c.OverseerTaskProcessor According to ZK I (id=94820801340833797-127.0.0.1:58837_solr-n_0000000001) am no longer a leader.
[junit4] 2> 877074 INFO (Thread-1484) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:58837 58837
[junit4] 2> 877075 WARN (Thread-1484) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 4 /solr/aliases.json
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 4 /solr/clusterstate.json
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 4 /solr/live_nodes
[junit4] 2> 4 /solr/collections
[junit4] 2> 2 /solr/overseer/queue
[junit4] 2> 2 /solr/overseer/collection-queue-work
[junit4] 2> 2 /solr/overseer/queue-work
[junit4] 2>
[junit4] 2> 877077 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testOverseerFailure
[junit4] 2> 877077 INFO (TEST-OverseerTest.testOverseerFailure-seed#[A8631044D0ECD97]) [ ] o.a.s.c.Overseer Overseer (id=94820801340833797-127.0.0.1:58837_solr-n_0000000001) closing
[junit4] 2> 877077 INFO (OverseerStateUpdate-94820801340833797-127.0.0.1:58837_solr-n_0000000001) [ ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:58837_solr
[junit4] 2> 877078 ERROR (OverseerExitThread) [ ] o.a.s.c.Overseer could not read the data
[junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:353)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:350)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:350)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:304)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:87)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$2.run(Overseer.java:265)
[junit4] 2> 877081 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testPlaceholders
[junit4] 2> 877082 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 877082 INFO (Thread-1485) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 877082 INFO (Thread-1485) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 877220 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.ZkTestServer start zk server on port:58875
[junit4] 2> 877220 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 877222 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 877227 INFO (zkCallback-1370-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1a5a12fc name:ZooKeeperConnection Watcher:127.0.0.1:58875/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 877227 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 877227 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 877227 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 877229 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 877230 INFO (zkCallback-1371-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@9ed7110 name:ZooKeeperConnection Watcher:127.0.0.1:58875 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 877230 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 877230 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 877234 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 877237 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 877240 INFO (zkCallback-1372-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@797e6efb name:ZooKeeperConnection Watcher:127.0.0.1:58875 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 877240 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 877240 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 877240 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /solr
[junit4] 2> 877246 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
[junit4] 2> 877247 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections
[junit4] 2> 877250 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
[junit4] 2> 877251 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
[junit4] 2> 877254 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /security.json
[junit4] 2> 877256 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 877258 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 877260 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 877261 INFO (zkCallback-1373-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@2e57ec46 name:ZooKeeperConnection Watcher:127.0.0.1:58875/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 877263 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 877263 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 877264 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 877268 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/node1
[junit4] 2> 877270 INFO (zkCallback-1370-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 0)
[junit4] 2> 877270 INFO (zkCallback-1373-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 0)
[junit4] 2> 877270 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 877272 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 877274 INFO (zkCallback-1374-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@6ab97940 name:ZooKeeperConnection Watcher:127.0.0.1:58875/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 877274 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 877274 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 877274 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=600000&connTimeout=60000&retry=true
[junit4] 2> 877279 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
[junit4] 2> 877281 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
[junit4] 2> 877282 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 877284 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/94820801965457412-127.0.0.1:58875_solr-n_0000000000
[junit4] 2> 877284 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:58875_solr
[junit4] 2> 877284 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
[junit4] 2> 877286 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.Overseer Overseer (id=94820801965457412-127.0.0.1:58875_solr-n_0000000000) starting
[junit4] 2> 877288 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
[junit4] 2> 877291 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
[junit4] 2> 877293 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
[junit4] 2> 877296 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
[junit4] 2> 877299 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
[junit4] 2> 877302 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
[junit4] 2> 877306 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 877308 INFO (OverseerCollectionConfigSetProcessor-94820801965457412-127.0.0.1:58875_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
[junit4] 2> 877308 INFO (OverseerStateUpdate-94820801965457412-127.0.0.1:58875_solr-n_0000000000) [ ] o.a.s.c.Overseer Starting to work on the main queue
[junit4] 2> 877308 INFO (OverseerStateUpdate-94820801965457412-127.0.0.1:58875_solr-n_0000000000) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 877313 INFO (OverseerStateUpdate-94820801965457412-127.0.0.1:58875_solr-n_0000000000) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"12",
[junit4] 2> "base_url":"http://node1/solr/"} current state version: 0
[junit4] 2> 877313 INFO (OverseerStateUpdate-94820801965457412-127.0.0.1:58875_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=12 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"12",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 877313 INFO (OverseerStateUpdate-94820801965457412-127.0.0.1:58875_solr-n_0000000000) [ ] o.a.s.c.o.ClusterStateMutator building a new cName: collection1
[junit4] 2> 877313 INFO (OverseerStateUpdate-94820801965457412-127.0.0.1:58875_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard2
[junit4] 2> 877315 INFO (zkCallback-1370-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 877315 INFO (zkCallback-1374-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 877315 INFO (zkCallback-1373-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 878028 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 878030 INFO (zkCallback-1370-thread-1) [ ] o.a.s.c.c.ZkStateReader A collections change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections, has occurred - updating...
[junit4] 2> 878030 INFO (zkCallback-1373-thread-1) [ ] o.a.s.c.c.ZkStateReader A collections change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections, has occurred - updating...
[junit4] 2> 878030 INFO (zkCallback-1374-thread-1) [ ] o.a.s.c.c.ZkStateReader A collections change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections, has occurred - updating...
[junit4] 2> 878034 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /collections/collection1/leader_elect/shard2/election/94820801965457411-node1_core1-n_0000000000
[junit4] 2> 878035 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/leaders/shard2
[junit4] 2> 878040 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node
[junit4] 2> 878042 INFO (OverseerStateUpdate-94820801965457412-127.0.0.1:58875_solr-n_0000000000) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "shard":"shard2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "base_url":"http://node1/solr/",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "core":"core1"} current state version: 1
[junit4] 2> 878044 INFO (OverseerStateUpdate-94820801965457412-127.0.0.1:58875_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=null message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "shard":"shard2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "base_url":"http://node1/solr/",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "core":"core1"}
[junit4] 2> 878047 INFO (zkCallback-1370-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 878047 INFO (zkCallback-1373-thread-1) [ ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 880187 INFO (OverseerCollectionConfigSetProcessor-94820801965457412-127.0.0.1:58875_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor
[junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:353)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:350)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:350)
[junit4] 2> at org.apache.solr.cloud.OverseerTaskProcessor.amILeader(OverseerTaskProcessor.java:355)
[junit4] 2> at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:172)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 880187 INFO (OverseerCollectionConfigSetProcessor-94820801965457412-127.0.0.1:58875_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor According to ZK I (id=94820801965457412-127.0.0.1:58875_solr-n_0000000000) am no longer a leader.
[junit4] 2> 880204 WARN (OverseerStateUpdate-94820801965457412-127.0.0.1:58875_solr-n_0000000000) [ ] o.a.s.c.Overseer Solr cannot talk to ZK, exiting Overseer main queue loop
[junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer/queue
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:336)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:333)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:333)
[junit4] 2> at org.apache.solr.cloud.DistributedQueue.fetchZkChildren(DistributedQueue.java:310)
[junit4] 2> at org.apache.solr.cloud.DistributedQueue.firstChild(DistributedQueue.java:289)
[junit4] 2> at org.apache.solr.cloud.DistributedQueue.firstElement(DistributedQueue.java:358)
[junit4] 2> at org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:165)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:233)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 880204 INFO (OverseerStateUpdate-94820801965457412-127.0.0.1:58875_solr-n_0000000000) [ ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:58875_solr
[junit4] 2> 880205 ERROR (OverseerExitThread) [ ] o.a.s.c.Overseer could not read the data
[junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:353)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:350)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:350)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:304)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:87)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$2.run(Overseer.java:265)
[junit4] 2> 880206 WARN (zkCallback-1373-thread-1) [ ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 880209 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:58875 58875
[junit4] 2> 883448 INFO (Thread-1485) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:58875 58875
[junit4] 2> 883451 WARN (Thread-1485) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 3 /solr/aliases.json
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 3 /solr/clusterstate.json
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 3 /solr/live_nodes
[junit4] 2> 3 /solr/collections
[junit4] 2>
[junit4] 2> 883451 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testPlaceholders
[junit4] 2> 883451 INFO (TEST-OverseerTest.testPlaceholders-seed#[A8631044D0ECD97]) [ ] o.a.s.c.Overseer Overseer (id=94820801965457412-127.0.0.1:58875_solr-n_0000000000) closing
[junit4] 2> 883452 INFO (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[A8631044D0ECD97]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testExternalClusterStateChangeBehavior
[junit4] 2> 883454 INFO (TEST-OverseerTest.testExternalClusterStateChangeBehavior-seed#[A8631044D0ECD97]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 883455 INFO (Thread-1486) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 883455 INFO (Thread-1486) [ ]
[...truncated too long message...]
INFO (OverseerStateUpdate-94820806997573635-127.0.0.1:59314_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=null message={
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1/solr",
[junit4] 2> "node_name":"node1",
[junit4] 2> "shard":"s1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "core":"core2",
[junit4] 2> "roles":"",
[junit4] 2> "state":"recovering"}
[junit4] 2> 953980 INFO (zkCallback-1451-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 953980 INFO (zkCallback-1454-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 953981 INFO (OverseerStateUpdate-94820806997573635-127.0.0.1:59314_solr-n_0000000000) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1/solr",
[junit4] 2> "node_name":"node1",
[junit4] 2> "shard":"s1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "core":"core3",
[junit4] 2> "roles":"",
[junit4] 2> "state":"recovering"} current state version: 2
[junit4] 2> 953981 INFO (OverseerStateUpdate-94820806997573635-127.0.0.1:59314_solr-n_0000000000) [ ] o.a.s.c.o.ReplicaMutator Update state numShards=null message={
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1/solr",
[junit4] 2> "node_name":"node1",
[junit4] 2> "shard":"s1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "core":"core3",
[junit4] 2> "roles":"",
[junit4] 2> "state":"recovering"}
[junit4] 2> 954052 INFO (zkCallback-1454-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 954052 INFO (zkCallback-1451-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 954116 INFO (TEST-OverseerTest.testReplay-seed#[A8631044D0ECD97]) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:59314 59314
[junit4] 2> 955353 INFO (OverseerCollectionConfigSetProcessor-94820806997573635-127.0.0.1:59314_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor
[junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:353)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:350)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:350)
[junit4] 2> at org.apache.solr.cloud.OverseerTaskProcessor.amILeader(OverseerTaskProcessor.java:355)
[junit4] 2> at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:172)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 955353 INFO (OverseerCollectionConfigSetProcessor-94820806997573635-127.0.0.1:59314_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor According to ZK I (id=94820806997573635-127.0.0.1:59314_solr-n_0000000000) am no longer a leader.
[junit4] 2> 955665 INFO (Thread-1494) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:59314 59314
[junit4] 2> 955665 WARN (Thread-1494) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 2 /solr/aliases.json
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 2 /solr/clusterstate.json
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 2 /solr/live_nodes
[junit4] 2> 2 /solr/collections
[junit4] 2>
[junit4] 2> 955665 INFO (TEST-OverseerTest.testReplay-seed#[A8631044D0ECD97]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testReplay
[junit4] 2> 955665 INFO (TEST-OverseerTest.testReplay-seed#[A8631044D0ECD97]) [ ] o.a.s.c.Overseer Overseer (id=94820806997573635-127.0.0.1:59314_solr-n_0000000000) closing
[junit4] 2> 955665 INFO (OverseerStateUpdate-94820806997573635-127.0.0.1:59314_solr-n_0000000000) [ ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:59314_solr
[junit4] 2> 955667 ERROR (OverseerExitThread) [ ] o.a.s.c.Overseer could not read the data
[junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:353)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:350)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:350)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:304)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:87)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$2.run(Overseer.java:265)
[junit4] 2> 957731 INFO (SUITE-OverseerTest-seed#[A8631044D0ECD97]-worker) [ ] o.a.s.SolrTestCaseJ4 ###deleteCore
[junit4] 2> NOTE: leaving temporary files on disk at: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.OverseerTest_A8631044D0ECD97-001
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene60): {}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=true,coord=yes): {}, locale=es_PE, timezone=SystemV/YST9YDT
[junit4] 2> NOTE: Windows 7 6.1 amd64/Oracle Corporation 1.8.0_66 (64-bit)/cpus=3,threads=1,free=36174160,total=231890944
[junit4] 2> NOTE: All tests run in this JVM: [TestLRUStatsCache, TestCollapseQParserPlugin, TestDistribDocBasedVersion, TestMinMaxOnMultiValuedField, TestCloudPivotFacet, GraphQueryTest, TestCustomSort, HttpPartitionTest, TestCrossCoreJoin, DOMUtilTest, TestSchemaResource, TestManagedStopFilterFactory, TestRestManager, TestManagedSchema, TestExtendedDismaxParser, RegexBoostProcessorTest, StandardRequestHandlerTest, PreAnalyzedFieldTest, ActionThrottleTest, PolyFieldTest, AsyncMigrateRouteKeyTest, DirectUpdateHandlerTest, TestReloadDeadlock, TestMergePolicyConfig, HighlighterTest, TestDynamicLoading, TestJmxMonitoredMap, JSONWriterTest, AliasIntegrationTest, RankQueryTest, VersionInfoTest, TestExpandComponent, UpdateParamsTest, SpatialRPTFieldTypeTest, TestCloudManagedSchemaConcurrent, LeaderElectionTest, TestLeaderElectionZkExpiry, HardAutoCommitTest, HdfsChaosMonkeySafeLeaderTest, CdcrUpdateLogTest, BadCopyFieldTest, JavabinLoaderTest, TestSolrCloudWithKerberosAlt, CloneFieldUpdateProcessorFactoryTest, TestReqParamsAPI, DocValuesTest, TestManagedSchemaFieldTypeResource, SolrRequestParserTest, ClusterStateTest, TestConfigSets, TestStressReorder, TestCloudSchemaless, SharedFSAutoReplicaFailoverUtilsTest, TestArbitraryIndexDir, TestConfigSetsAPIExclusivity, CdcrReplicationDistributedZkTest, TestXIncludeConfig, SampleTest, DocumentBuilderTest, UUIDUpdateProcessorFallbackTest, PingRequestHandlerTest, OverseerTest]
[junit4] Completed [203/555] on J0 in 97.74s, 12 tests, 1 failure, 1 skipped <<< FAILURES!
[...truncated 1142 lines...]
BUILD FAILED
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:775: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:719: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:59: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build.xml:233: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\common-build.xml:516: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:1432: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:987: There were test failures: 555 suites (8 ignored), 2200 tests, 1 failure, 115 ignored (77 assumptions) [seed: A8631044D0ECD97]
Total time: 71 minutes 9 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any