You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/05/07 14:02:10 UTC

[JENKINS] Lucene-Solr-4.x-Windows (64bit/jdk1.7.0_55) - Build # 3936 - Failure!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Windows/3936/
Java: 64bit/jdk1.7.0_55 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC

1 tests failed.
REGRESSION:  org.apache.solr.cloud.OverseerTest.testOverseerFailure

Error Message:
Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed

Stack Trace:
org.apache.solr.common.SolrException: Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed
	at __randomizedtesting.SeedInfo.seed([9DAA2E11012A1506:99A2A1E2138FFA27]:0)
	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:136)
	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:142)
	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:110)
	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:134)
	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:267)
	at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:155)
	at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:661)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.common.SolrException: org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1
	at org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:40)
	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:123)
	... 47 more
Caused by: org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
	at org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:432)
	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:429)
	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:386)
	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:373)
	at org.apache.solr.cloud.ShardLeaderElectionContextBase$1.execute(ElectionContext.java:128)
	at org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:34)
	... 48 more




Build Log:
[...truncated 11394 lines...]
   [junit4] Suite: org.apache.solr.cloud.OverseerTest
   [junit4]   2> Creating dataDir: C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0\.\solr.cloud.OverseerTest-9DAA2E11012A1506-001\init-core-data-001
   [junit4]   2> 2665436 T7636 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (true)
   [junit4]   2> 2665436 T7636 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> 2665436 T7636 oas.SolrTestCaseJ4.initCore ####initCore end
   [junit4]   2> 2665443 T7636 oas.SolrTestCaseJ4.setUp ###Starting testReplay
   [junit4]   2> 2665446 T7636 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2665447 T7637 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 2665554 T7636 oasc.ZkTestServer.run start zk server on port:55193
   [junit4]   2> 2665555 T7636 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2665572 T7643 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@38733ab8 name:ZooKeeperConnection Watcher:127.0.0.1:55193/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2665573 T7636 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2665575 T7636 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2665577 T7645 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2236ff78 name:ZooKeeperConnection Watcher:127.0.0.1:55193 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2665578 T7636 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2665585 T7638 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145d676a6bc0001, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 2665588 T7636 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2665592 T7647 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7371d917 name:ZooKeeperConnection Watcher:127.0.0.1:55193 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2665593 T7636 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2665594 T7636 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 2665601 T7636 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 2665602 T7638 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145d676a6bc0002, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 2665606 T7636 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 2665610 T7636 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 2665614 T7636 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2665619 T7636 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 2665626 T7636 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2665628 T7649 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@65e3ac11 name:ZooKeeperConnection Watcher:127.0.0.1:55193/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2665628 T7636 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2665629 T7636 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 2665632 T7636 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 2665640 T7636 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:55193_solr
   [junit4]   2> 2665640 T7636 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 2665643 T7636 oasc.Overseer.start Overseer (id=91715172522786819-127.0.0.1:55193_solr-n_0000000000) starting
   [junit4]   2> 2665646 T7636 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 2665653 T7636 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 2665658 T7636 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 2665665 T7636 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 2665672 T7636 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 2665681 T7651 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 2665683 T7651 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes at 127.0.0.1:55193_solr
   [junit4]   2> 2665689 T7650 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2665690 T7650 oasc.Overseer$ClusterStateUpdater.run Replaying operations from work queue.
   [junit4]   2> 2665691 T7650 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "shard":"s1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "roles":"",
   [junit4]   2> 	  "state":"recovering"}
   [junit4]   2> 2665693 T7643 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
   [junit4]   2> 2665699 T7650 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "shard":"s1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core":"core2",
   [junit4]   2> 	  "roles":"",
   [junit4]   2> 	  "state":"recovering"}
   [junit4]   2> 2665700 T7643 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
   [junit4]   2> 2665704 T7650 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 2665707 T7650 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2665708 T7650 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "shard":"s1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core":"core3",
   [junit4]   2> 	  "roles":"",
   [junit4]   2> 	  "state":"recovering"}
   [junit4]   2> 2665713 T7649 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2665715 T7643 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
   [junit4]   2> 2665797 T7638 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145d676a6bc0003, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 2665799 T7638 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145d676a6bc0000, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 2665799 T7636 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:55193 55193
   [junit4]   2> 2668204 T7637 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:55193 55193
   [junit4]   2> 2668206 T7636 oas.SolrTestCaseJ4.tearDown ###Ending testReplay
   [junit4]   2> 2668207 T7650 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:55193_solr
   [junit4]   2> 2668209 T7652 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:302)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:291)
   [junit4]   2> 
   [junit4]   2> 2668216 T7636 oas.SolrTestCaseJ4.setUp ###Starting testOverseerFailure
   [junit4]   2> 2668219 T7636 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2668220 T7653 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 2668327 T7636 oasc.ZkTestServer.run start zk server on port:55208
   [junit4]   2> 2668329 T7636 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2668336 T7659 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2b704d2f name:ZooKeeperConnection Watcher:127.0.0.1:55208 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2668337 T7636 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2668344 T7636 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2668349 T7661 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2a34648a name:ZooKeeperConnection Watcher:127.0.0.1:55208 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2668350 T7636 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2668350 T7636 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 2668358 T7654 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145d676b1900001, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 2668359 T7636 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2668363 T7663 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@48d923be name:ZooKeeperConnection Watcher:127.0.0.1:55208/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2668363 T7636 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2668363 T7636 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 2668368 T7636 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 2668375 T7636 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 2668380 T7636 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2668387 T7636 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2668391 T7665 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7fd1b1d3 name:ZooKeeperConnection Watcher:127.0.0.1:55208/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2668391 T7636 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2668393 T7636 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2668399 T7636 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 2668404 T7663 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 2668405 T7665 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 2668405 T7636 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2668408 T7667 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@60e5ae23 name:ZooKeeperConnection Watcher:127.0.0.1:55208/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2668409 T7636 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2668410 T7636 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 2668415 T7636 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 2668425 T7636 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:55208_solr
   [junit4]   2> 2668426 T7636 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 2668430 T7636 oasc.Overseer.start Overseer (id=91715172704452612-127.0.0.1:55208_solr-n_0000000000) starting
   [junit4]   2> 2668435 T7636 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 2668442 T7636 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 2668449 T7636 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 2668455 T7636 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 2668460 T7636 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 2668467 T7636 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 2668479 T7669 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 2668480 T7669 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes at 127.0.0.1:55208_solr
   [junit4]   2> 2668480 T7668 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 2669551 T7667 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2669555 T7668 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2669556 T7668 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 2669557 T7668 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1]
   [junit4]   2> 2669557 T7668 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 2669563 T7667 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2669565 T7663 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2669565 T7665 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2670085 T7636 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 2670109 T7636 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 2670124 T7667 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2670126 T7636 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2670127 T7668 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2670135 T7667 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2670135 T7636 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 2670140 T7668 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 2670141 T7668 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 2670151 T7667 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2670263 T7665 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2670263 T7663 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2671367 T7636 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 2671380 T7636 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2671384 T7671 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@26394876 name:ZooKeeperConnection Watcher:127.0.0.1:55208/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2671384 T7636 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2671385 T7668 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:55208_solr
   [junit4]   2> 2671388 T7672 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:302)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:291)
   [junit4]   2> 
   [junit4]   2> 2671393 T7636 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:55208_solr
   [junit4]   2> 2671393 T7636 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 2671396 T7636 oasc.Overseer.start Overseer (id=91715172704452613-127.0.0.1:55208_solr-n_0000000001) starting
   [junit4]   2> 2671421 T7674 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 2671422 T7674 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes at 127.0.0.1:55208_solr
   [junit4]   2> 2671423 T7673 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 2671427 T7673 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2671428 T7673 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 2671428 T7673 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 2671434 T7671 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2671436 T7665 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2671436 T7663 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2671491 T7671 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2671491 T7636 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 2671496 T7673 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2671509 T7636 oasc.LeaderElector.checkIfIamLeader WARN Failed setting watch org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /collections/collection1/leader_elect/shard1/election/91715172704452611-node1_core1-n_0000000002
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:127)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:267)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:155)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:661)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   [junit4]   2> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> 	at java.lang.reflect.Method.invoke(Method.java:606)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
   [junit4]   2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
   [junit4]   2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 2671515 T7673 oasc.SolrException.log ERROR Problem cleaning up collection in zk:collection1:org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 		at org.apache.zookeeper.KeeperException.create(KeeperException.java:125)
   [junit4]   2> 		at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 		at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:177)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:174)
   [junit4]   2> 		at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.delete(SolrZkClient.java:174)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.clean(SolrZkClient.java:631)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.clean(SolrZkClient.java:626)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.clean(SolrZkClient.java:626)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.clean(SolrZkClient.java:626)
   [junit4]   2> 		at org.apache.solr.cloud.Overseer$ClusterStateUpdater.removeCore(Overseer.java:1069)
   [junit4]   2> 		at org.apache.solr.cloud.Overseer$ClusterStateUpdater.processMessage(Overseer.java:342)
   [junit4]   2> 		at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:241)
   [junit4]   2> 		at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 	
   [junit4]   2> 2671516 T7636 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 2671516 T7665 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 2671522 T7671 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2671532 T7671 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2671534 T7673 oasc.Overseer$ClusterStateUpdater.setShardLeader ERROR Could not mark shard leader for non existing collection:collection1
   [junit4]   2> 2671535 T7665 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 2671542 T7671 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2671550 T7671 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2671553 T7673 oasc.Overseer$ClusterStateUpdater.setShardLeader ERROR Could not mark shard leader for non existing collection:collection1
   [junit4]   2> 2671559 T7671 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2671659 T7663 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2671659 T7665 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2672488 T7636 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 2673456 T7636 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 2674422 T7636 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 2675388 T7636 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 2676355 T7636 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 2677323 T7636 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 2678289 T7636 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 2679255 T7636 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 2680221 T7636 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 2681143 T7636 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 2682021 T7636 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 2682891 T7636 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 2683761 T7636 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 2684632 T7636 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 2685502 T7636 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 2685509 T7665 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 2685510 T7663 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 2685513 T7654 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145d676b1900005, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 2685517 T7654 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145d676b1900002, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 2685518 T7636 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:55208 55208
   [junit4]   2> 2687466 T7653 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:55208 55208
   [junit4]   2> 2687469 T7636 oas.SolrTestCaseJ4.tearDown ###Ending testOverseerFailure
   [junit4]   2> 2687469 T7673 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:55208_solr
   [junit4]   2> 2687475 T7675 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:302)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:291)
   [junit4]   2> 
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=OverseerTest -Dtests.method=testOverseerFailure -Dtests.seed=9DAA2E11012A1506 -Dtests.slow=true -Dtests.locale=pt_PT -Dtests.timezone=MST -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   19.3s | OverseerTest.testOverseerFailure <<<
   [junit4]    > Throwable #1: org.apache.solr.common.SolrException: Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([9DAA2E11012A1506:99A2A1E2138FFA27]:0)
   [junit4]    > 	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:136)
   [junit4]    > 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:142)
   [junit4]    > 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:110)
   [junit4]    > 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:134)
   [junit4]    > 	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:267)
   [junit4]    > 	at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:155)
   [junit4]    > 	at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:661)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]    > Caused by: org.apache.solr.common.SolrException: org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1
   [junit4]    > 	at org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:40)
   [junit4]    > 	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:123)
   [junit4]    > 	... 47 more
   [junit4]    > Caused by: org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1
   [junit4]    > 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
   [junit4]    > 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]    > 	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:432)
   [junit4]    > 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:429)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:386)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:373)
   [junit4]    > 	at org.apache.solr.cloud.ShardLeaderElectionContextBase$1.execute(ElectionContext.java:128)
   [junit4]    > 	at org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:34)
   [junit4]    > 	... 48 more
   [junit4]   2> 2687493 T7636 oas.SolrTestCaseJ4.setUp ###Starting testShardAssignment
   [junit4]   2> 2687497 T7636 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2687498 T7676 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 2687584 T7636 oasc.ZkTestServer.run start zk server on port:55229
   [junit4]   2> 2687585 T7636 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2687590 T7682 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2893cb38 name:ZooKeeperConnection Watcher:127.0.0.1:55229 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2687590 T7636 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2687592 T7677 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145d676fcdd0000, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 2687593 T7636 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2687595 T7684 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@16c9a6d2 name:ZooKeeperConnection Watcher:127.0.0.1:55229 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2687596 T7636 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2687596 T7636 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 2687600 T7677 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145d676fcdd0001, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 2687601 T7636 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2687603 T7686 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@519bb3d5 name:ZooKeeperConnection Watcher:127.0.0.1:55229/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2687603 T7636 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2687603 T7636 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 2687607 T7636 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2687608 T7688 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6a62d44b name:ZooKeeperConnection Watcher:127.0.0.1:55229/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2687608 T7636 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2687609 T7636 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 2687612 T7636 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 2687621 T7636 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:55229_solr
   [junit4]   2> 2687621 T7636 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 2687626 T7636 oasc.Overseer.start Overseer (id=91715173967790083-127.0.0.1:55229_solr-n_0000000000) starting
   [junit4]   2> 2687630 T7636 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 2687635 T7636 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 2687644 T7636 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 2687654 T7636 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 2687659 T7636 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 2687666 T7636 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 2687675 T7690 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 2687676 T7636 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 2687677 T7690 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes at 127.0.0.1:55229_solr
   [junit4]   2> 2687680 T7636 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 2687681 T7689 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 2687684 T7636 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2687689 T7636 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2687692 T7692 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4d95912d name:ZooKeeperConnection Watcher:127.0.0.1:55229/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2687692 T7636 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2687694 T7636 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2687699 T7636 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1
   [junit4]   2> 2687703 T7686 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 2687704 T7692 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 2687707 T7688 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2687710 T7689 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2687710 T7689 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 2687710 T7689 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
   [junit4]   2> 2687710 T7689 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
   [junit4]   2> 2687716 T7688 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2687718 T7686 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2687719 T7692 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2688139 T7636 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
   [junit4]   2> 2688153 T7636 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
   [junit4]   2> 2688162 T7688 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2688165 T7689 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2688166 T7688 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2688172 T7689 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core2",
   [junit4]   2> 	  "core_node_name":"node2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 2688173 T7689 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 2688173 T7689 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 2688176 T7688 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2688268 T7686 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2688268 T7692 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2688599 T7636 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 2688610 T7636 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 2688617 T7688 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2688617 T7688 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2688617 T7688 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2688620 T7688 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2688621 T7689 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2688626 T7689 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core3",
   [junit4]   2> 	  "core_node_name":"node3",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 2688627 T7689 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 2688627 T7689 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 2688631 T7688 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2688720 T7686 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2688720 T7692 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2689052 T7636 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 2689069 T7636 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 2689079 T7688 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2689079 T7688 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2689079 T7688 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2689082 T7688 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2689083 T7689 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2689091 T7689 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core4",
   [junit4]   2> 	  "core_node_name":"node4",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 2689091 T7689 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 2689091 T7689 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
   [junit4]   2> 2689096 T7688 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2689186 T7686 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2689187 T7692 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2689537 T7636 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
   [junit4]   2> 2689559 T7688 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2689560 T7688 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2689560 T7688 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2689563 T7689 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2689564 T7689 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core5",
   [junit4]   2> 	  "core_node_name":"node5",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 2689564 T7689 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 2689564 T7689 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 2689568 T7688 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2689674 T7692 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2689674 T7686 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2690059 T7636 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 2690074 T7688 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2690075 T7688 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2690075 T7688 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2690078 T7689 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2690079 T7689 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core6",
   [junit4]   2> 	  "core_node_name":"node6",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 2690079 T7689 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 2690079 T7689 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 2690084 T7688 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2690189 T7692 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2690190 T7686 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2690575 T7636 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 2690591 T7677 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145d676fcdd0002, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 2690594 T7692 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 2690596 T7677 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145d676fcdd0004, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 2690601 T7677 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145d676fcdd0003, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 2690601 T7636 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:55229 55229
   [junit4]   2> 2692096 T7692 oasc.LeaderElector$ElectionWatcher.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:303)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> 2692854 T7692 oasc.LeaderElector$ElectionWatcher.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard3/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:303)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> 2692855 T7676 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:55229 55229
   [junit4]   2> 2692859 T7692 oasc.LeaderElector$ElectionWatcher.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:303)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> 2692861 T7636 oas.SolrTestCaseJ4.tearDown ###Ending testShardAssignment
   [junit4]   2> 2692863 T7689 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:55229_solr
   [junit4]   2> 2692868 T7693 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:302)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:291)
   [junit4]   2> 
   [junit4]   2> 2692868 T7636 oas.SolrTestCaseJ4.setUp ###Starting testDoubleAssignment
   [junit4]   2> 2692874 T7636 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2692875 T7694 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 2692976 T7636 oasc.ZkTestServer.run start zk server on port:55247
   [junit4]   2> 2692979 T7636 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2692985 T7700 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3b620481 name:ZooKeeperConnection Watcher:127.0.0.1:55247/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2692986 T7636 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2692988 T7636 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2692991 T7702 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@75b29691 name:ZooKeeperConnection Watcher:127.0.0.1:55247 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2692992 T7636 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2692996 T7695 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145d67711e00001, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 2692997 T7636 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2693001 T7704 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@8f8a09c name:ZooKeeperConnection Watcher:127.0.0.1:55247 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2693002 T7636 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2693002 T7636 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 2693010 T7636 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 2693010 T7695 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145d67711e00002, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 2693016 T7636 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 2693021 T7636 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 2693026 T7636 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2693032 T7636 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2693035 T7706 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5d229566 name:ZooKeeperConnection Watcher:127.0.0.1:55247/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2693036 T7636 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2693038 T7636 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2693049 T7636 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 2693057 T7700 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 2693057 T7706 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 2693059 T7636 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2693062 T7708 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2cdd3c20 name:ZooKeeperConnection Watcher:127.0.0.1:55247/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2693063 T7636 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2693064 T7636 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 2693069 T7636 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 2693079 T7636 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:55247_solr
   [junit4]   2> 2693079 T7636 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 2693085 T7636 oasc.Overseer.start Overseer (id=91715174320308228-127.0.0.1:55247_solr-n_0000000000) starting
   [junit4]   2> 2693091 T7636 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 2693098 T7636 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 2693105 T7636 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 2693111 T7636 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 2693118 T7636 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 2693125 T7636 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 2693139 T7709 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 2693141 T7710 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 2693143 T7710 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes at 127.0.0.1:55247_solr
   [junit4]   2> 2693145 T7708 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2693148 T7709 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2693149 T7709 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 2693149 T7709 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1]
   [junit4]   2> 2693149 T7709 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 2693153 T7708 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2693156 T7706 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2693156 T7700 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2693645 T7636 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 2693669 T7636 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 2693682 T7708 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2693684 T7636 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2693687 T7706 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 2693688 T7700 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 2693688 T7709 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2693692 T7636 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2693695 T7712 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@55c18f59 name:ZooKeeperConnection Watcher:127.0.0.1:55247/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2693696 T7636 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2693698 T7708 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2693699 T7636 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2693704 T7636 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 2693711 T7712 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 2693712 T7700 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 2693716 T7708 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2693717 T7636 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 2693720 T7709 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 2693720 T7709 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 2693728 T7708 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2693732 T7636 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 2693742 T7708 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2693750 T7708 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2693854 T7712 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2693855 T7700 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2693857 T7636 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2693861 T7695 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145d67711e00004, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 2693863 T7712 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 2693864 T7700 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 2693865 T7695 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145d67711e00005, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 2693869 T7695 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145d67711e00000, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 2693869 T7636 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:55247 55247
   [junit4]   2> 2695359 T7709 oasc.Overseer$ClusterStateUpdater.run WARN Solr cannot talk to ZK, exiting Overseer main queue loop org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer/queue-work/qn-0000000002
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.cloud.DistributedQueue.remove(DistributedQueue.java:178)
   [junit4]   2> 	at org.apache.solr.cloud.DistributedQueue.poll(DistributedQueue.java:493)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:267)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 2696122 T7709 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:55247_solr
   [junit4]   2> 2696120 T7694 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:55247 55247
   [junit4]   2> 2696122 T7636 oas.SolrTestCaseJ4.tearDown ###Ending testDoubleAssignment
   [junit4]   2> 2696129 T7713 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKe

[...truncated too long message...]

1 T8023 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 2723220 T8022 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2723324 T8016 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
   [junit4]   2> 2723361 T8011 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145d677875a0000, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 2723362 T8011 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145d677875a0003, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 2723362 T7636 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:55562 55562
   [junit4]   2> 2725596 T8010 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:55562 55562
   [junit4]   2> 2725598 T7636 oas.SolrTestCaseJ4.tearDown ###Ending testStateChange
   [junit4]   2> 2725599 T8023 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:55562_solr
   [junit4]   2> 2725600 T7636 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> 2725600 T7636 oas.SolrTestCaseJ4.initCore ####initCore end
   [junit4]   2> 2725602 T8025 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:302)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:291)
   [junit4]   2> 
   [junit4]   2> 2728570 T7636 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0\.\solr.cloud.OverseerTest-9DAA2E11012A1506-001
   [junit4]   2> NOTE: test params are: codec=Lucene46: {}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=true,coord=no): {}, locale=pt_PT, timezone=MST
   [junit4]   2> NOTE: Windows 7 6.1 amd64/Oracle Corporation 1.7.0_55 (64-bit)/cpus=2,threads=1,free=250758520,total=519438336
   [junit4]   2> NOTE: All tests run in this JVM: [DistributedQueryElevationComponentTest, IndexSchemaTest, SearchHandlerTest, TestFieldSortValues, TestSearcherReuse, RequiredFieldsTest, SliceStateTest, DistributedDebugComponentTest, DocValuesMultiTest, TestManagedSynonymFilterFactory, NumericFieldsTest, XmlUpdateRequestHandlerTest, TestDistributedGrouping, AnalysisAfterCoreReloadTest, TestFastOutputStream, TestPseudoReturnFields, TestIndexSearcher, CoreAdminCreateDiscoverTest, TestSolrDeletionPolicy2, TestRangeQuery, DateMathParserTest, TestLuceneMatchVersion, HdfsDirectoryTest, OpenExchangeRatesOrgProviderTest, TestQuerySenderListener, RecoveryZkTest, TestSolrXml, AutoCommitTest, TestDocBasedVersionConstraints, LukeRequestHandlerTest, TestCloudInspectUtil, QueryEqualityTest, SuggesterWFSTTest, TestNoOpRegenerator, TestSolrXmlPersistence, SystemInfoHandlerTest, TestCSVLoader, SolrIndexConfigTest, HdfsCollectionsAPIDistributedZkTest, IndexBasedSpellCheckerTest, TestConfigSets, BinaryUpdateRequestHandlerTest, HdfsBasicDistributedZkTest, TestStressVersions, TriLevelCompositeIdRoutingTest, SolrRequestParserTest, TestCursorMarkWithoutUniqueKey, TestSolrXmlPersistor, DistributedSuggestComponentTest, TestFieldCollectionResource, TestZkChroot, SchemaVersionSpecificBehaviorTest, TestXIncludeConfig, TestExpandComponent, TestSolrCoreProperties, TestDynamicFieldResource, ConnectionManagerTest, CachingDirectoryFactoryTest, ClusterStateUpdateTest, BJQParserTest, UnloadDistributedZkTest, ZkNodePropsTest, DistributedQueryComponentOptimizationTest, PingRequestHandlerTest, FieldAnalysisRequestHandlerTest, MBeansHandlerTest, TestMaxScoreQueryParser, TestCollationKeyRangeQueries, TestManagedSchema, TestSurroundQueryParser, TestTrie, LeaderElectionIntegrationTest, JsonLoaderTest, PathHierarchyTokenizerFactoryTest, XsltUpdateRequestHandlerTest, TestRealTimeGet, TestAnalyzedSuggestions, TestJmxIntegration, TestReload, UniqFieldsUpdateProcessorFactoryTest, ClusterStateTest, DeleteInactiveReplicaTest, TestSolrXMLSerializer, BlockDirectoryTest, TestManagedStopFilterFactory, TestQueryTypes, TermVectorComponentTest, EnumFieldTest, TestAtomicUpdateErrorCases, DistribDocExpirationUpdateProcessorTest, CurrencyFieldXmlFileTest, TestCustomSort, TestDynamicFieldCollectionResource, TestCopyFieldCollectionResource, TestFieldTypeCollectionResource, TestAnalyzeInfixSuggestions, WordBreakSolrSpellCheckerTest, PreAnalyzedFieldTest, PreAnalyzedUpdateProcessorTest, TestReplicationHandler, DirectUpdateHandlerTest, TestSuggestSpellingConverter, AlternateDirectoryTest, CoreMergeIndexesAdminHandlerTest, SuggesterFSTTest, TestFieldTypeResource, DeleteReplicaTest, TestFileDictionaryLookup, TestFunctionQuery, SolrInfoMBeanTest, CurrencyFieldOpenExchangeTest, TestDistributedSearch, DocValuesTest, SOLR749Test, FullSolrCloudDistribCmdsTest, ZkSolrClientTest, HdfsUnloadDistributedZkTest, TestHighFrequencyDictionaryFactory, TestIBSimilarityFactory, MinimalSchemaTest, ExternalFileFieldSortTest, TestCollectionAPI, TestFastWriter, TestManagedResourceStorage, DeleteShardTest, TestSolrJ, TestWriterPerf, SuggesterTSTTest, DocumentAnalysisRequestHandlerTest, TestHighlightDedupGrouping, SimplePostToolTest, CollectionsAPIAsyncDistributedZkTest, SortByFunctionTest, SuggesterTest, DisMaxRequestHandlerTest, BadIndexSchemaTest, FastVectorHighlighterTest, TestRequestStatusCollectionAPI, TestMultiCoreConfBootstrap, TestCloudManagedSchema, TestCoreDiscovery, TestDocSet, TestWordDelimiterFilterFactory, TestUniqueKeyFieldResource, TestMergePolicyConfig, TestFuzzyAnalyzedSuggestions, SimpleFacetsTest, TestFiltering, TestStressReorder, ShardRoutingCustomTest, TestSimpleQParserPlugin, TestInitQParser, InfoHandlerTest, HdfsLockFactoryTest, TestComponentsName, RAMDirectoryFactoryTest, LoggingHandlerTest, DocValuesMissingTest, TestRecoveryHdfs, TestDocumentBuilder, DefaultValueUpdateProcessorTest, SliceStateUpdateTest, OutputWriterTest, BlockCacheTest, PeerSyncTest, TimeZoneUtilsTest, HardAutoCommitTest, TestArbitraryIndexDir, BasicZkTest, ScriptEngineTest, SyncSliceTest, HdfsBasicDistributedZk2Test, TestShortCircuitedRequests, AtomicUpdatesTest, ZkCLITest, DateFieldTest, TestNRTOpen, SpellPossibilityIteratorTest, DistribCursorPagingTest, FileBasedSpellCheckerTest, TestConfig, DocExpirationUpdateProcessorFactoryTest, BadComponentTest, TestPHPSerializedResponseWriter, TestSolrQueryParserResource, EchoParamsTest, TestShardHandlerFactory, HighlighterTest, TestExceedMaxTermLength, ResponseLogComponentTest, TestReloadAndDeleteDocs, BasicDistributedZkTest, ChaosMonkeySafeLeaderTest, BasicDistributedZk2Test, CollectionsAPIDistributedZkTest, OpenCloseCoreStressTest, OverseerTest]
   [junit4] Completed in 63.16s, 9 tests, 1 error <<< FAILURES!

[...truncated 590 lines...]
BUILD FAILED
C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\build.xml:467: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\build.xml:447: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\build.xml:45: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\extra-targets.xml:37: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build.xml:189: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\common-build.xml:496: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\lucene\common-build.xml:1297: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\lucene\common-build.xml:921: There were test failures: 389 suites, 1630 tests, 1 error, 43 ignored (22 assumptions)

Total time: 105 minutes 33 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_55 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-4.x-Windows (64bit/jdk1.7.0_60-ea-b15) - Build # 3940 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Windows/3940/
Java: 64bit/jdk1.7.0_60-ea-b15 -XX:-UseCompressedOops -XX:+UseG1GC

2 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.lucene.codecs.lucene3x.TestTermInfosReaderIndex

Error Message:
_2.fnm in dir=RAMDirectory@34312b8a lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8218bcb

Stack Trace:
java.io.FileNotFoundException: _2.fnm in dir=RAMDirectory@34312b8a lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8218bcb
	at __randomizedtesting.SeedInfo.seed([18C00777C7D5AB95]:0)
	at org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:579)
	at org.apache.lucene.codecs.lucene3x.PreFlexRWFieldInfosReader.read(PreFlexRWFieldInfosReader.java:45)
	at org.apache.lucene.codecs.lucene3x.TestTermInfosReaderIndex.beforeClass(TestTermInfosReaderIndex.java:100)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:767)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.lucene.codecs.lucene3x.TestTermInfosReaderIndex

Error Message:


Stack Trace:
java.lang.NullPointerException
	at __randomizedtesting.SeedInfo.seed([18C00777C7D5AB95]:0)
	at org.apache.lucene.codecs.lucene3x.TestTermInfosReaderIndex.afterClass(TestTermInfosReaderIndex.java:118)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:790)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 1673 lines...]
   [junit4] Suite: org.apache.lucene.codecs.lucene3x.TestTermInfosReaderIndex
   [junit4]   2> NOTE: test params are: codec=Lucene46: {}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=true,coord=yes): {}, locale=es_UY, timezone=America/Maceio
   [junit4]   2> NOTE: Windows 7 6.1 amd64/Oracle Corporation 1.7.0_60-ea (64-bit)/cpus=2,threads=1,free=282883576,total=535822336
   [junit4]   2> NOTE: All tests run in this JVM: [TestAttributeSource, TestMultiTermConstantScore, TestRollingBuffer, TestRAMDirectory, TestWildcardRandom, TestIndexWriterUnicode, TestMultiDocValues, Before3, Before3, TestSortRandom, TestInPlaceMergeSorter, TestPerFieldDocValuesFormat, TestRegexpRandom2, TestSimilarity, TestCachingTokenFilter, TestAllFilesHaveCodecHeader, TestSimpleAttributeImpl, TestPhrasePrefixQuery, TestIndexWriterMergePolicy, TestTryDelete, TestSortRescorer, TestToken, TestLucene3xPostingsFormat, TestNeverDelete, TestPagedBytes, TestSpansAdvanced, TestCachingCollector, TestLevenshteinAutomata, TestIndexWriterDelete, TestSumDocFreq, TestDeterminizeLexicon, TestSloppyPhraseQuery, TestCrashCausesCorruptIndex, TestDeterminism, TestRamUsageEstimatorOnWildAnimals, TestMergeSchedulerExternal, TestCharFilter, TestPerFieldPostingsFormat2, TestControlledRealTimeReopenThread, Nested1, TestCrash, TestOmitPositions, TestDocumentWriter, TestOmitTf, TestRollingUpdates, TestIndexFileDeleter, TestQueryWrapperFilter, TestFilterDirectory, TestSizeBoundedForceMerge, TestDateTools, TestIndexReaderClose, TestAtomicUpdate, TestCompoundFile, TestCompiledAutomaton, TestSearchForDuplicates, TestQueryBuilder, Test2BDocs, TestPrefixInBooleanQuery, TestDocIdBitSet, Nested1, Nested2, Nested3, TestVersionComparator, Test2BPostingsBytes, TestMultiPhraseQuery, TestIndexCommit, TestMultiFields, TestDocument, TestFilteredQuery, TestIndexWriterOnJRECrash, TestScoreCachingWrappingScorer, TestWorstCaseTestBehavior, TestPositionIncrement, TestDocValuesWithThreads, TestNumericRangeQuery32, TestExternalCodecs, TestScorerPerf, TestLiveFieldValues, TestIsCurrent, TestDocumentsWriterStallControl, TestIntroSorter, TestFieldInfos, TestDocTermOrds, TestSimilarity2, TestTopDocsCollector, TestLongPostings, TestReusableStringReader, Test4GBStoredFields, TestSearchWithThreads, TestTerms, TestPrefixQuery, TestDisjunctionMaxQuery, TestParallelAtomicReader, TestIndexableBinaryStringTools, TestAddIndexes, TestNRTThreads, TestSameTokenSamePosition, TestSameScoresWithThreads, TestLucene3xTermVectorsFormat, TestPForDeltaDocIdSet, TestTermsEnum, TestBooleanQuery, TestBasicOperations, TestMaxTermFrequency, TestDeletionPolicy, TestPayloadExplanations, TestConstants, TestSimilarityBase, TestBroadWord, TestCustomNorms, TestSetOnce, TestFieldReuse, TestNGramPhraseQuery, Nested, Nested2, TestLucene45DocValuesFormat, TestDocumentsWriterDeleteQueue, Test2BSortedDocValues, TestFieldCacheRewriteMethod, TestAutomatonQuery, TestTermsEnum2, TestBinaryTerms, TestTieredMergePolicy, TestForTooMuchCloning, TestPayloadSpans, TestFieldCacheRangeFilter, TestSimpleFSDirectory, Nested, TestDuelingCodecs, TestEliasFanoSequence, TestFastCompressionMode, TestBinaryDocument, TestTopDocsMerge, TestIndexWriterExceptions2, Nested1, Nested2, TestTimeLimitingCollector, TestSort, TestExplanations, TestBooleanMinShouldMatch, TestLucene3xStoredFieldsFormat, TestLucene41StoredFieldsFormat, TestSpanFirstQuery, TestLucene40TermVectorsFormat, TestPositiveScoresOnlyCollector, TestAssertions, TestBagOfPositions, TestSentinelIntSet, TestPerSegmentDeletes, TestComplexExplanations, TestPayloadTermQuery, TestBytesRefHash, TestSegmentTermEnum, TestSpecialOperations, TestFieldCacheTermsFilter, TestByteSlices, TestFieldMaskingSpanQuery, TestIndexWriterNRTIsCurrent, TestShardSearching, TestNRTReaderWithThreads, TestIndexWriterOnDiskFull, TestIndexWriterForceMerge, TestBooleanOr, TestBytesRef, TestCharTermAttributeImpl, TestCodecs, TestDocValuesIndexing, TestFieldValueFilter, TestRamUsageEstimator, TestIndexWriter, TestSearchAfter, TestStressIndexing2, TestPriorityQueue, TestIdentityHashSet, TestLock, TestMinShouldMatch2, TestBooleanScorer, TestFuzzyQuery, TestSimpleExplanationsOfNonMatches, TestFlushByRamOrCountsPolicy, TestFieldCache, TestStressIndexing, TestDocValuesScoring, TestUnicodeUtil, TestParallelTermEnum, TestMathUtil, TestTransactionRollback, TestTermRangeQuery, TestOmitNorms, TestPayloadNearQuery, TestDocIdSet, TestFieldCacheSanityChecker, TestFixedBitSet, TestMixedDocValuesUpdates, TestNewestSegment, TestBoolean2, TestSearch, TestMultiMMap, TestSpanSearchEquivalence, TestFieldsReader, TestWindowsMMap, TestLucene42DocValuesFormat, TestConcurrentMergeScheduler, TestByteBlockPool, TestLucene40PostingsFormat, TestDocValuesFormat, TestIndexWriterReader, TestTermVectorsFormat, TestCompressingTermVectorsFormat, TestIndexWriterMerging, TestFSTs, TestNumericRangeQuery64, TestBytesStore, TestPackedInts, TestIndexWriterWithThreads, TestBackwardsCompatibility, TestGraphTokenizers, TestLookaheadTokenFilter, TestIndexWriterExceptions, TestBlockPostingsFormat3, TestSearcherManager, TestBasics, TestLucene40DocValuesFormat, TestPersistentSnapshotDeletionPolicy, TestStressAdvance, TestNorms, TestIndexWriterCommit, TestDirectoryReaderReopen, TestHighCompressionMode, TestOpenBitSet, TestMinimize, TestBlockPostingsFormat2, TestDocTermOrdsRewriteMethod, TestSnapshotDeletionPolicy, TestLucene40PostingsReader, TestPerFieldPostingsFormat, TestArrayUtil, TestBufferedIndexInput, TestConsistentFieldNumbers, TestWeakIdentityMap, TestSimpleExplanations, TestDirectoryReader, TestDocTermOrdsRangeFilter, TestCompressingStoredFieldsFormat, TestLockFactory, TestSegmentMerger, TestPhraseQuery, TestCollectionUtil, TestDoubleBarrelLRUCache, TestMockAnalyzer, TestNRTCachingDirectory, TestSegmentReader, TestFastDecompressionMode, TestParallelCompositeReader, TestTransactions, TestMultiThreadTermVectors, TestSegmentTermDocs, TestBlockPostingsFormat, TestBitVector, TestReuseDocsEnum, TestSimpleSearchEquivalence, TestCustomSearcherSort, TestPostingsFormat, TestHugeRamFile, TestNumericUtils, TestSpans, TestPostingsOffsets, TestMixedCodecs, TestFlex, TestUTF32ToUTF8, TestSortDocValues, TestSpanExplanations, TestLazyProxSkipping, TestDocsAndPositions, TestBagOfPostings, TestTermVectorsWriter, TestThreadedForceMerge, TestStressNRT, TestIndexInput, TestSloppyPhraseQuery2, TestForceMergeForever, TestNearSpansOrdered, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, TestPayloads, TestDoc, TestWildcard, TestTermRangeFilter, TestNoDeletionPolicy, TestTermVectors, TestRegexpQuery, TestTermVectorsReader, TestSpansAdvanced2, TestMultiValuedNumericRangeQuery, TestIntBlockPool, TestUniqueTermCount, TestComplexExplanationsOfNonMatches, TestPrefixCodedTerms, TestParallelReaderEmptyIndex, TestForUtil, TestTermScorer, TestTopFieldCollector, TestFileSwitchDirectory, TestIndexableField, TestPrefixRandom, TestDirectory, TestRegexpRandom, TestMultiLevelSkipList, TestCheckIndex, TestPayloadsOnVectors, TestCachingWrapperFilter, TestSpanMultiTermQueryWrapper, TestBooleanQueryVisitSubscorers, InBeforeClass, InAfterClass, InTestMethod, NonStringProperties, TestConstantScoreQuery, TestMatchAllDocsQuery, TestElevationComparator, TestFilteredSearch, TestSubScorerFreqs, TestRecyclingIntBlockAllocator, TestIndexWriterConfig, TestConjunctions, TestMultiTermQueryRewrites, Nested, Nested, TestField, TestDateSort, TestSpanExplanationsOfNonMatches, TestVirtualMethod, TestSmallFloat, TestSimilarityProvider, TestNumericTokenStream, TestFilterAtomicReader, ThrowInUncaught, TestFilterIterator, TestDateFilter, TestCharsRef, Test2BPostings, TestDocCount, TestAutomatonQueryUnicode, NestedSetupChain, NestedTeardownChain, TestTotalHitCountCollector, TestRecyclingByteBlockAllocator, TestMockDirectoryWrapper, Nested1, Nested, TestNoMergePolicy, TestNoMergeScheduler, TestNamedSPILoader, TestIndexWriterLockRelease, TestDemo, TestRollback, TestTwoPhaseCommitTool, TestNot, TestCloseableThreadLocal, TestMockCharFilter, TestPrefixFilter, TestVersion, Nested1, TestCodecHoldsOpenFiles, TestReaderClosed, MultiCollectorTest, TestDocBoost, TestIntsRef, Nested1, TestTermdocPerf, Test2BPositions, TestTerm, TestTopScoreDocCollector, Test2BTerms, TestByteArrayDataInput, Test2BNumericDocValues, Test2BBinaryDocValues, Test2BPagedBytes, TestPackedTokenAttributeImpl, TestImpersonation, TestSurrogates, TestTermInfosReaderIndex]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestTermInfosReaderIndex -Dtests.seed=18C00777C7D5AB95 -Dtests.slow=true -Dtests.locale=es_UY -Dtests.timezone=America/Maceio -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   0.00s | TestTermInfosReaderIndex (suite) <<<
   [junit4]    > Throwable #1: java.io.FileNotFoundException: _2.fnm in dir=RAMDirectory@34312b8a lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8218bcb
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([18C00777C7D5AB95]:0)
   [junit4]    > 	at org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:579)
   [junit4]    > 	at org.apache.lucene.codecs.lucene3x.PreFlexRWFieldInfosReader.read(PreFlexRWFieldInfosReader.java:45)
   [junit4]    > 	at org.apache.lucene.codecs.lucene3x.TestTermInfosReaderIndex.beforeClass(TestTermInfosReaderIndex.java:100)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)Throwable #2: java.lang.NullPointerException
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([18C00777C7D5AB95]:0)
   [junit4]    > 	at org.apache.lucene.codecs.lucene3x.TestTermInfosReaderIndex.afterClass(TestTermInfosReaderIndex.java:118)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4] Completed in 0.06s, 0 tests, 2 errors <<< FAILURES!

[...truncated 120 lines...]
BUILD FAILED
C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\build.xml:467: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\build.xml:447: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\build.xml:45: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\extra-targets.xml:37: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\lucene\build.xml:49: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\lucene\common-build.xml:1297: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\lucene\common-build.xml:921: There were test failures: 404 suites, 3132 tests, 2 suite-level errors, 98 ignored (88 assumptions)

Total time: 14 minutes 8 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_60-ea-b15 -XX:-UseCompressedOops -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-4.x-Windows (32bit/jdk1.7.0_55) - Build # 3939 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Windows/3939/
Java: 32bit/jdk1.7.0_55 -server -XX:+UseConcMarkSweepGC

All tests passed

Build Log:
[...truncated 20622 lines...]
BUILD FAILED
C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\build.xml:467: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\build.xml:92: The following files contain @author tags, tabs or nocommits:
* solr/core/src/test/org/apache/solr/handler/TestReplicationHandler.java

Total time: 123 minutes 40 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_55 -server -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-4.x-Windows (64bit/jdk1.7.0_60-ea-b15) - Build # 3938 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Windows/3938/
Java: 64bit/jdk1.7.0_60-ea-b15 -XX:-UseCompressedOops -XX:+UseG1GC

All tests passed

Build Log:
[...truncated 20647 lines...]
BUILD FAILED
C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\build.xml:467: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\build.xml:92: The following files contain @author tags, tabs or nocommits:
* solr/core/src/test/org/apache/solr/handler/TestReplicationHandler.java

Total time: 113 minutes 11 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_60-ea-b15 -XX:-UseCompressedOops -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-4.x-Windows (64bit/jdk1.7.0_55) - Build # 3937 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Windows/3937/
Java: 64bit/jdk1.7.0_55 -XX:-UseCompressedOops -XX:+UseParallelGC

All tests passed

Build Log:
[...truncated 20494 lines...]
BUILD FAILED
C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\build.xml:467: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\build.xml:92: The following files contain @author tags, tabs or nocommits:
* solr/core/src/test/org/apache/solr/handler/TestReplicationHandler.java

Total time: 113 minutes 44 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_55 -XX:-UseCompressedOops -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure