You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/08/08 13:33:45 UTC

[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_65) - Build # 10977 - Still Failing!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/10977/
Java: 32bit/jdk1.7.0_65 -client -XX:+UseG1GC

1 tests failed.
REGRESSION:  org.apache.solr.cloud.OverseerTest.testOverseerFailure

Error Message:
Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed

Stack Trace:
org.apache.solr.common.SolrException: Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed
	at __randomizedtesting.SeedInfo.seed([66EAD145A0F8583E:62E25EB6B25DB71F]:0)
	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:144)
	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:163)
	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:125)
	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:155)
	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:314)
	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:221)
	at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:155)
	at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:660)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
	at org.apache.solr.common.cloud.SolrZkClient$11.execute(SolrZkClient.java:457)
	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:454)
	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:411)
	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:398)
	at org.apache.solr.cloud.ShardLeaderElectionContextBase$1.execute(ElectionContext.java:136)
	at org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:31)
	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:131)
	... 47 more




Build Log:
[...truncated 11494 lines...]
   [junit4] Suite: org.apache.solr.cloud.OverseerTest
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.OverseerTest-66EAD145A0F8583E-001/init-core-data-001
   [junit4]   2> 1571545 T4229 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
   [junit4]   2> 1571546 T4229 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> 1571546 T4229 oas.SolrTestCaseJ4.initCore ####initCore end
   [junit4]   2> 1571551 T4229 oas.SolrTestCaseJ4.setUp ###Starting testOverseerFailure
   [junit4]   2> 1571552 T4229 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1571553 T4230 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1571652 T4229 oasc.ZkTestServer.run start zk server on port:44278
   [junit4]   2> 1571654 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1571656 T4236 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@437f97 name:ZooKeeperConnection Watcher:127.0.0.1:44278 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1571656 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1571658 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1571659 T4238 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@59d421 name:ZooKeeperConnection Watcher:127.0.0.1:44278 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1571659 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1571659 T4229 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1571662 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1571663 T4240 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b191d7 name:ZooKeeperConnection Watcher:127.0.0.1:44278/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1571663 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1571664 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1571666 T4229 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1571668 T4229 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1571669 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1571672 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1571673 T4242 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1915eda name:ZooKeeperConnection Watcher:127.0.0.1:44278/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1571673 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1571674 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1571677 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 1571680 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1571681 T4246 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b706d5 name:ZooKeeperConnection Watcher:127.0.0.1:44278/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1571681 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1571682 T4229 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 1571684 T4229 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1571687 T4229 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:44278_solr
   [junit4]   2> 1571687 T4229 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1571689 T4229 oasc.Overseer.start Overseer (id=92241659301527556-127.0.0.1:44278_solr-n_0000000000) starting
   [junit4]   2> 1571690 T4229 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1571692 T4229 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1571694 T4229 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 1571697 T4229 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 1571698 T4229 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 1571700 T4229 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1571703 T4248 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1571704 T4247 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1572705 T4249 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1572706 T4247 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1572707 T4247 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 1572707 T4247 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1]
   [junit4]   2> 1572708 T4247 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1572713 T4243 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1572713 T4244 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1573205 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 1573211 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 1573215 T4249 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1573215 T4229 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1573216 T4247 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1573218 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 1573219 T4247 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 1573219 T4247 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 1573220 T4249 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1573221 T4244 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1573221 T4243 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1573705 T4248 oasc.OverseerCollectionProcessor.amILeader  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:302)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:299)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:299)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerCollectionProcessor.amILeader(OverseerCollectionProcessor.java:514)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:269)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 1573706 T4248 oasc.OverseerCollectionProcessor.amILeader According to ZK I (id=92241659301527556-127.0.0.1:44278_solr-n_0000000000) am no longer a leader.
   [junit4]   2> 1574276 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 1574280 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1574280 T4251 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@989290 name:ZooKeeperConnection Watcher:127.0.0.1:44278/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1574281 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1574282 T4247 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:44278_solr
   [junit4]   2> 1574282 T4252 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:302)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:299)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:299)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:317)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:289)
   [junit4]   2> 
   [junit4]   2> 1574284 T4229 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:44278_solr
   [junit4]   2> 1574284 T4229 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1574286 T4229 oasc.Overseer.start Overseer (id=92241659301527557-127.0.0.1:44278_solr-n_0000000001) starting
   [junit4]   2> 1574292 T4254 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1574292 T4253 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1574294 T4253 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1574295 T4253 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 1574295 T4253 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 1574297 T4255 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1574297 T4244 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1574297 T4243 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1574350 T4255 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1574350 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 1574351 T4253 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1574354 T4229 oasc.LeaderElector.checkIfIamLeader WARN Failed setting watch org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /collections/collection1/leader_elect/shard1/election/92241659301527555-node1_core1-n_0000000001
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:302)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:299)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:299)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:148)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:314)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:221)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:155)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:660)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   [junit4]   2> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> 	at java.lang.reflect.Method.invoke(Method.java:606)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
   [junit4]   2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
   [junit4]   2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 1574356 T4256 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 1574355 T4253 oasc.SolrException.log ERROR Problem cleaning up collection in zk:collection1:org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 		at org.apache.zookeeper.KeeperException.create(KeeperException.java:125)
   [junit4]   2> 		at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 		at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:184)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:181)
   [junit4]   2> 		at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.delete(SolrZkClient.java:181)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.clean(SolrZkClient.java:665)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.clean(SolrZkClient.java:660)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.clean(SolrZkClient.java:660)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.clean(SolrZkClient.java:660)
   [junit4]   2> 		at org.apache.solr.cloud.Overseer$ClusterStateUpdater.removeCore(Overseer.java:1094)
   [junit4]   2> 		at org.apache.solr.cloud.Overseer$ClusterStateUpdater.processMessage(Overseer.java:359)
   [junit4]   2> 		at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:242)
   [junit4]   2> 		at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 	
   [junit4]   2> 1574354 T4244 oasc.LeaderElector.checkIfIamLeader WARN Our node is no longer in line to be leader
   [junit4]   2> 1574358 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 1574361 T4255 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1574362 T4253 oasc.Overseer$ClusterStateUpdater.setShardLeader ERROR Could not mark shard leader for non existing collection:collection1
   [junit4]   2> 1574465 T4256 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1574465 T4243 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1575361 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 1576364 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 1577367 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 1578370 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 1579373 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 1580378 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 1581380 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 1582383 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 1583386 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 1584389 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 1585391 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 1586394 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 1587397 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 1588400 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 1589403 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 1589411 T4229 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:44278 44278
   [junit4]   2> 1590247 T4230 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:44278 44278
   [junit4]   2> 1590248 T4229 oas.SolrTestCaseJ4.tearDown ###Ending testOverseerFailure
   [junit4]   2> 1590249 T4253 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:44278_solr
   [junit4]   2> 1590249 T4257 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:302)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:299)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:299)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:317)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:289)
   [junit4]   2> 
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=OverseerTest -Dtests.method=testOverseerFailure -Dtests.seed=66EAD145A0F8583E -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=in -Dtests.timezone=Etc/GMT+2 -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   18.7s J1 | OverseerTest.testOverseerFailure <<<
   [junit4]    > Throwable #1: org.apache.solr.common.SolrException: Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([66EAD145A0F8583E:62E25EB6B25DB71F]:0)
   [junit4]    > 	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:144)
   [junit4]    > 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:163)
   [junit4]    > 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:125)
   [junit4]    > 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:155)
   [junit4]    > 	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:314)
   [junit4]    > 	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:221)
   [junit4]    > 	at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:155)
   [junit4]    > 	at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:660)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]    > Caused by: org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1
   [junit4]    > 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
   [junit4]    > 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]    > 	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient$11.execute(SolrZkClient.java:457)
   [junit4]    > 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:454)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:411)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:398)
   [junit4]    > 	at org.apache.solr.cloud.ShardLeaderElectionContextBase$1.execute(ElectionContext.java:136)
   [junit4]    > 	at org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:31)
   [junit4]    > 	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:131)
   [junit4]    > 	... 47 more
   [junit4]   2> 1590285 T4229 oas.SolrTestCaseJ4.setUp ###Starting testBadQueueItem
   [junit4]   2> 1590286 T4229 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1590287 T4258 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1590387 T4229 oasc.ZkTestServer.run start zk server on port:38232
   [junit4]   2> 1590388 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1590390 T4264 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4721e2 name:ZooKeeperConnection Watcher:127.0.0.1:38232 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1590390 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1590393 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1590394 T4266 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@156c819 name:ZooKeeperConnection Watcher:127.0.0.1:38232 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1590394 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1590395 T4229 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1590398 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1590399 T4268 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@111fde2 name:ZooKeeperConnection Watcher:127.0.0.1:38232/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1590399 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1590400 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1590402 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1590403 T4270 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5dc221 name:ZooKeeperConnection Watcher:127.0.0.1:38232/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1590403 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1590404 T4229 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 1590405 T4229 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1590408 T4229 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:38232_solr
   [junit4]   2> 1590408 T4229 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1590409 T4229 oasc.Overseer.start Overseer (id=92241660529278979-127.0.0.1:38232_solr-n_0000000000) starting
   [junit4]   2> 1590411 T4229 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1590412 T4229 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1590414 T4229 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 1590416 T4229 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 1590417 T4229 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 1590419 T4229 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1590422 T4272 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1590423 T4229 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1590423 T4271 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1590424 T4229 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1590425 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1590427 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1590428 T4274 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3db668 name:ZooKeeperConnection Watcher:127.0.0.1:38232/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1590428 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1590429 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1590430 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1
   [junit4]   2> 1590433 T4277 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1590434 T4271 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1590435 T4271 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 1590435 T4271 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
   [junit4]   2> 1590435 T4271 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 1590437 T4275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1590437 T4276 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1590933 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 1590939 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 1590943 T4277 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1590944 T4271 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1590946 T4271 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core2",
   [junit4]   2> 	  "core_node_name":"node2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 1590946 T4271 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 1590946 T4271 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1590948 T4277 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1590949 T4276 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1590949 T4275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1591444 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 1591449 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 1591452 T4277 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1591453 T4271 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1591455 T4271 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core3",
   [junit4]   2> 	  "core_node_name":"node3",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 1591456 T4271 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 1591456 T4271 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
   [junit4]   2> 1591457 T4277 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1591458 T4275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1591458 T4276 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1591954 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
   [junit4]   2> 1591959 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
   [junit4]   2> 1591962 T4277 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1591963 T4271 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1592065 T4276 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1592065 T4275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1592113 T4277 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1592115 T4271 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1592116 T4271 oasc.Overseer$ClusterStateUpdater.checkKeyExistence ERROR Skipping invalid Overseer message because it has no collection specified: {
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core0",
   [junit4]   2> 	  "core_node_name":"node0",
   [junit4]   2> 	  "collection":"",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 1592118 T4271 oasc.Overseer$ClusterStateUpdater.checkKeyExistence ERROR Skipping invalid Overseer message because it has no collection specified: {
   [junit4]   2> 	  "operation":"deletecore",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core0",
   [junit4]   2> 	  "core_node_name":"node0",
   [junit4]   2> 	  "collection":""}
   [junit4]   2> 1592120 T4277 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1592120 T4271 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection2",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 1592121 T4271 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection2 with shards [shard1, shard2, shard3]
   [junit4]   2> 1592121 T4271 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 1592224 T4275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1592224 T4276 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1592617 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection2/leader_elect/shard2/election
   [junit4]   2> 1592624 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection2/leaders/shard2
   [junit4]   2> 1592631 T4277 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1592633 T4271 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1592638 T4271 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core2",
   [junit4]   2> 	  "core_node_name":"node2",
   [junit4]   2> 	  "collection":"collection2",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 1592639 T4271 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 1592639 T4271 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1592641 T4277 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1592642 T4275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1592642 T4276 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1593132 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection2/leader_elect/shard1/election
   [junit4]   2> 1593138 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection2/leaders/shard1
   [junit4]   2> 1593141 T4277 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1593142 T4271 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1593144 T4271 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core3",
   [junit4]   2> 	  "core_node_name":"node3",
   [junit4]   2> 	  "collection":"collection2",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 1593144 T4271 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 1593145 T4271 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
   [junit4]   2> 1593146 T4277 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1593148 T4275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1593148 T4276 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1593642 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection2/leader_elect/shard3/election
   [junit4]   2> 1593648 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection2/leaders/shard3
   [junit4]   2> 1593651 T4277 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1593653 T4271 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1593758 T4276 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1593758 T4275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1593806 T4229 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:38232 38232
   [junit4]   2> 1594253 T4258 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:38232 38232
   [junit4]   2> 1594254 T4229 oas.SolrTestCaseJ4.tearDown ###Ending testBadQueueItem
   [junit4]   2> 1594254 T4271 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:38232_solr
   [junit4]   2> 1594255 T4278 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:302)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:299)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:299)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:317)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:289)
   [junit4]   2> 
   [junit4]   2> 1594258 T4229 oas.SolrTestCaseJ4.setUp ###Starting testPlaceholders
   [junit4]   2> 1594259 T4229 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1594260 T4279 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1594360 T4229 oasc.ZkTestServer.run start zk server on port:48272
   [junit4]   2> 1594361 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1594367 T4285 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6845b9 name:ZooKeeperConnection Watcher:127.0.0.1:48272/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1594368 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1594369 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1594370 T4287 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@9e54c name:ZooKeeperConnection Watcher:127.0.0.1:48272 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1594370 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1594372 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1594373 T4289 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1be2ae5 name:ZooKeeperConnection Watcher:127.0.0.1:48272 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1594373 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1594374 T4229 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1594376 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1594378 T4229 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1594380 T4229 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1594381 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1594383 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1594384 T4291 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ca9788 name:ZooKeeperConnection Watcher:127.0.0.1:48272/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1594384 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1594385 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1594387 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 1594394 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1594394 T4295 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1e0354b name:ZooKeeperConnection Watcher:127.0.0.1:48272/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1594395 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1594397 T4229 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 1594398 T4229 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1594401 T4229 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:48272_solr
   [junit4]   2> 1594401 T4229 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1594403 T4229 oasc.Overseer.start Overseer (id=92241660789653508-127.0.0.1:48272_solr-n_0000000000) starting
   [junit4]   2> 1594404 T4229 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1594406 T4229 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1594408 T4229 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 1594410 T4229 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 1594412 T4229 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 1594414 T4229 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1594417 T4297 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1594418 T4296 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1594420 T4296 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1594421 T4296 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=12 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"12",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 1594421 T4296 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3, shard4, shard5, shard6, shard7, shard8, shard9, shard10, shard11, shard12]
   [junit4]   2> 1594421 T4296 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 1594424 T4298 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1594424 T4292 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1594431 T4293 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1594920 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 1594926 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 1594929 T4298 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1594929 T4229 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1594930 T4296 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1594931 T4296 oasc.Overseer$ClusterStateUpdater.run WARN Solr cannot talk to ZK, exiting Overseer main queue loop org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /clusterstate.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:302)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:299)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:299)
   [junit4]   2> 	at org.apache.solr.common.cloud.ClusterState.load(ClusterState.java:239)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.updateClusterState(ZkStateReader.java:445)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.updateClusterState(ZkStateReader.java:242)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:234)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 1594932 T4296 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:48272_solr
   [junit4]   2> 1594933 T4299 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:302)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:299)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:299)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:317)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:289)
   [junit4]   2> 
   [junit4]   2> 1596420 T4297 oasc.OverseerCollectionProcessor.amILeader  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:302)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:299)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:299)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerCollectionProcessor.amILeader(OverseerCollectionProcessor.java:514)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:269)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 1596421 T4297 oasc.OverseerCollectionProcessor.amILeader According to ZK I (id=92241660789653508-127.0.0.1:48272_solr-n_0000000000) am no longer a leader.
   [junit4]   2> 1596432 T4292 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 1596435 T4229 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:48272 48272
   [junit4]   2> 1597028 T4279 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:48272 48272
   [junit4]   2> 1597029 T4229 oas.SolrTestCaseJ4.tearDown ###Ending testPlaceholders
   [junit4]   2> 1597033 T4229 oas.SolrTestCaseJ4.setUp ###Starting testShardAssignmentBigger
   [junit4]   2> 1597034 T4229 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1597035 T4300 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1597135 T4229 oasc.ZkTestServer.run start zk server on port:37174
   [junit4]   2> 1597136 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1597138 T4306 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f86206 name:ZooKeeperConnection Watcher:127.0.0.1:37174 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1597138 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1597140 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1597141 T4308 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@911f76 name:ZooKeeperConnection Watcher:127.0.0.1:37174 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1597141 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1597142 T4229 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1597147 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1597148 T4310 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6814f3 name:ZooKeeperConnection Watcher:127.0.0.1:37174/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1597148 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1597149 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1597151 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1597151 T4312 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1886595 name:ZooKeeperConnection Watcher:127.0.0.1:37174/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1597152 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1597153 T4229 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 1597154 T4229 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1597157 T4229 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:37174_solr
   [junit4]   2> 1597157 T4229 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1597159 T4229 oasc.Overseer.start Overseer (id=92241660971515907-127.0.0.1:37174_solr-n_0000000000) starting
   [junit4]   2> 1597160 T4229 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1597162 T4229 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1597164 T4229 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 1597165 T4229 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 1597167 T4229 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 1597169 T4229 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1597172 T4314 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1597173 T4313 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1597172 T4229 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1597174 T4229 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1597175 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1597177 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1597178 T4316 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@10c651c name:ZooKeeperConnection Watcher:127.0.0.1:37174/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1597179 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1597180 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1597181 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes/node0
   [junit4]   2> 1597183 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1597184 T4320 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b6ce8a name:ZooKeeperConnection Watcher:127.0.0.1:37174/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1597185 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1597187 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1597189 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 1597191 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1597192 T4323 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5bd290 name:ZooKeeperConnection Watcher:127.0.0.1:37174/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1597192 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1597193 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1597195 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes/node2
   [junit4]   2> 1597198 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1597198 T4326 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@633195 name:ZooKeeperConnection Watcher:127.0.0.1:37174/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1597199 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1597201 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1597203 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes/node3
   [junit4]   2> 1597206 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1597206 T4329 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@56716b name:ZooKeeperConnection Watcher:127.0.0.1:37174/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1597207 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1597208 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1597209 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes/node4
   [junit4]   2> 1597211 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1597212 T4332 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1bd2054 name:ZooKeeperConnection Watcher:127.0.0.1:37174/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1597213 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1597213 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1597215 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes/node5
   [junit4]   2> 1597218 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1597218 T4335 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@880735 name:ZooKeeperConnection Watcher:127.0.0.1:37174/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1597219 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1597220 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1597221 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes/node6
   [junit4]   2> 1597224 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1597225 T4338 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5666ad name:ZooKeeperConnection Watcher:127.0.0.1:37174/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1597226 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1597227 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1597228 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes/node7
   [junit4]   2> 1597231 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1597232 T4341 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@11ec740 name:ZooKeeperConnection Watcher:127.0.0.1:37174/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1597232 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1597233 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1597234 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes/node8
   [junit4]   2> 1597238 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1597239 T4344 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f44fb2 name:ZooKeeperConnection Watcher:127.0.0.1:37174/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1597239 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1597240 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1597241 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes/node9
   [junit4]   2> 1597245 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1597246 T4347 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e4b02a name:ZooKeeperConnection Watcher:127.0.0.1:37174/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1597246 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1597247 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1597249 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes/node10
   [junit4]   2> 1597253 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1597254 T4350 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@757e3f name:ZooKeeperConnection Watcher:127.0.0.1:37174/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1597254 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1597255 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1597256 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes/node11
   [junit4]   2> 1597260 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1597261 T4353 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@14a871 name:ZooKeeperConnection Watcher:127.0.0.1:37174/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1597262 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1597262 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1597264 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes/node12
   [junit4]   2> 1597268 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1597269 T4356 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@8cb2cf name:ZooKeeperConnection Watcher:127.0.0.1:37174/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1597269 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1597270 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1597272 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes/node13
   [junit4]   2> 1597276 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1597277 T4359 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@16dc3b5 name:ZooKeeperConnection Watcher:127.0.0.1:37174/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1597277 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1597278 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1597279 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes/node14
   [junit4]   2> 1597284 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1597285 T4362 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@12dae6d name:ZooKeeperConnection Watcher:127.0.0.1:37174/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1597286 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1597287 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1597289 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes/node15
   [junit4]   2> 1597293 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1597294 T4365 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1316bf8 name:ZooKeeperConnection Watcher:127.0.0.1:37174/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1597295 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1597296 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1597297 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes/node16
   [junit4]   2> 1597304 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1597304 T4368 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1bdd2df name:ZooKeeperConnection Watcher:127.0.0.1:37174/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1597305 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1597306 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1597308 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes/node17
   [junit4]   2> 1597312 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1597313 T4371 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@9ff248 name:ZooKeeperConnection Watcher:127.0.0.1:37174/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1597313 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1597314 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1597316 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes/node18
   [junit4]   2> 1597320 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1597321 T4374 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@da4bb0 name:ZooKeeperConnection Watcher:127.0.0.1:37174/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1597321 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1597322 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1597324 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes/node19
   [junit4]   2> 1597328 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1597329 T4377 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7ec993 name:ZooKeeperConnection Watcher:127.0.0.1:37174/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1597329 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1597330 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1597331 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes/node20
   [junit4]   2> 1597335 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1597336 T4380 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@183989d name:ZooKeeperConnection Watcher:127.0.0.1:37174/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1597336 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1597337 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1597339 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes/node21
   [junit4]   2> 1597342 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1597344 T4382 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@a8244 name:ZooKeeperConnection Watcher:127.0.0.1:37174/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1597344 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1597345 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1597346 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes/node22
   [junit4]   2> 1597351 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1597352 T4386 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b11909 name:ZooKeeperConnection Watcher:127.0.0.1:37174/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1597352 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1597353 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1597354 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes/node23
   [junit4]   2> 1597358 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1597359 T4389 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ca9531 name:ZooKeeperConnection Watcher:127.0.0.1:37174/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1597360 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1597360 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1597362 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes/node24
   [junit4]   2> 1597367 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1597368 T4392 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@baba79 name:ZooKeeperConnection Watcher:127.0.0.1:37174/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1597368 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1597369 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1597370 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes/node25
   [junit4]   2> 1597375 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1597376 T4395 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@dbc447 name:ZooKeeperConnection Watcher:127.0.0.1:37174/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1597376 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1597377 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1597379 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes/node26
   [junit4]   2> 1597385 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1597386 T4398 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d1fcaa name:ZooKeeperConnection Watcher:127.0.0.1:37174/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1597386 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1597387 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1597388 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes/node27
   [junit4]   2> 1597395 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1597396 T4401 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1bdc83f name:ZooKeeperConnection Watcher:127.0.0.1:37174/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]

[...truncated too long message...]

cc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@19cc474 name:ZooKeeperConnection Watcher:127.0.0.1:47578/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1648911 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1648912 T4229 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 1648913 T4229 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1648915 T4229 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:47578_solr
   [junit4]   2> 1648915 T4229 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1648917 T4229 oasc.Overseer.start Overseer (id=92241664363397123-127.0.0.1:47578_solr-n_0000000000) starting
   [junit4]   2> 1648918 T4229 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1648920 T4229 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1648922 T4229 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 1648923 T4229 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 1648925 T4229 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 1648927 T4229 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1648931 T4927 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1648931 T4926 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1648932 T4229 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1648933 T4926 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1648934 T4926 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "roles":"",
   [junit4]   2> 	  "state":"recovering"}
   [junit4]   2> 1648934 T4926 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1648935 T4928 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1648936 T4929 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
   [junit4]   2> 1648984 T4229 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1648986 T4928 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1648988 T4926 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1648989 T4926 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "roles":"",
   [junit4]   2> 	  "state":"active"}
   [junit4]   2> 1648990 T4926 oasc.Overseer$ClusterStateUpdater.updateState node=core_node1 is already registered
   [junit4]   2> 1648990 T4926 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 1649095 T4929 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
   [junit4]   2> 1649140 T4229 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:47578 47578
   [junit4]   2> 1649712 T4913 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:47578 47578
   [junit4]   2> 1649716 T4229 oas.SolrTestCaseJ4.tearDown ###Ending testStateChange
   [junit4]   2> 1649717 T4926 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:47578_solr
   [junit4]   2> 1649717 T4930 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:302)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:299)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:299)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:317)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:289)
   [junit4]   2> 
   [junit4]   2> 1652719 T4229 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.OverseerTest-66EAD145A0F8583E-001
   [junit4]   2> NOTE: test params are: codec=Lucene42, sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=in, timezone=Etc/GMT+2
   [junit4]   2> NOTE: Linux 3.13.0-32-generic i386/Oracle Corporation 1.7.0_65 (32-bit)/cpus=8,threads=1,free=311994352,total=525336576
   [junit4]   2> NOTE: All tests run in this JVM: [HttpPartitionTest, OpenCloseCoreStressTest, TestUtils, DistribCursorPagingTest, TestImplicitCoreProperties, BlockDirectoryTest, TestFreeTextSuggestions, SortByFunctionTest, TestDistribDocBasedVersion, TestPostingsSolrHighlighter, DocValuesMissingTest, QueryParsingTest, TestNonNRTOpen, RequestHandlersTest, TermVectorComponentDistributedTest, ModifyConfFileTest, TestCSVResponseWriter, TestCollapseQParserPlugin, CachingDirectoryFactoryTest, QueryFacetTest, RollingRestartTest, DeleteShardTest, HdfsUnloadDistributedZkTest, TestSimpleQParserPlugin, TestExpandComponent, HdfsChaosMonkeySafeLeaderTest, UpdateRequestProcessorFactoryTest, DistributedQueryComponentCustomSortTest, HdfsRecoveryZkTest, DistributedQueryComponentOptimizationTest, TestBinaryField, AlternateDirectoryTest, ScriptEngineTest, TestQueryUtils, FieldFacetExtrasTest, SampleTest, TestModifyConfFiles, TestGroupingSearch, TestPseudoReturnFields, TestSerializedLuceneMatchVersion, TestUniqueKeyFieldResource, TestRandomFaceting, EnumFieldTest, MultiThreadedOCPTest, AliasIntegrationTest, TestSolrQueryParserResource, TestRecoveryHdfs, TestReversedWildcardFilterFactory, SolrIndexConfigTest, AutoCommitTest, AnalysisAfterCoreReloadTest, TermVectorComponentTest, TestLFUCache, RangeFacetTest, CSVRequestHandlerTest, TestDistributedGrouping, TestSolrQueryParserDefaultOperatorResource, NumericFieldsTest, DebugComponentTest, TestCloudManagedSchema, TestSchemaNameResource, TestSolrXmlPersistor, StressHdfsTest, PreAnalyzedFieldTest, TestFunctionQuery, DateMathParserTest, LegacyHTMLStripCharFilterTest, TestQueryTypes, TestRestManager, TestSweetSpotSimilarityFactory, TestIndexSearcher, ResourceLoaderTest, TestCoreContainer, ConvertedLegacyTest, TestIndexingPerformance, TestValueSourceCache, TestBadConfig, TestPhraseSuggestions, TestXIncludeConfig, TestMultiCoreConfBootstrap, HardAutoCommitTest, SoftAutoCommitTest, TestNRTOpen, LukeRequestHandlerTest, SpatialFilterTest, CollectionsAPIAsyncDistributedZkTest, TestFieldSortValues, FileBasedSpellCheckerTest, IndexSchemaRuntimeFieldTest, SynonymTokenizerTest, TestJoin, TestZkChroot, TestOmitPositions, UUIDFieldTest, NoFacetTest, DistributedSuggestComponentTest, TestCSVLoader, ClusterStateTest, PrimUtilsTest, UnloadDistributedZkTest, SyncSliceTest, OverseerTest]
   [junit4] Completed on J1 in 81.21s, 9 tests, 1 error <<< FAILURES!

[...truncated 642 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:474: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:454: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1298: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:922: There were test failures: 421 suites, 1761 tests, 1 error, 39 ignored (18 assumptions)

Total time: 104 minutes 26 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 32bit/jdk1.7.0_65 -client -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_11) - Build # 10980 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/10980/
Java: 64bit/jdk1.8.0_11 -XX:+UseCompressedOops -XX:+UseSerialGC

1 tests failed.
REGRESSION:  org.apache.solr.schema.TestCloudSchemaless.testDistribSearch

Error Message:
Timeout occured while waiting response from server at: https://127.0.0.1:54847/ed_pfk/mz/collection1

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: https://127.0.0.1:54847/ed_pfk/mz/collection1
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:561)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
	at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:124)
	at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:68)
	at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:54)
	at org.apache.solr.schema.TestCloudSchemaless.doTest(TestCloudSchemaless.java:140)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
	at sun.reflect.GeneratedMethodAccessor48.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(SocketInputStream.java:150)
	at java.net.SocketInputStream.read(SocketInputStream.java:121)
	at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
	at sun.security.ssl.InputRecord.read(InputRecord.java:503)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:954)
	at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:911)
	at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:260)
	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:271)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:682)
	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:486)
	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:449)
	... 46 more




Build Log:
[...truncated 12184 lines...]
   [junit4] Suite: org.apache.solr.schema.TestCloudSchemaless
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-96B09B3C79C0A379-001/init-core-data-001
   [junit4]   2> 2825501 T8885 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
   [junit4]   2> 2825501 T8885 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /ed_pfk/mz
   [junit4]   2> 2825504 T8885 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 2825504 T8885 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2825505 T8886 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 2825605 T8885 oasc.ZkTestServer.run start zk server on port:46389
   [junit4]   2> 2825606 T8885 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2825608 T8892 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1e9759e1 name:ZooKeeperConnection Watcher:127.0.0.1:46389 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2825608 T8885 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2825608 T8885 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 2825613 T8885 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2825614 T8894 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@21660033 name:ZooKeeperConnection Watcher:127.0.0.1:46389/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2825614 T8885 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2825615 T8885 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 2825616 T8885 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 2825617 T8885 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 2825618 T8885 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 2825620 T8885 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-schemaless.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 2825620 T8885 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 2825622 T8885 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema-add-schema-fields-update-processor.xml to /configs/conf1/schema.xml
   [junit4]   2> 2825622 T8885 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 2825623 T8885 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2825624 T8885 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2825625 T8885 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 2825625 T8885 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 2825626 T8885 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 2825627 T8885 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 2825628 T8885 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 2825628 T8885 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 2825629 T8885 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 2825630 T8885 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 2825631 T8885 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2825631 T8885 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2825633 T8885 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2825633 T8885 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2825634 T8885 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 2825635 T8885 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 2825636 T8885 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 2825637 T8885 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 2825638 T8885 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2825639 T8896 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2a7e3567 name:ZooKeeperConnection Watcher:127.0.0.1:46389/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2825639 T8885 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2825954 T8885 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2825957 T8885 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 2825958 T8885 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:55485
   [junit4]   2> 2825960 T8885 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2825960 T8885 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2825960 T8885 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-96B09B3C79C0A379-001/tempDir-002
   [junit4]   2> 2825960 T8885 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-96B09B3C79C0A379-001/tempDir-002/'
   [junit4]   2> 2825972 T8885 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-96B09B3C79C0A379-001/tempDir-002/solr.xml
   [junit4]   2> 2825979 T8885 oasc.CoreContainer.<init> New CoreContainer 1074387795
   [junit4]   2> 2825979 T8885 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-96B09B3C79C0A379-001/tempDir-002/]
   [junit4]   2> 2825980 T8885 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2825980 T8885 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 2825980 T8885 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2825981 T8885 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2825981 T8885 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2825981 T8885 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2825981 T8885 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2825982 T8885 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2825982 T8885 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2825983 T8885 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 2825984 T8885 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2825984 T8885 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2825984 T8885 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2825984 T8885 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:46389/solr
   [junit4]   2> 2825985 T8885 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2825985 T8885 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2825986 T8907 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@28722923 name:ZooKeeperConnection Watcher:127.0.0.1:46389 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2825986 T8885 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2825988 T8885 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2825988 T8909 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6492ec73 name:ZooKeeperConnection Watcher:127.0.0.1:46389/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2825989 T8885 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2825989 T8885 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 2825991 T8885 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 2825993 T8885 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 2825994 T8885 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 2825996 T8885 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 2825998 T8885 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 2825999 T8885 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55485_ed_pfk%2Fmz
   [junit4]   2> 2825999 T8885 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:55485_ed_pfk%2Fmz
   [junit4]   2> 2826001 T8885 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 2826002 T8885 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 2826004 T8885 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:55485_ed_pfk%2Fmz
   [junit4]   2> 2826004 T8885 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 2826005 T8885 oasc.Overseer.start Overseer (id=92244181760868356-127.0.0.1:55485_ed_pfk%2Fmz-n_0000000000) starting
   [junit4]   2> 2826007 T8885 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 2826012 T8911 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 2826013 T8885 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 2826014 T8885 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 2826015 T8885 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2826016 T8910 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 2826019 T8912 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 2826019 T8912 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2826019 T8912 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2826019 T8913 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2826020 T8910 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2826021 T8910 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:55485/ed_pfk/mz",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:55485_ed_pfk%2Fmz",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 2826021 T8910 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 2826021 T8910 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 2826022 T8913 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2827019 T8912 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 2827020 T8912 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 2827020 T8912 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2827021 T8912 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 2827021 T8912 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2827021 T8912 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-96B09B3C79C0A379-001/tempDir-002/collection1/'
   [junit4]   2> 2827022 T8912 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-96B09B3C79C0A379-001/tempDir-002/collection1/lib/.svn/' to classloader
   [junit4]   2> 2827022 T8912 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-96B09B3C79C0A379-001/tempDir-002/collection1/lib/classes/' to classloader
   [junit4]   2> 2827023 T8912 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-96B09B3C79C0A379-001/tempDir-002/collection1/lib/README' to classloader
   [junit4]   2> 2827039 T8912 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
   [junit4]   2> 2827047 T8912 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2827047 T8912 oass.ManagedIndexSchemaFactory.create The schema is configured as managed, but managed schema resource managed-schema not found - loading non-managed schema schema.xml instead
   [junit4]   2> 2827048 T8912 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2827050 T8912 oass.IndexSchema.readSchema [collection1] Schema name=add-schema-fields-update-processor
   [junit4]   2> 2827055 T8912 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2827056 T8912 oass.ManagedIndexSchema.persistManagedSchemaToZooKeeper Created and persisted managed schema znode at /configs/conf1/managed-schema
   [junit4]   2> 2827057 T8912 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml.bak
   [junit4]   2> 2827059 T8912 oass.ManagedIndexSchemaFactory.zkUgradeToManagedSchema After upgrading to managed schema in ZooKeeper, renamed the non-managed schema /configs/conf1/schema.xml to /configs/conf1/schema.xml.bak
   [junit4]   2> 2827059 T8912 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 2827059 T8912 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
   [junit4]   2> 2827059 T8912 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-96B09B3C79C0A379-001/tempDir-002/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-96B09B3C79C0A379-001/tempDir-002/collection1/data/
   [junit4]   2> 2827059 T8912 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 2827060 T8912 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-96B09B3C79C0A379-001/tempDir-002/collection1/data
   [junit4]   2> 2827060 T8912 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-96B09B3C79C0A379-001/tempDir-002/collection1/data/index/
   [junit4]   2> 2827060 T8912 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-96B09B3C79C0A379-001/tempDir-002/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 2827061 T8912 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-96B09B3C79C0A379-001/tempDir-002/collection1/data/index
   [junit4]   2> 2827061 T8912 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=40, maxMergeAtOnceExplicit=24, maxMergedSegmentMB=67.6982421875, floorSegmentMB=0.9951171875, forceMergeDeletesPctAllowed=13.454031946870305, segmentsPerTier=28.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 2846833 T8912 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=NRTCachingDirectory(MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-96B09B3C79C0A379-001/tempDir-002/collection1/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@731b7756; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
   [junit4]   2> 2846834 T8912 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2846835 T8912 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "add-unknown-fields-to-the-schema"
   [junit4]   2> 2846836 T8912 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2846836 T8912 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2846836 T8912 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2846836 T8912 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2846836 T8912 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2846836 T8912 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 2846837 T8912 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 2846837 T8912 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 2846837 T8912 oasc.RequestHandlers.initHandlersFromConfig WARN Multiple requestHandler registered to the same name: /update ignoring: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2846837 T8912 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 2846837 T8912 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 2846837 T8912 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 2846839 T8912 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2846840 T8912 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2846840 T8912 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2846841 T8912 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2846841 T8912 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2846842 T8912 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 2846843 T8912 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 2846843 T8912 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=39, maxMergeAtOnceExplicit=46, maxMergedSegmentMB=55.0341796875, floorSegmentMB=1.162109375, forceMergeDeletesPctAllowed=20.418511391805534, segmentsPerTier=30.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 2846844 T8912 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=NRTCachingDirectory(MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-96B09B3C79C0A379-001/tempDir-002/collection1/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@731b7756; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
   [junit4]   2> 2846844 T8912 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2846844 T8912 oass.SolrIndexSearcher.<init> Opening Searcher@63e0028b[collection1] main
   [junit4]   2> 2846844 T8912 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 2846845 T8912 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2846845 T8912 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 2846846 T8912 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2846846 T8912 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 2846846 T8912 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2846847 T8912 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 2846847 T8912 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2846847 T8912 oascc.SolrZkClient.makePath makePath: /configs/conf1/_rest_managed.json
   [junit4]   2> 2846849 T8912 oasr.ManagedResourceStorage$ZooKeeperStorageIO$1.close Wrote 38 bytes to new znode /configs/conf1/_rest_managed.json
   [junit4]   2> 2846849 T8912 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2846849 T8912 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 2846850 T8912 oass.ZkIndexSchemaReader.createSchemaWatcher Creating ZooKeeper watch for the managed schema at /configs/conf1/managed-schema ...
   [junit4]   2> 2846869 T8914 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@63e0028b[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> 2846869 T8912 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 2846870 T8917 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:55485/ed_pfk/mz collection:control_collection shard:shard1
   [junit4]   2> 2846870 T8885 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 2846870 T8885 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2846871 T8917 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 2846873 T8885 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2846873 T8919 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@40edb6b7 name:ZooKeeperConnection Watcher:127.0.0.1:46389/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2846874 T8885 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2846874 T8917 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 2846874 T8885 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2846875 T8913 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2846875 T8917 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 2846876 T8917 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 2846876 T8885 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 2846876 T8910 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> ASYNC  NEW_CORE C3162 name=collection1 org.apache.solr.core.SolrCore@4465f2c3 url=https://127.0.0.1:55485/ed_pfk/mz/collection1 node=127.0.0.1:55485_ed_pfk%2Fmz C3162_STATE=coll:control_collection core:collection1 props:{state=down, base_url=https://127.0.0.1:55485/ed_pfk/mz, core=collection1, node_name=127.0.0.1:55485_ed_pfk%2Fmz}
   [junit4]   2> 2846876 T8917 C3162 P55485 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:55485/ed_pfk/mz/collection1/
   [junit4]   2> 2846876 T8917 C3162 P55485 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 2846877 T8917 C3162 P55485 oasc.SyncStrategy.syncToMe https://127.0.0.1:55485/ed_pfk/mz/collection1/ has no replicas
   [junit4]   2> 2846877 T8917 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:55485/ed_pfk/mz/collection1/ shard1
   [junit4]   2> 2846877 T8917 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 2846878 T8913 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2846878 T8920 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2846880 T8913 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2846881 T8910 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2846983 T8913 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2846983 T8920 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2847042 T8917 oasc.ZkController.register We are https://127.0.0.1:55485/ed_pfk/mz/collection1/ and leader is https://127.0.0.1:55485/ed_pfk/mz/collection1/
   [junit4]   2> 2847043 T8917 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:55485/ed_pfk/mz
   [junit4]   2> 2847043 T8917 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 2847043 T8917 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 2847043 T8917 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2847044 T8913 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2847045 T8917 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2847045 T8910 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2847046 T8910 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:55485/ed_pfk/mz",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:55485_ed_pfk%2Fmz",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 2847147 T8920 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2847148 T8913 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2847215 T8885 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 2847215 T8885 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2847218 T8885 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 2847219 T8885 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:39024
   [junit4]   2> 2847221 T8885 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2847222 T8885 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2847222 T8885 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-96B09B3C79C0A379-001/tempDir-003
   [junit4]   2> 2847222 T8885 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-96B09B3C79C0A379-001/tempDir-003/'
   [junit4]   2> 2847235 T8885 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-96B09B3C79C0A379-001/tempDir-003/solr.xml
   [junit4]   2> 2847244 T8885 oasc.CoreContainer.<init> New CoreContainer 1974412040
   [junit4]   2> 2847244 T8885 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-96B09B3C79C0A379-001/tempDir-003/]
   [junit4]   2> 2847245 T8885 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2847245 T8885 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 2847246 T8885 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2847246 T8885 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2847246 T8885 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2847247 T8885 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2847247 T8885 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2847247 T8885 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2847248 T8885 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2847249 T8885 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 2847250 T8885 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2847251 T8885 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2847251 T8885 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2847252 T8885 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:46389/solr
   [junit4]   2> 2847252 T8885 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2847253 T8885 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2847254 T8931 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@34d50d7f name:ZooKeeperConnection Watcher:127.0.0.1:46389 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2847254 T8885 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2847256 T8885 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2847257 T8933 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7fffffe1 name:ZooKeeperConnection Watcher:127.0.0.1:46389/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2847257 T8885 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2847262 T8885 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2848264 T8885 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39024_ed_pfk%2Fmz
   [junit4]   2> 2848265 T8885 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:39024_ed_pfk%2Fmz
   [junit4]   2> 2848271 T8935 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 2848271 T8935 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2848272 T8913 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2848272 T8935 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2848273 T8910 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2848273 T8910 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:39024/ed_pfk/mz",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:39024_ed_pfk%2Fmz",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 2848273 T8910 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3, shard4]
   [junit4]   2> 2848274 T8910 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 2848378 T8934 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2848378 T8920 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2848378 T8913 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2849272 T8935 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 2849273 T8935 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 2849273 T8935 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2849274 T8935 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 2849275 T8935 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2849275 T8935 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-96B09B3C79C0A379-001/tempDir-003/collection1/'
   [junit4]   2> 2849276 T8935 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-96B09B3C79C0A379-001/tempDir-003/collection1/lib/.svn/' to classloader
   [junit4]   2> 2849276 T8935 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-96B09B3C79C0A379-001/tempDir-003/collection1/lib/classes/' to classloader
   [junit4]   2> 2849277 T8935 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-96B09B3C79C0A379-001/tempDir-003/collection1/lib/README' to classloader
   [junit4]   2> 2849303 T8935 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
   [junit4]   2> 2849310 T8935 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2849312 T8935 oass.IndexSchema.readSchema Reading Solr Schema from managed-schema
   [junit4]   2> 2849314 T8935 oass.IndexSchema.readSchema [collection1] Schema name=add-schema-fields-update-processor
   [junit4]   2> 2849319 T8935 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2849319 T8935 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 2849319 T8935 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
   [junit4]   2> 2849320 T8935 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-96B09B3C79C0A379-001/tempDir-003/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-96B09B3C79C0A379-001/tempDir-003/collection1/data/
   [junit4]   2> 2849320 T8935 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 2849320 T8935 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-96B09B3C79C0A379-001/tempDir-003/collection1/data
   [junit4]   2> 2849320 T8935 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-96B09B3C79C0A379-001/tempDir-003/collection1/data/index/
   [junit4]   2> 2849321 T8935 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-96B09B3C79C0A379-001/tempDir-003/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 2849321 T8935 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-96B09B3C79C0A379-001/tempDir-003/collection1/data/index
   [junit4]   2> 2849321 T8935 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=40, maxMergeAtOnceExplicit=24, maxMergedSegmentMB=67.6982421875, floorSegmentMB=0.9951171875, forceMergeDeletesPctAllowed=13.454031946870305, segmentsPerTier=28.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 2849329 T8935 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=NRTCachingDirectory(MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-96B09B3C79C0A379-001/tempDir-003/collection1/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2d766676; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
   [junit4]   2> 2849330 T8935 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2849332 T8935 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "add-unknown-fields-to-the-schema"
   [junit4]   2> 2849333 T8935 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2849334 T8935 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2849334 T8935 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2849335 T8935 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2849335 T8935 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2849335 T8935 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 2849336 T8935 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 2849336 T8935 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 2849336 T8935 oasc.RequestHandlers.initHandlersFromConfig WARN Multiple requestHandler registered to the same name: /update ignoring: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2849337 T8935 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 2849337 T8935 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 2849337 T8935 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 2849340 T8935 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2849342 T8935 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2849342 T8935 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2849343 T8935 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2849344 T8935 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2849346 T8935 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 2849346 T8935 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 2849347 T8935 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=39, maxMergeAtOnceExplicit=46, maxMergedSegmentMB=55.0341796875, floorSegmentMB=1.162109375, forceMergeDeletesPctAllowed=20.418511391805534, segmentsPerTier=30.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 2849348 T8935 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=NRTCachingDirectory(MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-96B09B3C79C0A379-001/tempDir-003/collection1/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2d766676; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
   [junit4]   2> 2849348 T8935 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2849348 T8935 oass.SolrIndexSearcher.<init> Opening Searcher@f95722f[collection1] main
   [junit4]   2> 2849349 T8935 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 2849350 T8935 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2849351 T8935 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 2849351 T8935 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2849352 T8935 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 2849352 T8935 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2849354 T8935 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
   [junit4]   2> 2849354 T8935 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2849354 T8935 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
   [junit4]   2> 2849354 T8935 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 2849355 T8935 oass.ZkIndexSchemaReader.createSchemaWatcher Creating ZooKeeper watch for the managed schema at /configs/conf1/managed-schema ...
   [junit4]   2> 2849356 T8935 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 2849356 T8936 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@f95722f[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> 2849357 T8939 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:39024/ed_pfk/mz collection:collection1 shard:shard2
   [junit4]   2> 2849357 T8885 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 2849358 T8885 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2849358 T8939 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 2849361 T8939 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 2849362 T8913 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2849362 T8939 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 2849362 T8939 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C3163 name=collection1 org.apache.solr.core.SolrCore@1d4d6c37 url=https://127.0.0.1:39024/ed_pfk/mz/collection1 node=127.0.0.1:39024_ed_pfk%2Fmz C3163_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:39024/ed_pfk/mz, core=collection1, node_name=127.0.0.1:39024_ed_pfk%2Fmz}
   [junit4]   2> 2849362 T8939 C3163 P39024 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:39024/ed_pfk/mz/collection1/
   [junit4]   2> 2849363 T8939 C3163 P39024 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 2849363 T8910 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2849363 T8939 C3163 P39024 oasc.SyncStrategy.syncToMe https://127.0.0.1:39024/ed_pfk/mz/collection1/ has no replicas
   [junit4]   2> 2849363 T8939 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:39024/ed_pfk/mz/collection1/ shard2
   [junit4]   2> 2849363 T8939 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 2849365 T8913 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2849468 T8934 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2849468 T8913 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2849468 T8920 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2849516 T8939 oasc.ZkController.register We are https://127.0.0.1:39024/ed_pfk/mz/collection1/ and leader is https://127.0.0.1:39024/ed_pfk/mz/collection1/
   [junit4]   2> 2849516 T8939 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:39024/ed_pfk/mz
   [junit4]   2> 2849517 T8939 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 2849517 T8939 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 2849517 T8939 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2849539 T8913 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2849539 T8939 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2849540 T8910 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2849540 T8910 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:39024/ed_pfk/mz",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:39024_ed_pfk%2Fmz",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 2849644 T8920 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2849644 T8934 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2849644 T8913 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2849682 T8885 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 2849683 T8885 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2849685 T8885 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 2849686 T8885 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:54847
   [junit4]   2> 2849688 T8885 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2849688 T8885 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2849688 T8885 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-96B09B3C79C0A379-001/tempDir-004
   [junit4]   2> 2849688 T8885 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-96B09B3C79C0A379-001/tempDir-004/'
   [junit4]   2> 2849700 T8885 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-96B09B3C79C0A379-001/tempDir-004/solr.xml
   [junit4]   2> 2849706 T8885 oasc.CoreContainer.<init> New CoreContainer 1215001439
   [junit4]   2> 2849707 T8885 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-96B09B3C79C0A379-001/tempDir-004/]
   [junit4]   2> 2849707 T8885 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2849708 T8885 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 2849708 T8885 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2849708 T8885 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2849708 T8885 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2849708 T8885 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2849709 T8885 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2849709 T8885 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2849709 T8885 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2849710 T8885 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 2849711 T8885 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2849711 T8885 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2849711 T8885 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2849712 T8885 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:46389/solr
   [junit4]   2> 2849712 T8885 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2849713 T8885 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2849714 T8950 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7b37055d name:ZooKeeperConnection Watcher:127.0.0.1:46389 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2849714 T8885 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2849716 T8885 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2849717 T8952 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@419d9fa3 name:ZooKeeperConnection Watcher:127.0.0.1:46389/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2849717 T8885 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2849723 T8885 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2850726 T8885 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54847_ed_pfk%2Fmz
   [junit4]   2> 2850727 T8885 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:54847_ed_pfk%2Fmz
   [junit4]   2> 2850737 T8954 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 2850738 T8954 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2850739 T8913 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2850739 T8954 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2850741 T8910 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2850742 T8910 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:54847/ed_pfk/mz",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:54847_ed_pfk%2Fmz",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 2850742 T8910 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=4
   [junit4]   2> 2850742 T8910 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
   [junit4]   2> 2850846 T8934 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2850847 T8920 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2850847 T8913 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2850847 T8953 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2851740 T8954 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 2851740 T8954 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 2851741 T8954 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2851741 T8954 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 2851742 T8954 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2851742 T8954 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-96B09B3C79C0A379-001/tempDir-004/collection1/'
   [junit4]   2> 2851743 T8954 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-96B09B3C79C0A379-001/tempDir-004/collection1/lib/.svn/' to classloader
   [junit4]   2> 2851744 T8954 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-96B09B3C79C0A379-001/tempDir-004/collection1/lib/classes/' to classloader
   [junit4]   2> 2851744 T8954 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-96B09B3C79C0A379-001/tempDir-004/collection1/lib/README' to classloader
   [junit4]   2> 2851768 T8954 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
   [junit4]   2> 2851775 T8954 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2851776 T8954 oass.IndexSchema.readSchema Reading Solr Schema from managed-schema
   [junit4]   2> 2851778 T8954 oass.IndexSchema.readSchema [collection1] Schema name=add-schema-fields-update-processor
   [junit4]   2> 2851784 T8954 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2851784 T8954 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 2851784 T8954 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
   [junit4]   2> 2851785 T8954 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-96B09B3C79C0A379-001/tempDir-004/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-96B09B3C79C0A379-001/tempDir-004/collection1/data/
   [junit4]   2> 2851785 T8954 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 2851785 T8954 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-96B09B3C79C0A379-001/tempDir-004/collection1/data
   [junit4]   2> 2851786 T8954 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-96B09B3C79C0A379-001/tempDir-004/collection1/data/index/
   [junit4]   2> 2851786 T8954 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-96B09B3C79C0A379-001/tempDir-004/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 2851786 T8954 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-96B09B3C79C0A379-001/tempDir-004/collection1/data/index
   [junit4]   2> 2851787 T8954 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=40, maxMergeAtOnceExplicit=24, maxMergedSegmentMB=67.6982421875, floorSegmentMB=0.9951171875, forceMergeDeletesPctAllowed=13.454031946870305, segmentsPerTier=28.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 2851794 T8954 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=NRTCachingDirectory(MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-96B09B3C79C0A379-001/tempDir-004/collection1/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@35af5d21; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
   [junit4]   2> 2851794 T8954 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2851795 T8954 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "add-unknown-fields-to-the-schema"
   [junit4]   2> 2851796 T8954 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2851796 T8954 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2851796 T8954 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2851796 T8954 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2851797 T8954 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2851797 T8954 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 2851797 T8954 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 2851798 T8954 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 2851798 T8954 oasc.RequestHandlers.initHandlersFromConfig WARN Multiple requestHandler registered to the same name: /update ignoring: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2851798 T8954 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 2851798 T8954 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 2851798 T8954 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 2851800 T8954 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2851801 T8954 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2851802 T8954 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2851802 T8954 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2851803 T8954 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2851804 T8954 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 2851805 T8954 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 2851805 T8954 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=39, maxMergeAtOnceExplicit=46, maxMergedSegmentMB=55.0341796875, floorSegmentMB=1.162109375, forceMergeDeletesPctAllowed=20.418511391805534, segmentsPerTier=30.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 2851806 T8954 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=NRTCachingDirectory(MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-96B09B3C79C0A379-001/tempDir-004/collection1/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@35af5d21; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
   [junit4]   2> 2851806 T8954 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2851806 T8954 oass.SolrIndexSearcher.<init> Opening Searcher@5f08e846[collection1] main
   [junit4]   2> 2851807 T8954 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 2851808 T8954 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2851808 T8954 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 2851809 T8954 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2851809 T8954 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 2851809 T8954 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2851810 T8954 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
   [junit4]   2> 2851811 T8954 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2851811 T8954 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
   [junit4]   2> 2851811 T8954 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 2851811 T8954 oass.ZkIndexSchemaReader.createSchemaWatcher Creating ZooKeeper watch for the managed schema at /configs/conf1/managed-schema ...
   [junit4]   2> 2851812 T8955 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5f08e846[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> 2851812 T8954 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 2851813 T8958 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:54847/ed_pfk/mz collection:collection1 shard:shard3
   [junit4]   2> 2851813 T8885 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 2851814 T8885 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2851814 T8958 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
   [junit4]   2> 2851818 T8958 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard3
   [junit4]   2> 2851819 T8913 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2851819 T8958 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 2851819 T8958 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C3164 name=collection1 org.apache.solr.core.SolrCore@16f9d99b url=https://127.0.0.1:54847/ed_pfk/mz/collection1 node=127.0.0.1:54847_ed_pfk%2Fmz C3164_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:54847/ed_pfk/mz, core=collection1, node_name=127.0.0.1:54847_ed_pfk%2Fmz}
   [junit4]   2> 2851819 T8958 C3164 P54847 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:54847/ed_pfk/mz/collection1/
   [junit4]   2> 2851820 T8910 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2851820 T8958 C3164 P54847 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 2851820 T8958 C3164 P54847 oasc.SyncStrategy.syncToMe https://127.0.0.1:54847/ed_pfk/mz/collection1/ has no replicas
   [junit4]   2> 2851820 T8958 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:54847/ed_pfk/mz/collection1/ shard3
   [junit4]   2> 2851820 T8958 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
   [junit4]   2> 2851823 T8913 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2851925 T8934 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2851925 T8913 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2851925 T8920 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2851925 T8953 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2851974 T8958 oasc.ZkController.register We are https://127.0.0.1:54847/ed_pfk/mz/collection1/ and leader is https://127.0.0.1:54847/ed_pfk/mz/collection1/
   [junit4]   2> 2851974 T8958 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:54847/ed_pfk/mz
   [junit4]   2> 2851974 T8958 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 2851974 T8958 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 2851974 T8958 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2851975 T8913 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2851975 T8958 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2851976 T8910 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2851993 T8910 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:54847/ed_pfk/mz",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:54847_ed_pfk%2Fmz",
   [junit4]   2> 	  "shard":"shard3",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 2852095 T8920 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2852095 T8913 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2852095 T8934 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2852095 T8953 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2852133 T8885 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 2852134 T8885 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2852136 T8885 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 2852137 T8885 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:36933
   [junit4]   2> 2852139 T8885 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2852139 T8885 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2852139 T8885 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-96B09B3C79C0A379-001/tempDir-005
   [junit4]   2> 2852139 T8885 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-96B09B3C79C0A379-001/tempDir-005/'
   [junit4]   2> 2852152 T8885 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-96B09B3C79C0A379-001/tempDir-005/solr.xml
   [junit4]   2> 2852158 T8885 oasc.CoreContainer.<init> New CoreContainer 1211926672
   [junit4]   2> 2852158 T8885 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-96B09B3C79C0A379-001/tempDir-005/]
   [junit4]   2> 2852159 T8885 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2852159 T8885 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 2852159 T8885 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2852159 T8885 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2852159 T8885 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2852160 T8885 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2852160 T8885 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2852160 T8885 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2852160 T8885 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2852161 T8885 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 2852162 T8885 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2852162 T8885 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2852162 T8885 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2852163 T8885 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:46389/solr
   [junit4]   2> 2852163 T8885 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2852164 T8885 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2852165 T8969 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6d2a3cf8 name:ZooKeeperConnection Watcher:127.0.0.1:46389 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2852165 T8885 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2852167 T8885 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2852168 T8971 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@64041f41 name:ZooKeeperConnection Watcher:127.0.0.1:46389/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2852169 T8885 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2852177 T8885 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2853180 T8885 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36933_ed_pfk%2Fmz
   [junit4]   2> 2853182 T8885 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:36933_ed_pfk%2Fmz
   [junit4]   2> 2853192 T8973 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 2853193 T8973 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2853193 T8913 oas

[...truncated too long message...]

> 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
   [junit4]    > 	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:260)
   [junit4]    > 	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
   [junit4]    > 	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
   [junit4]    > 	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
   [junit4]    > 	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:271)
   [junit4]    > 	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
   [junit4]    > 	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:682)
   [junit4]    > 	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:486)
   [junit4]    > 	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
   [junit4]    > 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
   [junit4]    > 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
   [junit4]    > 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:449)
   [junit4]    > 	... 46 moreThrowable #2: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=9086, name=updateExecutor-1226-thread-2, state=RUNNABLE, group=TGRP-TestCloudSchemaless]
   [junit4]    > Caused by: java.util.concurrent.RejectedExecutionException: Task org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner@5e4d271c rejected from java.util.concurrent.ThreadPoolExecutor@1b4fa2b1[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 114]
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([96B09B3C79C0A379]:0)
   [junit4]    > 	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
   [junit4]    > 	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
   [junit4]    > 	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:263)
   [junit4]    > 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]    > 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 3138190 T8885 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-96B09B3C79C0A379-001
   [junit4]   2> NOTE: test params are: codec=Lucene46, sim=RandomSimilarityProvider(queryNorm=false,coord=crazy): {}, locale=mt, timezone=Asia/Irkutsk
   [junit4]   2> NOTE: Linux 3.13.0-32-generic amd64/Oracle Corporation 1.8.0_11 (64-bit)/cpus=8,threads=1,free=141387264,total=340295680
   [junit4]   2> NOTE: All tests run in this JVM: [TestConfig, TestStressUserVersions, AliasIntegrationTest, TestImplicitCoreProperties, CachingDirectoryFactoryTest, TestDocumentBuilder, TestStressVersions, StressHdfsTest, AddBlockUpdateTest, TestQuerySenderListener, UUIDUpdateProcessorFallbackTest, TestFastWriter, OpenExchangeRatesOrgProviderTest, TestDocSet, ShardSplitTest, NotRequiredUniqueKeyTest, TestCollationFieldDocValues, TestLRUCache, TermsComponentTest, DocValuesMultiTest, TestSearchPerf, TestCodecSupport, SignatureUpdateProcessorFactoryTest, TestJmxIntegration, TimeZoneUtilsTest, TestManagedResourceStorage, TestFoldingMultitermQuery, FileUtilsTest, AnalysisErrorHandlingTest, HttpPartitionTest, TestNRTOpen, BJQParserTest, TestSolrDeletionPolicy1, TestStressReorder, TestRandomDVFaceting, TestShortCircuitedRequests, QueryEqualityTest, CurrencyFieldOpenExchangeTest, DistribCursorPagingTest, DocumentBuilderTest, HdfsUnloadDistributedZkTest, DirectUpdateHandlerTest, ShowFileRequestHandlerTest, TestSolrIndexConfig, StandardRequestHandlerTest, FieldFacetExtrasTest, AnalyticsQueryTest, CustomCollectionTest, SuggesterTest, OverseerTest, TestClassNameShortening, TestManagedSchemaFieldResource, DistributedSpellCheckComponentTest, TestSort, TestDFRSimilarityFactory, DisMaxRequestHandlerTest, TestCopyFieldCollectionResource, DocumentAnalysisRequestHandlerTest, IndexBasedSpellCheckerTest, ClusterStateUpdateTest, TestHighFrequencyDictionaryFactory, FieldMutatingUpdateProcessorTest, TestSolrXmlPersistence, ReplicationFactorTest, TestSolrXmlPersistor, UpdateParamsTest, PrimitiveFieldTypeTest, SolrCmdDistributorTest, TestCoreDiscovery, TestReplicationHandler, TestLMJelinekMercerSimilarityFactory, TestSuggestSpellingConverter, TestTrie, WordBreakSolrSpellCheckerTest, HdfsLockFactoryTest, BlockCacheTest, AddSchemaFieldsUpdateProcessorFactoryTest, BadComponentTest, TestDefaultSimilarityFactory, MoreLikeThisHandlerTest, SuggesterTSTTest, DOMUtilTest, TestRemoteStreaming, TestDynamicFieldCollectionResource, NoCacheHeaderTest, XmlUpdateRequestHandlerTest, TestFastOutputStream, TestStressRecovery, TestSimpleQParserPlugin, JsonLoaderTest, BlockDirectoryTest, QueryFacetTest, TestManagedSynonymFilterFactory, SimpleFacetsTest, TestCloudInspectUtil, XsltUpdateRequestHandlerTest, BasicFunctionalityTest, HighlighterConfigTest, HdfsBasicDistributedZkTest, TestRestManager, BasicDistributedZkTest, UnloadDistributedZkTest, SyncSliceTest, ShardRoutingTest, BasicZkTest, RecoveryZkTest, LeaderElectionTest, ZkCLITest, ShardRoutingCustomTest, TestDistributedSearch, TestFaceting, TestRecovery, TestHashPartitioner, ZkControllerTest, TestRealTimeGet, TestMultiCoreConfBootstrap, TestReload, HardAutoCommitTest, TestRangeQuery, TestSolr4Spatial, StatsComponentTest, SpellCheckComponentTest, TestGroupingSearch, PeerSyncTest, BadIndexSchemaTest, ConvertedLegacyTest, TestFunctionQuery, TestLazyCores, TestBadConfig, OverseerCollectionProcessorTest, HighlighterTest, SoftAutoCommitTest, CurrencyFieldXmlFileTest, SolrIndexSplitterTest, AnalysisAfterCoreReloadTest, TestExtendedDismaxParser, SpellCheckCollatorTest, CoreAdminHandlerTest, SolrRequestParserTest, DocValuesTest, TestStressLucene, SpatialFilterTest, SuggesterWFSTTest, SchemaVersionSpecificBehaviorTest, SolrCoreCheckLockOnStartupTest, TestPseudoReturnFields, TestUpdate, TestWordDelimiterFilterFactory, DirectUpdateHandlerOptimizeTest, DefaultValueUpdateProcessorTest, StatelessScriptUpdateProcessorFactoryTest, SolrInfoMBeanTest, TestSurroundQueryParser, DirectSolrSpellCheckerTest, TestQueryTypes, PathHierarchyTokenizerFactoryTest, RequiredFieldsTest, FastVectorHighlighterTest, IndexSchemaTest, UpdateRequestProcessorFactoryTest, JSONWriterTest, TestAnalyzedSuggestions, TestPHPSerializedResponseWriter, TestComponentsName, TestBinaryResponseWriter, ResponseLogComponentTest, SampleTest, MinimalSchemaTest, TestFuzzyAnalyzedSuggestions, ExternalFileFieldSortTest, TestSolrCoreProperties, ChaosMonkeyNothingIsSafeTest, SolrTestCaseJ4Test, TestCursorMarkWithoutUniqueKey, TestHighlightDedupGrouping, TestTolerantSearch, ExpressionTest, RangeFacetTest, DeleteShardTest, MigrateRouteKeyTest, RollingRestartTest, HdfsBasicDistributedZk2Test, HdfsCollectionsAPIDistributedZkTest, HdfsRecoveryZkTest, HdfsSyncSliceTest, HdfsWriteToMultipleCollectionsTest, QueryResultKeyTest, TestConfigSets, TestInfoStreamLogging, TestNonNRTOpen, TestShardHandlerFactory, TestSolrXml, CoreAdminCreateDiscoverTest, CoreAdminRequestStatusTest, DistributedDebugComponentTest, DistributedExpandComponentTest, DistributedQueryComponentCustomSortTest, DistributedQueryComponentOptimizationTest, HighlighterMaxOffsetTest, TestManagedResource, TestFieldCollectionResource, TestFieldResource, TestSchemaNameResource, TestSchemaResource, TestSchemaSimilarityResource, TestSchemaVersionResource, TestUniqueKeyFieldResource, BadCopyFieldTest, ChangedSchemaMergeTest, DateRangeFieldTest, DocValuesMissingTest, EnumFieldTest, ModifyConfFileTest, TestCloudManagedSchema, TestCloudSchemaless]
   [junit4] Completed on J0 in 312.71s, 1 test, 1 error <<< FAILURES!

[...truncated 10 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:474: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:454: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1298: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:922: There were test failures: 421 suites, 1761 tests, 1 error, 39 ignored (18 assumptions)

Total time: 92 minutes 59 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.8.0_11 -XX:+UseCompressedOops -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.8.0_11) - Build # 10979 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/10979/
Java: 32bit/jdk1.8.0_11 -client -XX:+UseG1GC

1 tests failed.
REGRESSION:  org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity

Error Message:
Address already in use

Stack Trace:
java.net.BindException: Address already in use
	at __randomizedtesting.SeedInfo.seed([73DD1B0E9C4E4405:DFC2D514B3CBA9DD]:0)
	at sun.nio.ch.Net.bind0(Native Method)
	at sun.nio.ch.Net.bind(Net.java:414)
	at sun.nio.ch.Net.bind(Net.java:406)
	at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
	at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
	at org.eclipse.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187)
	at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)
	at org.eclipse.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
	at org.eclipse.jetty.server.Server.doStart(Server.java:291)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
	at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:418)
	at org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterStartupWithNoActivity(TestReplicationHandler.java:1107)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11201 lines...]
   [junit4] Suite: org.apache.solr.handler.TestReplicationHandler
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandler-73DD1B0E9C4E4405-001/init-core-data-001
   [junit4]   2> 360954 T935 oas.SolrTestCaseJ4.setUp ###Starting doTestStopPoll
   [junit4]   2> 360959 T935 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 360993 T935 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:42823
   [junit4]   2> 360994 T935 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 360994 T935 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 360995 T935 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandler-73DD1B0E9C4E4405-001/solr-instance-001
   [junit4]   2> 360995 T935 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandler-73DD1B0E9C4E4405-001/solr-instance-001/'
   [junit4]   2> 361019 T935 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandler-73DD1B0E9C4E4405-001/solr-instance-001/solr.xml
   [junit4]   2> 361033 T935 oasc.CoreContainer.<init> New CoreContainer 21136658
   [junit4]   2> 361033 T935 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandler-73DD1B0E9C4E4405-001/solr-instance-001/]
   [junit4]   2> 361034 T935 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 361034 T935 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 361034 T935 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 361034 T935 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 361035 T935 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 361035 T935 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 361035 T935 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 361036 T935 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 361036 T935 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 361036 T935 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 361037 T935 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 361037 T935 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 361037 T935 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 361039 T945 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandler-73DD1B0E9C4E4405-001/solr-instance-001/collection1/'
   [junit4]   2> 361074 T945 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
   [junit4]   2> 361085 T945 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 361086 T945 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 361088 T945 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 361093 T945 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 361093 T945 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from instancedir /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandler-73DD1B0E9C4E4405-001/solr-instance-001/collection1/
   [junit4]   2> 361094 T945 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 361094 T945 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandler-73DD1B0E9C4E4405-001/solr-instance-001/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandler-73DD1B0E9C4E4405-001/solr-instance-001/collection1/data/
   [junit4]   2> 361094 T945 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 361095 T945 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandler-73DD1B0E9C4E4405-001/solr-instance-001/collection1/data
   [junit4]   2> 361095 T945 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandler-73DD1B0E9C4E4405-001/solr-instance-001/collection1/data/index/
   [junit4]   2> 361095 T945 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandler-73DD1B0E9C4E4405-001/solr-instance-001/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 361096 T945 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandler-73DD1B0E9C4E4405-001/solr-instance-001/collection1/data/index
   [junit4]   2> 361096 T945 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=39, maxMergeAtOnceExplicit=41, maxMergedSegmentMB=7.591796875, floorSegmentMB=0.5703125, forceMergeDeletesPctAllowed=24.972670944710128, segmentsPerTier=11.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8189385800850579
   [junit4]   2> 361097 T945 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@ba372 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@d6cc23),segFN=segments_1,generation=1}
   [junit4]   2> 361097 T945 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 361099 T945 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 361099 T945 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 361099 T945 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 361100 T945 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 361100 T945 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 361101 T945 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 361101 T945 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 361101 T945 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
   [junit4]   2> 361102 T945 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
   [junit4]   2> 361102 T945 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
   [junit4]   2> 361102 T945 oasc.RequestHandlers.initHandlersFromConfig WARN Multiple requestHandler registered to the same name: /update ignoring: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 361102 T945 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 361105 T945 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 361107 T945 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 361108 T945 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 361109 T945 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 361110 T945 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 361111 T945 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 361111 T945 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 361111 T945 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=33, maxMergeAtOnceExplicit=45, maxMergedSegmentMB=79.578125, floorSegmentMB=1.1142578125, forceMergeDeletesPctAllowed=3.158288460184311, segmentsPerTier=49.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 361111 T945 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@ba372 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@d6cc23),segFN=segments_1,generation=1}
   [junit4]   2> 361112 T945 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 361112 T945 oass.SolrIndexSearcher.<init> Opening Searcher@175d41d[collection1] main
   [junit4]   2> 361112 T945 oasr.ManagedResourceStorage$FileStorageIO.configure File-based storage initialized to use dir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandler-73DD1B0E9C4E4405-001/solr-instance-001/collection1/conf
   [junit4]   2> 361113 T945 oasr.RestManager.init Initializing RestManager with initArgs: {storageDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandler-73DD1B0E9C4E4405-001/solr-instance-001/collection1/conf}
   [junit4]   2> 361113 T945 oasr.ManagedResourceStorage.load Reading _rest_managed.json using file:dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandler-73DD1B0E9C4E4405-001/solr-instance-001/collection1/conf
   [junit4]   2> 361113 T945 oasr.ManagedResource.reloadFromStorage WARN No stored data found for /rest/managed
   [junit4]   2> 361113 T945 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using file:dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandler-73DD1B0E9C4E4405-001/solr-instance-001/collection1/conf
   [junit4]   2> 361114 T945 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 361114 T945 oash.ReplicationHandler.inform Replication enabled for following config files: schema.xml,xslt/dummy.xsl
   [junit4]   2> 361114 T945 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 361115 T946 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@175d41d[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> 361115 T945 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 361115 T935 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 361116 T935 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 361124 T935 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 361128 T935 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:34297
   [junit4]   2> 361129 T935 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 361130 T935 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 361131 T935 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandler-73DD1B0E9C4E4405-001/solr-instance-002
   [junit4]   2> 361131 T935 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandler-73DD1B0E9C4E4405-001/solr-instance-002/'
   [junit4]   2> 361149 T935 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandler-73DD1B0E9C4E4405-001/solr-instance-002/solr.xml
   [junit4]   2> 361160 T935 oasc.CoreContainer.<init> New CoreContainer 19827030
   [junit4]   2> 361161 T935 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandler-73DD1B0E9C4E4405-001/solr-instance-002/]
   [junit4]   2> 361161 T935 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 361161 T935 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 361162 T935 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 361162 T935 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 361162 T935 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 361163 T935 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 361163 T935 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 361163 T935 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 361163 T935 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 361164 T935 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 361164 T935 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 361165 T935 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 361165 T935 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 361166 T957 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandler-73DD1B0E9C4E4405-001/solr-instance-002/collection1/'
   [junit4]   2> 361200 T957 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
   [junit4]   2> 361227 T957 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 361227 T957 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 361231 T957 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 361233 T957 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 361234 T957 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from instancedir /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandler-73DD1B0E9C4E4405-001/solr-instance-002/collection1/
   [junit4]   2> 361234 T957 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 361234 T957 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandler-73DD1B0E9C4E4405-001/solr-instance-002/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandler-73DD1B0E9C4E4405-001/solr-instance-002/collection1/data/
   [junit4]   2> 361234 T957 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 361235 T957 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandler-73DD1B0E9C4E4405-001/solr-instance-002/collection1/data
   [junit4]   2> 361235 T957 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandler-73DD1B0E9C4E4405-001/solr-instance-002/collection1/data/index/
   [junit4]   2> 361235 T957 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandler-73DD1B0E9C4E4405-001/solr-instance-002/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 361236 T957 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandler-73DD1B0E9C4E4405-001/solr-instance-002/collection1/data/index
   [junit4]   2> 361236 T957 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=39, maxMergeAtOnceExplicit=41, maxMergedSegmentMB=7.591796875, floorSegmentMB=0.5703125, forceMergeDeletesPctAllowed=24.972670944710128, segmentsPerTier=11.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8189385800850579
   [junit4]   2> 361237 T957 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@12596a6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@c33a13),segFN=segments_1,generation=1}
   [junit4]   2> 361237 T957 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 361238 T957 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 361238 T957 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 361238 T957 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 361239 T957 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 361239 T957 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 361239 T957 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 361239 T957 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
   [junit4]   2> 361239 T957 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
   [junit4]   2> 361240 T957 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
   [junit4]   2> 361240 T957 oasc.RequestHandlers.initHandlersFromConfig WARN Multiple requestHandler registered to the same name: /update ignoring: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 361240 T957 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 361240 T957 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 361241 T957 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 361242 T957 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 361243 T957 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 361244 T957 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 361245 T957 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 361246 T957 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 361246 T957 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 361246 T957 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=33, maxMergeAtOnceExplicit=45, maxMergedSegmentMB=79.578125, floorSegmentMB=1.1142578125, forceMergeDeletesPctAllowed=3.158288460184311, segmentsPerTier=49.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 361247 T957 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@12596a6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@c33a13),segFN=segments_1,generation=1}
   [junit4]   2> 361247 T957 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 361247 T957 oass.SolrIndexSearcher.<init> Opening Searcher@116ada7[collection1] main
   [junit4]   2> 361248 T957 oasr.ManagedResourceStorage$FileStorageIO.configure File-based storage initialized to use dir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandler-73DD1B0E9C4E4405-001/solr-instance-002/collection1/conf
   [junit4]   2> 361248 T957 oasr.RestManager.init Initializing RestManager with initArgs: {storageDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandler-73DD1B0E9C4E4405-001/solr-instance-002/collection1/conf}
   [junit4]   2> 361248 T957 oasr.ManagedResourceStorage.load Reading _rest_managed.json using file:dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandler-73DD1B0E9C4E4405-001/solr-instance-002/collection1/conf
   [junit4]   2> 361248 T957 oasr.ManagedResource.reloadFromStorage WARN No stored data found for /rest/managed
   [junit4]   2> 361249 T957 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using file:dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandler-73DD1B0E9C4E4405-001/solr-instance-002/collection1/conf
   [junit4]   2> 361249 T957 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 361251 T957 oash.SnapPuller.startExecutorService Poll Scheduled at an interval of 1000ms
   [junit4]   2> 361251 T957 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 361251 T958 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@116ada7[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> 361251 T935 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 361252 T935 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> ASYNC  NEW_CORE C450 name=collection1 org.apache.solr.core.SolrCore@d7d790
   [junit4]   2> 361260 T941 C450 oasc.SolrCore.execute [collection1] webapp=/solr path=/select params={q=*:*&sort=id+desc&wt=javabin&version=2} hits=0 status=0 QTime=5 
   [junit4]   2> 361265 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[0]} 0 1
   [junit4]   2> 361268 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[1]} 0 0
   [junit4]   2> 361271 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[2]} 0 0
   [junit4]   2> 361275 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[3]} 0 1
   [junit4]   2> 361277 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[4]} 0 0
   [junit4]   2> 361280 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[5]} 0 0
   [junit4]   2> 361283 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[6]} 0 0
   [junit4]   2> 361286 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[7]} 0 0
   [junit4]   2> 361289 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[8]} 0 0
   [junit4]   2> 361292 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[9]} 0 1
   [junit4]   2> 361294 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[10]} 0 0
   [junit4]   2> 361297 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[11]} 0 0
   [junit4]   2> 361300 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[12]} 0 1
   [junit4]   2> 361302 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[13]} 0 0
   [junit4]   2> 361305 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[14]} 0 0
   [junit4]   2> 361308 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[15]} 0 0
   [junit4]   2> 361311 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[16]} 0 0
   [junit4]   2> 361313 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[17]} 0 0
   [junit4]   2> 361316 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[18]} 0 0
   [junit4]   2> 361319 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[19]} 0 0
   [junit4]   2> 361322 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[20]} 0 0
   [junit4]   2> 361325 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[21]} 0 1
   [junit4]   2> 361327 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[22]} 0 0
   [junit4]   2> 361330 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[23]} 0 0
   [junit4]   2> 361333 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[24]} 0 0
   [junit4]   2> 361336 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[25]} 0 0
   [junit4]   2> 361339 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[26]} 0 1
   [junit4]   2> 361341 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[27]} 0 0
   [junit4]   2> 361344 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[28]} 0 0
   [junit4]   2> 361347 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[29]} 0 0
   [junit4]   2> 361350 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[30]} 0 0
   [junit4]   2> 361352 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[31]} 0 0
   [junit4]   2> 361354 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[32]} 0 0
   [junit4]   2> 361357 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[33]} 0 1
   [junit4]   2> 361359 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[34]} 0 0
   [junit4]   2> 361362 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[35]} 0 1
   [junit4]   2> 361364 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[36]} 0 0
   [junit4]   2> 361367 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[37]} 0 1
   [junit4]   2> 361369 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[38]} 0 0
   [junit4]   2> 361371 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[39]} 0 0
   [junit4]   2> 361374 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[40]} 0 0
   [junit4]   2> 361376 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[41]} 0 0
   [junit4]   2> 361379 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[42]} 0 0
   [junit4]   2> 361381 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[43]} 0 0
   [junit4]   2> 361384 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[44]} 0 1
   [junit4]   2> 361386 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[45]} 0 0
   [junit4]   2> 361389 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[46]} 0 1
   [junit4]   2> 361391 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[47]} 0 0
   [junit4]   2> 361393 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[48]} 0 0
   [junit4]   2> 361396 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[49]} 0 0
   [junit4]   2> 361399 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[50]} 0 1
   [junit4]   2> 361401 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[51]} 0 0
   [junit4]   2> 361404 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[52]} 0 1
   [junit4]   2> 361406 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[53]} 0 0
   [junit4]   2> 361409 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[54]} 0 0
   [junit4]   2> 361411 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[55]} 0 0
   [junit4]   2> 361413 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[56]} 0 0
   [junit4]   2> 361415 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[57]} 0 0
   [junit4]   2> 361417 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[58]} 0 0
   [junit4]   2> 361420 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[59]} 0 1
   [junit4]   2> 361422 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[60]} 0 1
   [junit4]   2> 361424 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[61]} 0 0
   [junit4]   2> 361426 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[62]} 0 0
   [junit4]   2> 361428 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[63]} 0 0
   [junit4]   2> 361430 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[64]} 0 0
   [junit4]   2> 361432 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[65]} 0 0
   [junit4]   2> 361434 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[66]} 0 0
   [junit4]   2> 361436 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[67]} 0 0
   [junit4]   2> 361438 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[68]} 0 0
   [junit4]   2> 361441 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[69]} 0 0
   [junit4]   2> 361443 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[70]} 0 0
   [junit4]   2> 361445 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[71]} 0 0
   [junit4]   2> 361447 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[72]} 0 0
   [junit4]   2> 361449 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[73]} 0 0
   [junit4]   2> 361452 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[74]} 0 0
   [junit4]   2> 361454 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[75]} 0 0
   [junit4]   2> 361456 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[76]} 0 0
   [junit4]   2> 361458 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[77]} 0 0
   [junit4]   2> 361461 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[78]} 0 0
   [junit4]   2> 361465 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[79]} 0 0
   [junit4]   2> 361468 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[80]} 0 0
   [junit4]   2> 361471 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[81]} 0 1
   [junit4]   2> 361473 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[82]} 0 0
   [junit4]   2> 361476 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[83]} 0 0
   [junit4]   2> 361479 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[84]} 0 0
   [junit4]   2> 361482 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[85]} 0 0
   [junit4]   2> 361484 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[86]} 0 0
   [junit4]   2> 361487 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[87]} 0 0
   [junit4]   2> 361490 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[88]} 0 0
   [junit4]   2> 361493 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[89]} 0 0
   [junit4]   2> 361496 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[90]} 0 0
   [junit4]   2> 361499 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[91]} 0 0
   [junit4]   2> 361501 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[92]} 0 0
   [junit4]   2> 361504 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[93]} 0 0
   [junit4]   2> 361507 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[94]} 0 0
   [junit4]   2> 361510 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[95]} 0 0
   [junit4]   2> 361513 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[96]} 0 0
   [junit4]   2> 361516 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[97]} 0 1
   [junit4]   2> 361518 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[98]} 0 0
   [junit4]   2> 361521 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[99]} 0 0
   [junit4]   2> 361524 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[100]} 0 0
   [junit4]   2> 361527 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[101]} 0 0
   [junit4]   2> 361530 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[102]} 0 0
   [junit4]   2> 361532 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[103]} 0 0
   [junit4]   2> 361535 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[104]} 0 0
   [junit4]   2> 361538 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[105]} 0 1
   [junit4]   2> 361540 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[106]} 0 0
   [junit4]   2> 361543 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[107]} 0 1
   [junit4]   2> 361545 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[108]} 0 0
   [junit4]   2> 361547 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[109]} 0 0
   [junit4]   2> 361549 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[110]} 0 0
   [junit4]   2> 361552 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[111]} 0 1
   [junit4]   2> 361554 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[112]} 0 0
   [junit4]   2> 361557 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[113]} 0 0
   [junit4]   2> 361560 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[114]} 0 0
   [junit4]   2> 361563 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[115]} 0 0
   [junit4]   2> 361566 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[116]} 0 1
   [junit4]   2> 361569 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[117]} 0 1
   [junit4]   2> 361571 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[118]} 0 0
   [junit4]   2> 361574 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[119]} 0 0
   [junit4]   2> 361576 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[120]} 0 0
   [junit4]   2> 361579 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[121]} 0 1
   [junit4]   2> 361581 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[122]} 0 0
   [junit4]   2> 361584 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[123]} 0 0
   [junit4]   2> 361586 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[124]} 0 0
   [junit4]   2> 361589 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[125]} 0 0
   [junit4]   2> 361591 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[126]} 0 0
   [junit4]   2> 361594 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[127]} 0 1
   [junit4]   2> 361596 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[128]} 0 0
   [junit4]   2> 361598 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[129]} 0 0
   [junit4]   2> 361601 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[130]} 0 1
   [junit4]   2> 361603 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[131]} 0 0
   [junit4]   2> 361605 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[132]} 0 0
   [junit4]   2> 361607 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[133]} 0 0
   [junit4]   2> 361609 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[134]} 0 0
   [junit4]   2> 361612 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[135]} 0 0
   [junit4]   2> 361614 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[136]} 0 0
   [junit4]   2> 361616 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[137]} 0 0
   [junit4]   2> 361619 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[138]} 0 1
   [junit4]   2> 361621 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[139]} 0 0
   [junit4]   2> 361624 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[140]} 0 0
   [junit4]   2> 361627 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[141]} 0 0
   [junit4]   2> 361630 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[142]} 0 1
   [junit4]   2> 361632 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[143]} 0 0
   [junit4]   2> 361635 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[144]} 0 0
   [junit4]   2> 361638 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[145]} 0 1
   [junit4]   2> 361640 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[146]} 0 0
   [junit4]   2> 361643 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[147]} 0 1
   [junit4]   2> 361645 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[148]} 0 0
   [junit4]   2> 361648 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[149]} 0 0
   [junit4]   2> 361651 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[150]} 0 0
   [junit4]   2> 361652 T941 C450 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0 
   [junit4]   2> 361654 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[151]} 0 0
   [junit4]   2> 361657 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[152]} 0 1
   [junit4]   2> 361659 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[153]} 0 0
   [junit4]   2> 361662 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[154]} 0 0
   [junit4]   2> 361665 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[155]} 0 0
   [junit4]   2> 361668 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[156]} 0 0
   [junit4]   2> 361671 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[157]} 0 0
   [junit4]   2> 361674 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[158]} 0 0
   [junit4]   2> 361677 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[159]} 0 1
   [junit4]   2> 361679 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[160]} 0 0
   [junit4]   2> 361681 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[161]} 0 0
   [junit4]   2> 361683 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[162]} 0 0
   [junit4]   2> 361686 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[163]} 0 0
   [junit4]   2> 361689 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[164]} 0 0
   [junit4]   2> 361692 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[165]} 0 0
   [junit4]   2> 361695 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[166]} 0 0
   [junit4]   2> 361698 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[167]} 0 1
   [junit4]   2> 361701 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[168]} 0 0
   [junit4]   2> 361703 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[169]} 0 0
   [junit4]   2> 361706 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[170]} 0 0
   [junit4]   2> 361709 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[171]} 0 1
   [junit4]   2> 361711 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[172]} 0 0
   [junit4]   2> 361713 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[173]} 0 0
   [junit4]   2> 361715 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[174]} 0 0
   [junit4]   2> 361717 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[175]} 0 0
   [junit4]   2> 361720 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[176]} 0 1
   [junit4]   2> 361722 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[177]} 0 0
   [junit4]   2> 361725 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[178]} 0 0
   [junit4]   2> 361727 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[179]} 0 0
   [junit4]   2> 361729 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[180]} 0 0
   [junit4]   2> 361731 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[181]} 0 0
   [junit4]   2> 361733 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[182]} 0 0
   [junit4]   2> 361735 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[183]} 0 0
   [junit4]   2> 361738 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[184]} 0 0
   [junit4]   2> 361740 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[185]} 0 0
   [junit4]   2> 361742 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[186]} 0 0
   [junit4]   2> 361745 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[187]} 0 0
   [junit4]   2> 361747 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[188]} 0 0
   [junit4]   2> 361750 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[189]} 0 0
   [junit4]   2> 361752 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[190]} 0 0
   [junit4]   2> 361754 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[191]} 0 0
   [junit4]   2> 361756 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[192]} 0 0
   [junit4]   2> 361759 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[193]} 0 1
   [junit4]   2> 361761 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[194]} 0 0
   [junit4]   2> 361763 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[195]} 0 0
   [junit4]   2> 361765 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[196]} 0 0
   [junit4]   2> 361768 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[197]} 0 0
   [junit4]   2> 361770 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[198]} 0 0
   [junit4]   2> 361772 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[199]} 0 0
   [junit4]   2> 361774 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[200]} 0 0
   [junit4]   2> 361776 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[201]} 0 0
   [junit4]   2> 361778 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[202]} 0 0
   [junit4]   2> 361781 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[203]} 0 1
   [junit4]   2> 361783 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[204]} 0 1
   [junit4]   2> 361785 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[205]} 0 0
   [junit4]   2> 361787 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[206]} 0 0
   [junit4]   2> 361789 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[207]} 0 1
   [junit4]   2> 361791 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[208]} 0 1
   [junit4]   2> 361793 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[209]} 0 1
   [junit4]   2> 361795 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[210]} 0 0
   [junit4]   2> 361797 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[211]} 0 0
   [junit4]   2> 361799 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[212]} 0 0
   [junit4]   2> 361801 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[213]} 0 0
   [junit4]   2> 361803 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[214]} 0 0
   [junit4]   2> 361805 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[215]} 0 0
   [junit4]   2> 361807 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[216]} 0 0
   [junit4]   2> 361809 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[217]} 0 0
   [junit4]   2> 361811 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[218]} 0 0
   [junit4]   2> 361813 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[219]} 0 0
   [junit4]   2> 361815 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[220]} 0 0
   [junit4]   2> 361817 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[221]} 0 0
   [junit4]   2> 361819 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[222]} 0 0
   [junit4]   2> 361822 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[223]} 0 1
   [junit4]   2> 361824 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[224]} 0 0
   [junit4]   2> 361826 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[225]} 0 0
   [junit4]   2> 361829 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[226]} 0 0
   [junit4]   2> 361831 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[227]} 0 0
   [junit4]   2> 361834 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[228]} 0 0
   [junit4]   2> 361836 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[229]} 0 0
   [junit4]   2> 361838 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[230]} 0 0
   [junit4]   2> 361841 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[231]} 0 1
   [junit4]   2> 361843 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[232]} 0 0
   [junit4]   2> 361846 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[233]} 0 1
   [junit4]   2> 361848 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[234]} 0 0
   [junit4]   2> 361850 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[235]} 0 0
   [junit4]   2> 361853 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[236]} 0 0
   [junit4]   2> 361856 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[237]} 0 0
   [junit4]   2> 361859 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[238]} 0 0
   [junit4]   2> 361861 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[239]} 0 0
   [junit4]   2> 361864 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[240]} 0 0
   [junit4]   2> 361867 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[241]} 0 1
   [junit4]   2> 361869 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[242]} 0 0
   [junit4]   2> 361872 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[243]} 0 0
   [junit4]   2> 361875 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[244]} 0 1
   [junit4]   2> 361877 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[245]} 0 0
   [junit4]   2> 361880 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[246]} 0 0
   [junit4]   2> 361883 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[247]} 0 1
   [junit4]   2> 361885 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[248]} 0 0
   [junit4]   2> 361888 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[249]} 0 0
   [junit4]   2> 361891 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[250]} 0 1
   [junit4]   2> 361893 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[251]} 0 0
   [junit4]   2> 361895 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[252]} 0 0
   [junit4]   2> 361898 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[253]} 0 0
   [junit4]   2> 361901 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[254]} 0 1
   [junit4]   2> 361903 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[255]} 0 0
   [junit4]   2> 361906 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[256]} 0 0
   [junit4]   2> 361908 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[257]} 0 0
   [junit4]   2> 361911 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[258]} 0 0
   [junit4]   2> 361913 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[259]} 0 0
   [junit4]   2> 361916 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[260]} 0 1
   [junit4]   2> 361918 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[261]} 0 0
   [junit4]   2> 361920 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[262]} 0 0
   [junit4]   2> 361923 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[263]} 0 1
   [junit4]   2> 361925 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[264]} 0 0
   [junit4]   2> 361927 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[265]} 0 0
   [junit4]   2> 361930 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[266]} 0 0
   [junit4]   2> 361932 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[267]} 0 0
   [junit4]   2> 361934 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[268]} 0 0
   [junit4]   2> 361937 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[269]} 0 0
   [junit4]   2> 361940 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[270]} 0 1
   [junit4]   2> 361942 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[271]} 0 0
   [junit4]   2> 361945 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[272]} 0 0
   [junit4]   2> 361948 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[273]} 0 1
   [junit4]   2> 361950 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[274]} 0 0
   [junit4]   2> 361953 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[275]} 0 0
   [junit4]   2> 361955 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[276]} 0 0
   [junit4]   2> 361958 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[277]} 0 0
   [junit4]   2> 361961 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[278]} 0 1
   [junit4]   2> 361963 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[279]} 0 0
   [junit4]   2> 361966 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[280]} 0 0
   [junit4]   2> 361969 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[281]} 0 0
   [junit4]   2> 361972 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[282]} 0 0
   [junit4]   2> 361974 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[283]} 0 0
   [junit4]   2> 361977 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[284]} 0 0
   [junit4]   2> 361979 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[285]} 0 0
   [junit4]   2> 361981 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[286]} 0 0
   [junit4]   2> 361984 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[287]} 0 0
   [junit4]   2> 361986 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[288]} 0 0
   [junit4]   2> 361988 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[289]} 0 0
   [junit4]   2> 361990 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[290]} 0 0
   [junit4]   2> 361993 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[291]} 0 0
   [junit4]   2> 361996 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[292]} 0 0
   [junit4]   2> 361999 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[293]} 0 0
   [junit4]   2> 362002 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[294]} 0 1
   [junit4]   2> 362004 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[295]} 0 0
   [junit4]   2> 362007 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[296]} 0 0
   [junit4]   2> 362010 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[297]} 0 0
   [junit4]   2> 362012 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[298]} 0 0
   [junit4]   2> 362015 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[299]} 0 1
   [junit4]   2> 362017 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[300]} 0 0
   [junit4]   2> 362019 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[301]} 0 0
   [junit4]   2> 362021 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[302]} 0 0
   [junit4]   2> 362024 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[303]} 0 0
   [junit4]   2> 362026 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[304]} 0 0
   [junit4]   2> 362028 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[305]} 0 0
   [junit4]   2> 362031 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[306]} 0 1
   [junit4]   2> 362033 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[307]} 0 0
   [junit4]   2> 362035 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[308]} 0 0
   [junit4]   2> 362037 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[309]} 0 0
   [junit4]   2> 362039 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[310]} 0 0
   [junit4]   2> 362042 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[311]} 0 0
   [junit4]   2> 362044 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[312]} 0 0
   [junit4]   2> 362046 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[313]} 0 0
   [junit4]   2> 362048 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[314]} 0 0
   [junit4]   2> 362051 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[315]} 0 0
   [junit4]   2> 362054 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[316]} 0 0
   [junit4]   2> 362057 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[317]} 0 0
   [junit4]   2> 362059 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[318]} 0 0
   [junit4]   2> 362062 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[319]} 0 0
   [junit4]   2> 362065 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[320]} 0 0
   [junit4]   2> 362068 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[321]} 0 0
   [junit4]   2> 362071 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[322]} 0 0
   [junit4]   2> 362074 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[323]} 0 0
   [junit4]   2> 362077 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[324]} 0 0
   [junit4]   2> 362080 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[325]} 0 0
   [junit4]   2> 362083 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[326]} 0 0
   [junit4]   2> 362086 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[327]} 0 0
   [junit4]   2> 362089 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[328]} 0 0
   [junit4]   2> 362091 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[329]} 0 0
   [junit4]   2> 362094 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[330]} 0 0
   [junit4]   2> 362097 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[331]} 0 0
   [junit4]   2> 362100 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[332]} 0 0
   [junit4]   2> 362103 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[333]} 0 0
   [junit4]   2> 362106 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[334]} 0 1
   [junit4]   2> 362109 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[335]} 0 0
   [junit4]   2> 362112 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[336]} 0 1
   [junit4]   2> 362114 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[337]} 0 0
   [junit4]   2> 362117 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[338]} 0 0
   [junit4]   2> 362120 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[339]} 0 0
   [junit4]   2> 362123 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[340]} 0 0
   [junit4]   2> 362126 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[341]} 0 0
   [junit4]   2> 362129 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[342]} 0 0
   [junit4]   2> 362132 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[343]} 0 0
   [junit4]   2> 362134 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[344]} 0 0
   [junit4]   2> 362136 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[345]} 0 0
   [junit4]   2> 362138 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[346]} 0 0
   [junit4]   2> 362142 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[347]} 0 1
   [junit4]   2> 362145 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[348]} 0 0
   [junit4]   2> 362148 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[349]} 0 1
   [junit4]   2> 362150 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[350]} 0 0
   [junit4]   2> 362152 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[351]} 0 0
   [junit4]   2> 362155 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[352]} 0 1
   [junit4]   2> 362157 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[353]} 0 0
   [junit4]   2> 362159 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[354]} 0 0
   [junit4]   2> 362162 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[355]} 0 1
   [junit4]   2> 362164 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[356]} 0 0
   [junit4]   2> 362167 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[357]} 0 0
   [junit4]   2> 362170 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[358]} 0 0
   [junit4]   2> 362172 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[359]} 0 0
   [junit4]   2> 362174 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[360]} 0 0
   [junit4]   2> 362176 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[361]} 0 0
   [junit4]   2> 362179 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[362]} 0 0
   [junit4]   2> 362181 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[363]} 0 0
   [junit4]   2> 362185 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[364]} 0 1
   [junit4]   2> 362187 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[365]} 0 0
   [junit4]   2> 362190 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[366]} 0 0
   [junit4]   2> 362193 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[367]} 0 1
   [junit4]   2> 362195 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[368]} 0 0
   [junit4]   2> 362198 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[369]} 0 1
   [junit4]   2> 362200 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[370]} 0 0
   [junit4]   2> 362203 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[371]} 0 0
   [junit4]   2> 362205 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[372]} 0 0
   [junit4]   2> 362208 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[373]} 0 0
   [junit4]   2> 362211 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[374]} 0 1
   [junit4]   2> 362213 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[375]} 0 0
   [junit4]   2> 362216 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[376]} 0 0
   [junit4]   2> 362219 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[377]} 0 0
   [junit4]   2> 362221 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[378]} 0 0
   [junit4]   2> 362224 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[379]} 0 0
   [junit4]   2> 362227 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[380]} 0 1
   [junit4]   2> 362229 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[381]} 0 0
   [junit4]   2> 362232 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[382]} 0 0
   [junit4]   2> 362235 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[383]} 0 1
   [junit4]   2> 362237 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[384]} 0 0
   [junit4]   2> 362240 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[385]} 0 1
   [junit4]   2> 362242 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[386]} 0 0
   [junit4]   2> 362244 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[387]} 0 0
   [junit4]   2> 362247 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[388]} 0 0
   [junit4]   2> 362249 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[389]} 0 0
   [junit4]   2> 362251 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[390]} 0 0
   [junit4]   2> 362254 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[391]} 0 1
   [junit4]   2> 362256 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[392]} 0 0
   [junit4]   2> 362258 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[393]} 0 0
   [junit4]   2> 362260 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[394]} 0 0
   [junit4]   2> 362263 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[395]} 0 1
   [junit4]   2> 362265 T941 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[396]} 0 0
   [junit4]   2> 362267 T944 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[397]} 0 0
   [junit4]   2> 362269 T943 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[398]} 0 0
   [junit4]   2> 362272 T942 C450 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&

[...truncated too long message...]

nt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandler-73DD1B0E9C4E4405-001/solr-instance-002/collection1/data/index.20140808195033009
   [junit4]   2> 606778 T1571 C515 oasc.SolrCore.execute [collection1] webapp=/solr path=/select params={q=*:*&sort=id+desc&wt=javabin&version=2} hits=493 status=0 QTime=1 
   [junit4]   2> 606779 T935 oash.TestReplicationHandler.rQuery Waited for 600ms and found 493 docs
   [junit4]   2> 606780 T935 oas.SolrTestCaseJ4.tearDown ###Ending doTestReplicateAfterStartup
   [junit4]   2> 606805 T935 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1864257
   [junit4]   2> 606807 T935 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@c6eba6
   [junit4]   2> 606807 T935 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=493,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
   [junit4]   2> 606809 T935 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 606810 T935 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 606811 T935 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 606813 T935 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 606814 T935 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 606814 T935 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandler-73DD1B0E9C4E4405-001/solr-instance-001/collection1/data/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandler-73DD1B0E9C4E4405-001/solr-instance-001/collection1/data/index;done=false>>]
   [junit4]   2> 606815 T935 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandler-73DD1B0E9C4E4405-001/solr-instance-001/collection1/data/index
   [junit4]   2> 606815 T935 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandler-73DD1B0E9C4E4405-001/solr-instance-001/collection1/data [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandler-73DD1B0E9C4E4405-001/solr-instance-001/collection1/data;done=false>>]
   [junit4]   2> 606816 T935 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandler-73DD1B0E9C4E4405-001/solr-instance-001/collection1/data
   [junit4]   2> 606817 T935 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
   [junit4]   2> 607649 T1576 oash.SnapPuller.fetchLatestIndex ERROR Master at: http://127.0.0.1:51691/solr is not available. Index fetch failed. Exception: Server refused connection at: http://127.0.0.1:51691/solr
   [junit4]   2> 608649 T1576 oash.SnapPuller.fetchLatestIndex ERROR Master at: http://127.0.0.1:51691/solr is not available. Index fetch failed. Exception: Server refused connection at: http://127.0.0.1:51691/solr
   [junit4]   2> 609649 T1576 oash.SnapPuller.fetchLatestIndex ERROR Master at: http://127.0.0.1:51691/solr is not available. Index fetch failed. Exception: Server refused connection at: http://127.0.0.1:51691/solr
   [junit4]   2> 610649 T1576 oash.SnapPuller.fetchLatestIndex ERROR Master at: http://127.0.0.1:51691/solr is not available. Index fetch failed. Exception: Server refused connection at: http://127.0.0.1:51691/solr
   [junit4]   2> 611649 T1576 oash.SnapPuller.fetchLatestIndex ERROR Master at: http://127.0.0.1:51691/solr is not available. Index fetch failed. Exception: Server refused connection at: http://127.0.0.1:51691/solr
   [junit4]   2> 611818 T935 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=30376754
   [junit4]   2> 611819 T935 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@a623ab
   [junit4]   2> 611820 T935 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
   [junit4]   2> 611821 T935 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 611821 T935 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 611822 T935 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 611824 T935 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 611824 T935 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 611825 T935 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandler-73DD1B0E9C4E4405-001/solr-instance-002/collection1/data/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandler-73DD1B0E9C4E4405-001/solr-instance-002/collection1/data/index;done=false>>]
   [junit4]   2> 611825 T935 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandler-73DD1B0E9C4E4405-001/solr-instance-002/collection1/data/index
   [junit4]   2> 611825 T935 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandler-73DD1B0E9C4E4405-001/solr-instance-002/collection1/data [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandler-73DD1B0E9C4E4405-001/solr-instance-002/collection1/data;done=false>>]
   [junit4]   2> 611826 T935 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandler-73DD1B0E9C4E4405-001/solr-instance-002/collection1/data
   [junit4]   2> 611827 T935 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
   [junit4]   2> 616812 T935 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandler-73DD1B0E9C4E4405-001
   [junit4]   2> NOTE: test params are: codec=Lucene46, sim=DefaultSimilarity, locale=da_DK, timezone=Africa/Gaborone
   [junit4]   2> NOTE: Linux 3.13.0-32-generic i386/Oracle Corporation 1.8.0_11 (32-bit)/cpus=8,threads=1,free=73287960,total=320864256
   [junit4]   2> NOTE: All tests run in this JVM: [TestSolr4Spatial, DirectUpdateHandlerTest, TestBlendedInfixSuggestions, SolrCoreCheckLockOnStartupTest, TestModifyConfFiles, HdfsWriteToMultipleCollectionsTest, DirectSolrSpellCheckerTest, TimeZoneUtilsTest, ReturnFieldsTest, HdfsCollectionsAPIDistributedZkTest, TestStressUserVersions, NoCacheHeaderTest, TestMaxScoreQueryParser, FastVectorHighlighterTest, HdfsBasicDistributedZkTest, BasicDistributedZkTest, DocValuesMissingTest, AssignTest, TestArbitraryIndexDir, SystemInfoHandlerTest, PluginInfoTest, TestConfig, TestRTGBase, AsyncMigrateRouteKeyTest, TestLMDirichletSimilarityFactory, TestFuzzyAnalyzedSuggestions, TestSimpleQParserPlugin, HighlighterMaxOffsetTest, TestManagedResourceStorage, TestRestManager, TestHashPartitioner, TestRangeQuery, ClusterStateUpdateTest, TestReplicationHandler]
   [junit4] Completed on J0 in 255.90s, 13 tests, 1 error <<< FAILURES!

[...truncated 1067 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:474: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:454: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1298: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:922: There were test failures: 421 suites, 1761 tests, 1 error, 39 ignored (18 assumptions)

Total time: 97 minutes 11 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 32bit/jdk1.8.0_11 -client -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_11) - Build # 10978 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/10978/
Java: 64bit/jdk1.8.0_11 -XX:+UseCompressedOops -XX:+UseSerialGC

All tests passed

Build Log:
[...truncated 44662 lines...]
-documentation-lint:
     [echo] checking for broken html...
    [jtidy] Checking for broken html (such as invalid tags)...
   [delete] Deleting directory /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/build/jtidy_tmp
     [echo] Checking for broken links...
     [exec] 
     [exec] Crawl/parse...
     [exec] 
     [exec] Verify...
     [echo] Checking for missing docs...
     [exec] Traceback (most recent call last):
     [exec]   File "/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/dev-tools/scripts/checkJavaDocs.py", line 371, in <module>
     [exec]     if checkPackageSummaries(sys.argv[1], level):
     [exec]   File "/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/dev-tools/scripts/checkJavaDocs.py", line 351, in checkPackageSummaries
     [exec]     if checkClassSummaries(fullPath):
     [exec]   File "/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/dev-tools/scripts/checkJavaDocs.py", line 215, in checkClassSummaries
     [exec]     missing.append((lastCaption, unEscapeURL(lastItem)))
     [exec]   File "/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/dev-tools/scripts/checkJavaDocs.py", line 303, in unEscapeURL
     [exec]     s = s.replace('%20', ' ')
     [exec] AttributeError: 'NoneType' object has no attribute 'replace'

BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:474: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:63: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/build.xml:212: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/build.xml:242: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:2347: exec returned: 1

Total time: 102 minutes 36 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.8.0_11 -XX:+UseCompressedOops -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any