You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/05/30 11:13:57 UTC
[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_60-ea-b15) -
Build # 10426 - Failure!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/10426/
Java: 32bit/jdk1.7.0_60-ea-b15 -server -XX:+UseSerialGC
1 tests failed.
REGRESSION: org.apache.solr.cloud.OverseerTest.testOverseerFailure
Error Message:
Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed
Stack Trace:
org.apache.solr.common.SolrException: Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed
at __randomizedtesting.SeedInfo.seed([A1FD9A535D55EE7C:A5F515A04FF0015D]:0)
at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:135)
at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:142)
at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:110)
at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:134)
at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:267)
at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:155)
at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:661)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1
at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
at org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:432)
at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:429)
at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:386)
at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:373)
at org.apache.solr.cloud.ShardLeaderElectionContextBase$1.execute(ElectionContext.java:127)
at org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:31)
at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:122)
... 46 more
Build Log:
[...truncated 11382 lines...]
[junit4] Suite: org.apache.solr.cloud.OverseerTest
[junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.OverseerTest-A1FD9A535D55EE7C-001/init-core-data-001
[junit4] 2> 1198144 T3980 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
[junit4] 2> 1198144 T3980 oas.SolrTestCaseJ4.initCore ####initCore
[junit4] 2> 1198144 T3980 oas.SolrTestCaseJ4.initCore ####initCore end
[junit4] 2> 1198147 T3980 oas.SolrTestCaseJ4.setUp ###Starting testOverseerFailure
[junit4] 2> 1198147 T3980 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 1198148 T3981 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 1198248 T3980 oasc.ZkTestServer.run start zk server on port:49979
[junit4] 2> 1198249 T3980 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1211028 T3984 oazsp.FileTxnLog.commit WARN fsync-ing the write ahead log in SyncThread:0 took 12778ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
[junit4] 2> 1211030 T3982 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x1464c59ecad0000, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2>
[junit4] 2> 1211041 T3987 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@117ce6e name:ZooKeeperConnection Watcher:127.0.0.1:49979 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1211041 T3980 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1211044 T3980 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1211045 T3989 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@8cb747 name:ZooKeeperConnection Watcher:127.0.0.1:49979 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1211046 T3980 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1211046 T3980 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 1211049 T3980 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1211050 T3991 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1784f21 name:ZooKeeperConnection Watcher:127.0.0.1:49979/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1211050 T3980 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1211051 T3980 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 1211052 T3980 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 1211054 T3980 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 1211055 T3980 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1211058 T3980 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1211059 T3993 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1fa59e9 name:ZooKeeperConnection Watcher:127.0.0.1:49979/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1211060 T3980 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1211061 T3980 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1211063 T3980 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
[junit4] 2> 1211065 T3980 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1211066 T3995 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@121e1a2 name:ZooKeeperConnection Watcher:127.0.0.1:49979/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1211067 T3980 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1211067 T3980 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 1211069 T3980 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 1211071 T3980 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:49979_solr
[junit4] 2> 1211071 T3980 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 1211073 T3980 oasc.Overseer.start Overseer (id=91844791514234885-127.0.0.1:49979_solr-n_0000000000) starting
[junit4] 2> 1211074 T3980 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 1211076 T3980 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 1211078 T3980 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 1211080 T3980 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 1211081 T3980 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 1211084 T3980 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 1211087 T3997 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 1211088 T3996 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 1212090 T3995 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1212091 T3996 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1212092 T3996 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 1212092 T3996 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1]
[junit4] 2> 1212092 T3996 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1212094 T3995 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1212095 T3991 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1212095 T3993 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1212590 T3980 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1212598 T3980 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 1212602 T3995 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1212604 T3980 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1212605 T3996 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1212607 T3980 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1212608 T3995 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1212609 T3996 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 1212609 T3996 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
[junit4] 2> 1212612 T3995 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1212714 T3991 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1212715 T3993 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1213089 T3997 oasc.OverseerCollectionProcessor.run WARN Overseer cannot talk to ZK
[junit4] 2> 1213788 T3980 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1213793 T3980 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1213794 T3999 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a33ce3 name:ZooKeeperConnection Watcher:127.0.0.1:49979/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1213795 T3980 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1213795 T3996 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:49979_solr
[junit4] 2> 1213797 T4000 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:306)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:289)
[junit4] 2>
[junit4] 2> 1213800 T3980 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:49979_solr
[junit4] 2> 1213801 T3980 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 1213802 T3980 oasc.Overseer.start Overseer (id=91844791514234886-127.0.0.1:49979_solr-n_0000000001) starting
[junit4] 2> 1213826 T4002 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 1213827 T4001 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 1213836 T4001 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1213837 T4001 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 1213837 T4001 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
[junit4] 2> 1213840 T3999 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1213841 T3991 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1213841 T3993 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1213893 T3999 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1213893 T3980 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1213895 T4001 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1213897 T3980 oasc.LeaderElector.checkIfIamLeader WARN Failed setting watch org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /collections/collection1/leader_elect/shard1/election/91844791514234884-node1_core1-n_0000000002
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:127)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:267)
[junit4] 2> at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:155)
[junit4] 2> at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:661)
[junit4] 2> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit4] 2> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
[junit4] 2> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit4] 2> at java.lang.reflect.Method.invoke(Method.java:606)
[junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
[junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
[junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
[junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
[junit4] 2> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
[junit4] 2> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4] 2> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
[junit4] 2> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
[junit4] 2> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
[junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
[junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
[junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
[junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
[junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
[junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
[junit4] 2> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
[junit4] 2> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4] 2> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
[junit4] 2> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4] 2> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
[junit4] 2> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2>
[junit4] 2> 1213898 T3993 oasc.LeaderElector.checkIfIamLeader WARN Our node is no longer in line to be leader
[junit4] 2> 1213900 T4001 oasc.SolrException.log ERROR Problem cleaning up collection in zk:collection1:org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /collections/collection1/leader_elect/shard1/election
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:125)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:177)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:174)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.delete(SolrZkClient.java:174)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.clean(SolrZkClient.java:631)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.clean(SolrZkClient.java:626)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.clean(SolrZkClient.java:626)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.clean(SolrZkClient.java:626)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.removeCore(Overseer.java:1066)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.processMessage(Overseer.java:346)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:242)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2>
[junit4] 2> 1213900 T3993 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 1213900 T3980 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 1213903 T3999 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1213904 T3999 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1213904 T4001 oasc.Overseer$ClusterStateUpdater.setShardLeader ERROR Could not mark shard leader for non existing collection:collection1
[junit4] 2> 1213906 T3999 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1214008 T3991 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1214008 T3993 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1214902 T3980 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 1215905 T3980 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 1216908 T3980 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 1217911 T3980 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 1218914 T3980 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 1219917 T3980 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 1220920 T3980 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 1221923 T3980 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 1222926 T3980 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 1223929 T3980 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 1224933 T3980 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 1225936 T3980 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 1226940 T3980 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 1227943 T3980 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 1228946 T3980 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 1228952 T3980 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:49979 49979
[junit4] 2> 1229846 T4002 oasc.OverseerCollectionProcessor.run WARN Overseer cannot talk to ZK
[junit4] 2> 1239291 T3981 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:49979 49979
[junit4] 2> 1239291 T3980 oas.SolrTestCaseJ4.tearDown ###Ending testOverseerFailure
[junit4] 2> 1239292 T4001 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:49979_solr
[junit4] 2> 1239293 T4003 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:306)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:289)
[junit4] 2>
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=OverseerTest -Dtests.method=testOverseerFailure -Dtests.seed=A1FD9A535D55EE7C -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=es_HN -Dtests.timezone=America/North_Dakota/Center -Dtests.file.encoding=UTF-8
[junit4] ERROR 41.2s J1 | OverseerTest.testOverseerFailure <<<
[junit4] > Throwable #1: org.apache.solr.common.SolrException: Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed
[junit4] > at __randomizedtesting.SeedInfo.seed([A1FD9A535D55EE7C:A5F515A04FF0015D]:0)
[junit4] > at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:135)
[junit4] > at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:142)
[junit4] > at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:110)
[junit4] > at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:134)
[junit4] > at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:267)
[junit4] > at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:155)
[junit4] > at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:661)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] > Caused by: org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1
[junit4] > at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
[junit4] > at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] > at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
[junit4] > at org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:432)
[junit4] > at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] > at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:429)
[junit4] > at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:386)
[junit4] > at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:373)
[junit4] > at org.apache.solr.cloud.ShardLeaderElectionContextBase$1.execute(ElectionContext.java:127)
[junit4] > at org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:31)
[junit4] > at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:122)
[junit4] > ... 46 more
[junit4] 2> 1239309 T3980 oas.SolrTestCaseJ4.setUp ###Starting testShardAssignment
[junit4] 2> 1239309 T3980 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 1239310 T4004 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 1239410 T3980 oasc.ZkTestServer.run start zk server on port:40306
[junit4] 2> 1239410 T3980 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1239413 T4010 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@27124a name:ZooKeeperConnection Watcher:127.0.0.1:40306 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1239413 T3980 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1239415 T3980 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1239416 T4012 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@14f0143 name:ZooKeeperConnection Watcher:127.0.0.1:40306 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1239416 T3980 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1239416 T3980 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 1239419 T3980 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1239420 T4014 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@bd02ef name:ZooKeeperConnection Watcher:127.0.0.1:40306/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1239420 T3980 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1239420 T3980 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 1239422 T3980 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1239423 T4016 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@42a2de name:ZooKeeperConnection Watcher:127.0.0.1:40306/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1239423 T3980 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1239424 T3980 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 1239425 T3980 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 1239428 T3980 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:40306_solr
[junit4] 2> 1239428 T3980 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 1239430 T3980 oasc.Overseer.start Overseer (id=91844794211827715-127.0.0.1:40306_solr-n_0000000000) starting
[junit4] 2> 1239431 T3980 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 1239434 T3980 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 1239436 T3980 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 1239440 T3980 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 1239443 T3980 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 1239447 T3980 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 1239451 T4018 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 1239451 T3980 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 1239451 T4017 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 1239453 T3980 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 1239454 T3980 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1239456 T3980 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1239457 T4020 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@151206d name:ZooKeeperConnection Watcher:127.0.0.1:40306/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1239457 T3980 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1239458 T3980 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1239459 T3980 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1
[junit4] 2> 1239462 T4016 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1239463 T4017 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1239463 T4017 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 1239463 T4017 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
[junit4] 2> 1239464 T4017 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 1239465 T4016 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1239466 T4020 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1239466 T4014 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1239962 T3980 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 1239970 T3980 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4] 2> 1239974 T4016 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1239976 T4016 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1239976 T4017 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1239980 T4017 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core2",
[junit4] 2> "core_node_name":"node2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 1239981 T4017 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 1239981 T4017 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
[junit4] 2> 1239983 T4016 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1240086 T4020 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1240086 T4014 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1240476 T3980 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4] 2> 1240482 T3980 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
[junit4] 2> 1240486 T4016 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1240487 T4016 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1240487 T4016 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1240488 T4017 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1240488 T4016 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1240490 T4017 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core3",
[junit4] 2> "core_node_name":"node3",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 1240490 T4017 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 1240491 T4017 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1240493 T4016 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1240595 T4020 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1240595 T4014 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1240988 T3980 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1240994 T3980 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 1240997 T4016 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1240997 T4016 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1240997 T4016 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1240998 T4016 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1240999 T4017 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1241001 T4017 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core4",
[junit4] 2> "core_node_name":"node4",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 1241001 T4017 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 1241001 T4017 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 1241003 T4016 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1241105 T4014 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1241105 T4020 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1241498 T3980 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 1241505 T4016 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1241505 T4016 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1241505 T4016 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1241506 T4017 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1241506 T4017 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core5",
[junit4] 2> "core_node_name":"node5",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 1241507 T4017 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 1241507 T4017 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
[junit4] 2> 1241508 T4016 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1241611 T4014 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1241611 T4020 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1242005 T3980 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4] 2> 1242011 T4016 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1242011 T4016 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1242011 T4016 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1242012 T4017 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1242012 T4017 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core6",
[junit4] 2> "core_node_name":"node6",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 1242012 T4017 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 1242013 T4017 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1242014 T4016 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1242117 T4014 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1242117 T4020 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1242511 T3980 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1242519 T3980 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:40306 40306
[junit4] 2> 1243454 T4018 oasc.OverseerCollectionProcessor.run WARN Overseer cannot talk to ZK
[junit4] 2> 1244018 T4020 oasc.LeaderElector$ElectionWatcher.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:303)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
[junit4] 2>
[junit4] 2> 1252831 T4004 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:40306 40306
[junit4] 2> 1252831 T4020 oasc.LeaderElector$ElectionWatcher.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard2/election
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:303)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
[junit4] 2>
[junit4] 2> 1252832 T4020 oasc.LeaderElector$ElectionWatcher.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard3/election
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:303)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
[junit4] 2>
[junit4] 2> 1252832 T3980 oas.SolrTestCaseJ4.tearDown ###Ending testShardAssignment
[junit4] 2> 1252833 T4017 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:40306_solr
[junit4] 2> 1252833 T4021 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:306)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:289)
[junit4] 2>
[junit4] 2> 1252836 T3980 oas.SolrTestCaseJ4.setUp ###Starting testPlaceholders
[junit4] 2> 1252837 T3980 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 1252837 T4022 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 1252937 T3980 oasc.ZkTestServer.run start zk server on port:48804
[junit4] 2> 1252938 T3980 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1252950 T4028 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a77b31 name:ZooKeeperConnection Watcher:127.0.0.1:48804/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1252950 T3980 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1252951 T3980 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1252953 T4030 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@13ca4e8 name:ZooKeeperConnection Watcher:127.0.0.1:48804 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1252953 T3980 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1252955 T3980 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1252956 T4032 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@8cd6a4 name:ZooKeeperConnection Watcher:127.0.0.1:48804 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1252956 T3980 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1252956 T3980 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 1252959 T3980 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 1252960 T3980 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 1252962 T3980 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 1252963 T3980 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1252965 T3980 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1252965 T4034 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1636e45 name:ZooKeeperConnection Watcher:127.0.0.1:48804/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1252966 T3980 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1252966 T3980 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1252968 T3980 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
[junit4] 2> 1252970 T3980 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1252971 T4036 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@17a2a91 name:ZooKeeperConnection Watcher:127.0.0.1:48804/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1252971 T3980 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1252971 T3980 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 1252972 T3980 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 1252975 T3980 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:48804_solr
[junit4] 2> 1252975 T3980 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 1252976 T3980 oasc.Overseer.start Overseer (id=91844795098398724-127.0.0.1:48804_solr-n_0000000000) starting
[junit4] 2> 1252977 T3980 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 1252982 T3980 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 1252984 T3980 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 1252986 T3980 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 1252988 T3980 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 1252990 T3980 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 1252993 T4038 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 1252994 T4037 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 1252996 T4037 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1252996 T4037 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=12 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"12",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 1252996 T4037 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3, shard4, shard5, shard6, shard7, shard8, shard9, shard10, shard11, shard12]
[junit4] 2> 1252996 T4037 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 1252998 T4036 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1252999 T4034 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1252999 T4028 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1253495 T3980 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 1253503 T3980 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4] 2> 1253506 T4036 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1253506 T3980 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1253507 T4037 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1253509 T3980 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:48804 48804
[junit4] 2> 1254995 T4038 oasc.OverseerCollectionProcessor.run WARN Overseer cannot talk to ZK
[junit4] 2> 1255007 T4037 oasc.Overseer$ClusterStateUpdater.run WARN Solr cannot talk to ZK, exiting Overseer main queue loop org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /clusterstate.json
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.common.cloud.ClusterState.load(ClusterState.java:239)
[junit4] 2> at org.apache.solr.common.cloud.ZkStateReader.updateClusterState(ZkStateReader.java:445)
[junit4] 2> at org.apache.solr.common.cloud.ZkStateReader.updateClusterState(ZkStateReader.java:242)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:234)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2>
[junit4] 2> 1259247 T4022 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:48804 48804
[junit4] 2> 1259247 T4037 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:48804_solr
[junit4] 2> 1259248 T4039 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:306)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:289)
[junit4] 2>
[junit4] 2> 1259248 T3980 oas.SolrTestCaseJ4.tearDown ###Ending testPlaceholders
[junit4] 2> 1259252 T3980 oas.SolrTestCaseJ4.setUp ###Starting testShardLeaderChange
[junit4] 2> 1259253 T3980 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 1259253 T4040 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 1259353 T3980 oasc.ZkTestServer.run start zk server on port:38771
[junit4] 2> 1259354 T3980 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1259356 T4046 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1fec2c name:ZooKeeperConnection Watcher:127.0.0.1:38771/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1259357 T3980 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1259357 T3980 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1259358 T4048 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@11c185 name:ZooKeeperConnection Watcher:127.0.0.1:38771 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1259358 T3980 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1259360 T3980 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1259361 T4050 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1de51fa name:ZooKeeperConnection Watcher:127.0.0.1:38771 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1259361 T3980 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1259362 T3980 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 1259364 T3980 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 1259366 T4051 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1259367 T3980 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1259367 T4054 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@11468cc name:ZooKeeperConnection Watcher:127.0.0.1:38771/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1259368 T4051 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1259368 T4055 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ec641a name:ZooKeeperConnection Watcher:127.0.0.1:38771/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1259368 T3980 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1259368 T4051 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 1259368 T3980 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 1259370 T4051 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 1259370 T3980 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 1259371 T3980 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1259372 T4051 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:38771_solr
[junit4] 2> 1259372 T4051 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 1259373 T3980 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
[junit4] 2> 1259373 T4051 oasc.Overseer.start Overseer (id=91844795518877699-127.0.0.1:38771_solr-n_0000000000) starting
[junit4] 2> 1259375 T4051 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 1259375 T3980 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 1259376 T4051 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 1259378 T4051 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 1259381 T4051 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 1259384 T4051 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 1259387 T4051 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 1259391 T4057 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 1259391 T4051 oasc.OverseerTest$OverseerRestarter.run Killing overseer.
[junit4] 2> 1259391 T4056 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 1259392 T4051 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1259393 T4059 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7fda3a name:ZooKeeperConnection Watcher:127.0.0.1:38771/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1259393 T4051 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1259393 T4056 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=91844795518877699-127.0.0.1:38771_solr-n_0000000000) am no longer a leader.
[junit4] 2> 1259393 T4057 oasc.OverseerCollectionProcessor.run ERROR Unable to prioritize overseer org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /roles.json
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:226)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:223)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:223)
[junit4] 2> at org.apache.solr.cloud.OverseerCollectionProcessor.prioritizeOverseerNodes(OverseerCollectionProcessor.java:408)
[junit4] 2> at org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:256)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2>
[junit4] 2> 1259393 T4056 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:38771_solr
[junit4] 2> 1259395 T4060 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:306)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:289)
[junit4] 2>
[junit4] 2> 1259395 T4051 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:38771_solr
[junit4] 2> 1259395 T4051 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 1259397 T4051 oasc.Overseer.start Overseer (id=91844795518877701-127.0.0.1:38771_solr-n_0000000001) starting
[junit4] 2> 1259403 T4062 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 1259403 T4061 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 1259404 T4061 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1259405 T4061 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"state1",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 1259405 T4061 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1]
[junit4] 2> 1259405 T4061 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1259407 T4059 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1259407 T4055 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1259877 T3980 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1259884 T3980 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 1259887 T4059 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1259890 T4059 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1259890 T3980 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1259890 T4061 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1259895 T4061 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"state2",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 1259895 T4061 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
[junit4] 2> 1259897 T3980 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1259897 T4059 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1259898 T4064 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@146a7d1 name:ZooKeeperConnection Watcher:127.0.0.1:38771/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1259898 T3980 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1259899 T3980 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1259901 T3980 oascc.SolrZkClient.makePath makePath: /live_nodes/node2
[junit4] 2> 1259905 T4059 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1259905 T3980 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1259906 T4061 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"state1",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 1259906 T4061 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
[junit4] 2> 1259908 T4059 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1259911 T3980 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1260009 T4064 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1260009 T4055 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1260112 T3980 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1260114 T4059 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1260114 T4059 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1260114 T4059 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1260115 T4061 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1260116 T4061 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"state2",
[junit4] 2> "node_name":"node2",
[junit4] 2> "core":"core4",
[junit4] 2> "core_node_name":"node2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node2/solr/"}
[junit4] 2> 1260116 T4061 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4] 2> 1260116 T4061 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1260118 T4059 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1260220 T4055 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1260220 T4064 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1260614 T3980 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1260620 T3980 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1260620 T4064 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 1260624 T4059 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1260624 T4059 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1260624 T4059 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1260625 T4061 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1260626 T4059 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1260728 T4064 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1260821 T3980 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1260823 T3980 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1260824 T4066 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f6eb1e name:ZooKeeperConnection Watcher:127.0.0.1:38771/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1260824 T3980 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1260825 T3980 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1260827 T3980 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
[junit4] 2> 1260830 T4059 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1260830 T3980 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1260830 T4059 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1260831 T4059 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1260831 T4061 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1260831 T4061 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"state1",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 1260831 T4061 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
[junit4] 2> 1260833 T4059 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1260835 T4066 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 1260836 T4059 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1260836 T3980 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1260837 T4061 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"state2",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 1260837 T4061 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
[junit4] 2> 1260840 T4059 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1260844 T4059 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1260844 T3980 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1260845 T4068 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@135af94 name:ZooKeeperConnection Watcher:127.0.0.1:38771/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1260845 T3980 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1260846 T3980 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1260847 T3980 oascc.SolrZkClient.makePath makePath: /live_nodes/node2
[junit4] 2> 1260851 T4059 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1260851 T3980 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1260852 T4061 oasc.Overseer$ClusterStateU
[...truncated too long message...]
t is connected to ZooKeeper
[junit4] 2> 1316056 T3980 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 1316058 T3980 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 1316060 T3980 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:53690_solr
[junit4] 2> 1316060 T3980 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 1316061 T3980 oasc.Overseer.start Overseer (id=91844799233589251-127.0.0.1:53690_solr-n_0000000000) starting
[junit4] 2> 1316062 T3980 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 1316064 T3980 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 1316066 T3980 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 1316067 T3980 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 1316069 T3980 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 1316074 T4501 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 1316076 T4500 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1316076 T4500 oasc.Overseer$ClusterStateUpdater.run Replaying operations from work queue.
[junit4] 2> 1316076 T4500 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1/solr",
[junit4] 2> "node_name":"node1",
[junit4] 2> "shard":"s1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "core":"core1",
[junit4] 2> "roles":"",
[junit4] 2> "state":"recovering"}
[junit4] 2> 1316077 T4493 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 1316079 T4500 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1/solr",
[junit4] 2> "node_name":"node1",
[junit4] 2> "shard":"s1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "core":"core2",
[junit4] 2> "roles":"",
[junit4] 2> "state":"recovering"}
[junit4] 2> 1316079 T4493 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 1316080 T4500 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 1316081 T4500 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1316081 T4500 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1/solr",
[junit4] 2> "node_name":"node1",
[junit4] 2> "shard":"s1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "core":"core3",
[junit4] 2> "roles":"",
[junit4] 2> "state":"recovering"}
[junit4] 2> 1316083 T4499 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1316184 T4493 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 1316278 T3980 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:53690 53690
[junit4] 2> 1318076 T4501 oasc.OverseerCollectionProcessor.run WARN Overseer cannot talk to ZK
[junit4] 2> 1321985 T4487 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:53690 53690
[junit4] 2> 1321986 T3980 oas.SolrTestCaseJ4.tearDown ###Ending testReplay
[junit4] 2> 1321986 T4500 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:53690_solr
[junit4] 2> 1321987 T4502 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:306)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:289)
[junit4] 2>
[junit4] 2> 1321987 T3980 oas.SolrTestCaseJ4.initCore ####initCore
[junit4] 2> 1321988 T3980 oas.SolrTestCaseJ4.initCore ####initCore end
[junit4] 2> 1324988 T3980 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4] 2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.OverseerTest-A1FD9A535D55EE7C-001
[junit4] 2> NOTE: test params are: codec=Lucene46: {}, docValues:{}, sim=DefaultSimilarity, locale=es_HN, timezone=America/North_Dakota/Center
[junit4] 2> NOTE: Linux 3.8.0-41-generic i386/Oracle Corporation 1.7.0_60-ea (32-bit)/cpus=8,threads=1,free=27683752,total=518979584
[junit4] 2> NOTE: All tests run in this JVM: [SuggesterWFSTTest, TestManagedSchemaFieldResource, TestLazyCores, TestIndexingPerformance, TestSolr4Spatial, PingRequestHandlerTest, DeleteReplicaTest, HdfsCollectionsAPIDistributedZkTest, PluginInfoTest, TestSolrCoreProperties, TestCollationField, LegacyHTMLStripCharFilterTest, DirectUpdateHandlerTest, TestOverriddenPrefixQueryForCustomFieldType, TestStandardQParsers, PathHierarchyTokenizerFactoryTest, HttpPartitionTest, FullSolrCloudDistribCmdsTest, TestStressReorder, TestCloudInspectUtil, RequiredFieldsTest, SpellCheckComponentTest, QueryFacetTest, TestShardHandlerFactory, SuggestComponentTest, TestSolrXMLSerializer, TestLMDirichletSimilarityFactory, DirectSolrSpellCheckerTest, TestCSVLoader, DebugComponentTest, TestConfigSets, CurrencyFieldOpenExchangeTest, CoreAdminRequestStatusTest, SystemInfoHandlerTest, TestHashPartitioner, TestCSVResponseWriter, CoreAdminCreateDiscoverTest, TestRandomFaceting, TestModifyConfFiles, TestSerializedLuceneMatchVersion, RemoteQueryErrorTest, TestFiltering, TriLevelCompositeIdRoutingTest, TestCoreDiscovery, TestOmitPositions, HdfsBasicDistributedZkTest, TestQueryTypes, HdfsDirectoryTest, SignatureUpdateProcessorFactoryTest, PrimUtilsTest, HardAutoCommitTest, TestBinaryField, SolrPluginUtilsTest, ResponseHeaderTest, AddSchemaFieldsUpdateProcessorFactoryTest, BasicZkTest, ChaosMonkeySafeLeaderTest, HdfsLockFactoryTest, TestSolrDeletionPolicy2, TestFieldTypeResource, DateMathParserTest, OverseerStatusTest, BadIndexSchemaTest, FieldFacetExtrasTest, TestSchemaResource, DocValuesMissingTest, DisMaxRequestHandlerTest, TestSolrQueryParserResource, TestCopyFieldCollectionResource, SolrRequestParserTest, NoCacheHeaderTest, TestAnalyzedSuggestions, TestIBSimilarityFactory, TestMultiCoreConfBootstrap, TestFoldingMultitermQuery, TestPartialUpdateDeduplication, XmlUpdateRequestHandlerTest, TestSolrJ, TestRecoveryHdfs, TestManagedStopFilterFactory, TestRTGBase, FileUtilsTest, MinimalSchemaTest, NumericFieldsTest, OpenCloseCoreStressTest, HdfsUnloadDistributedZkTest, NoFacetTest, TestSchemaSimilarityResource, DistributedDebugComponentTest, XsltUpdateRequestHandlerTest, DOMUtilTest, TestPhraseSuggestions, TestFastWriter, StatsComponentTest, TestAnalyzeInfixSuggestions, TestArbitraryIndexDir, NotRequiredUniqueKeyTest, QueryResultKeyTest, BasicDistributedZkTest, BasicDistributedZk2Test, OverseerTest]
[junit4] Completed on J1 in 126.86s, 9 tests, 1 error <<< FAILURES!
[...truncated 612 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1296: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:920: There were test failures: 406 suites, 1688 tests, 1 error, 36 ignored (15 assumptions)
Total time: 69 minutes 29 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_60-ea-b15 -server -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.7.0_60-ea-b15) -
Build # 10428 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/10428/
Java: 64bit/jdk1.7.0_60-ea-b15 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC
3 tests failed.
REGRESSION: org.apache.solr.cloud.BasicDistributedZkTest.testDistribSearch
Error Message:
commitWithin did not work expected:<68> but was:<67>
Stack Trace:
java.lang.AssertionError: commitWithin did not work expected:<68> but was:<67>
at __randomizedtesting.SeedInfo.seed([BD05FD5B9D125215:3CE37343EA4D3229]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.apache.solr.cloud.BasicDistributedZkTest.doTest(BasicDistributedZkTest.java:347)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:863)
at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
at java.lang.Thread.run(Thread.java:745)
REGRESSION: org.apache.solr.cloud.HttpPartitionTest.testDistribSearch
Error Message:
collection already exists: c8n_1x3
Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: collection already exists: c8n_1x3
at __randomizedtesting.SeedInfo.seed([BD05FD5B9D125215:3CE37343EA4D3229]:0)
at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:554)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1500)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1460)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1514)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1456)
at org.apache.solr.cloud.HttpPartitionTest.testRf3(HttpPartitionTest.java:223)
at org.apache.solr.cloud.HttpPartitionTest.doTest(HttpPartitionTest.java:145)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:863)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
at java.lang.Thread.run(Thread.java:745)
REGRESSION: org.apache.solr.cloud.ZkSolrClientTest.testReconnect
Error Message:
KeeperErrorCode = NodeExists for /collections/collection3
Stack Trace:
org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /collections/collection3
at __randomizedtesting.SeedInfo.seed([BD05FD5B9D125215:64267429DA8B9F6A]:0)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
at org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:432)
at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:429)
at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:386)
at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:373)
at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:329)
at org.apache.solr.cloud.ZkSolrClientTest.testReconnect(ZkSolrClientTest.java:136)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
at java.lang.Thread.run(Thread.java:745)
Build Log:
[...truncated 10718 lines...]
[junit4] Suite: org.apache.solr.cloud.HttpPartitionTest
[junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/init-core-data-001
[junit4] 2> 280206 T304 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /zp_tvz/hq
[junit4] 2> 280216 T304 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> 280217 T304 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 280218 T305 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 280317 T304 oasc.ZkTestServer.run start zk server on port:54428
[junit4] 2> 280319 T304 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 288897 T308 oazsp.FileTxnLog.commit WARN fsync-ing the write ahead log in SyncThread:0 took 8577ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
[junit4] 2> 288900 T311 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@158ee1f1 name:ZooKeeperConnection Watcher:127.0.0.1:54428 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 288900 T304 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 288901 T304 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 288941 T304 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 288943 T313 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4e919bec name:ZooKeeperConnection Watcher:127.0.0.1:54428/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 288944 T304 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 288945 T304 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 288950 T304 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 288954 T304 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 288957 T304 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 288960 T304 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 288961 T304 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 289047 T304 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4] 2> 289048 T304 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 289052 T304 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 289052 T304 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 289055 T304 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 289055 T304 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 289058 T304 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 289059 T304 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 289061 T304 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 289061 T304 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 289063 T304 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 289064 T304 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
[junit4] 2> 289066 T304 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 289067 T304 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 289069 T304 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 289069 T304 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 289072 T304 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 289072 T304 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 289074 T304 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 289075 T304 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 289347 T304 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 289361 T304 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:40134
[junit4] 2> 289362 T304 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 289362 T304 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 289363 T304 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-002
[junit4] 2> 289363 T304 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-002/'
[junit4] 2> 289383 T304 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-002/solr.xml
[junit4] 2> 289420 T304 oasc.CoreContainer.<init> New CoreContainer 1148617275
[junit4] 2> 289420 T304 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-002/]
[junit4] 2> 289422 T304 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 289422 T304 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 289423 T304 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 289423 T304 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 289423 T304 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 289424 T304 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 289424 T304 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 289425 T304 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 289425 T304 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 289426 T304 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 289426 T304 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 289427 T304 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 289427 T304 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54428/solr
[junit4] 2> 289427 T304 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 289428 T304 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 289452 T326 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4060129a name:ZooKeeperConnection Watcher:127.0.0.1:54428 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 289452 T304 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 289493 T304 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 289542 T328 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@31495ec1 name:ZooKeeperConnection Watcher:127.0.0.1:54428/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 289543 T304 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 289545 T304 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 289559 T304 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 289610 T304 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 289638 T304 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 289643 T304 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 289649 T304 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 289651 T304 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:48768_zp_tvz%2Fhq
[junit4] 2> 289653 T304 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:48768_zp_tvz%2Fhq
[junit4] 2> 289656 T304 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 289659 T304 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 289666 T304 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:48768_zp_tvz%2Fhq
[junit4] 2> 289667 T304 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 289670 T304 oasc.Overseer.start Overseer (id=91846406863126531-127.0.0.1:48768_zp_tvz%2Fhq-n_0000000000) starting
[junit4] 2> 289675 T304 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 289690 T330 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 289690 T304 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 289695 T304 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 289698 T304 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 289703 T329 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 289709 T331 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
[junit4] 2> 289710 T331 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 289712 T328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 289712 T331 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 289715 T329 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 289716 T329 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:48768/zp_tvz/hq",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:48768_zp_tvz%2Fhq",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":null}
[junit4] 2> 289716 T329 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
[junit4] 2> 289717 T329 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 289720 T328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 289720 T328 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 290713 T331 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 290713 T331 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 290715 T331 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 290715 T331 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 290717 T331 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 290717 T331 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-002/collection1/'
[junit4] 2> 290718 T331 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-002/collection1/lib/.svn/' to classloader
[junit4] 2> 290719 T331 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-002/collection1/lib/classes/' to classloader
[junit4] 2> 290720 T331 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-002/collection1/lib/README' to classloader
[junit4] 2> 290772 T331 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
[junit4] 2> 290806 T331 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 290808 T331 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 290815 T331 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 291025 T331 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 291032 T331 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 291034 T331 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 291049 T331 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 291052 T331 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 291054 T331 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 291056 T331 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 291057 T331 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 291057 T331 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 291058 T331 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 291058 T331 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 291058 T331 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
[junit4] 2> 291059 T331 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 291059 T331 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-002/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-001/control/data/
[junit4] 2> 291059 T331 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6e4bfd69
[junit4] 2> 291060 T331 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-001/control/data
[junit4] 2> 291060 T331 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-001/control/data/index/
[junit4] 2> 291060 T331 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
[junit4] 2> 291063 T331 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-001/control/data/index
[junit4] 2> 291063 T331 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=32, maxMergeAtOnceExplicit=41, maxMergedSegmentMB=31.873046875, floorSegmentMB=0.3115234375, forceMergeDeletesPctAllowed=5.064936651571199, segmentsPerTier=43.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3054563728191859
[junit4] 2> 291064 T331 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RateLimitedDirectoryWrapper(RAMDirectory@7f114a62 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7976c9c4)),segFN=segments_1,generation=1}
[junit4] 2> 291064 T331 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 291068 T331 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 291068 T331 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 291068 T331 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 291068 T331 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 291069 T331 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 291069 T331 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 291069 T331 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 291070 T331 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 291070 T331 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 291070 T331 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 291071 T331 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 291071 T331 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 291071 T331 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 291072 T331 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 291072 T331 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 291072 T331 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 291073 T331 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 291075 T331 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 291078 T331 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 291078 T331 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 291079 T331 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=21, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7367731281060768]
[junit4] 2> 291079 T331 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RateLimitedDirectoryWrapper(RAMDirectory@7f114a62 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7976c9c4)),segFN=segments_1,generation=1}
[junit4] 2> 291080 T331 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 291080 T331 oass.SolrIndexSearcher.<init> Opening Searcher@465652da[collection1] main
[junit4] 2> 291080 T331 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 291082 T331 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 291082 T331 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 291082 T331 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 291083 T331 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 291083 T331 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 291084 T331 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 291084 T331 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 291084 T331 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
[junit4] 2> 291085 T331 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 291087 T332 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@465652da[collection1] main{UninvertingDirectoryReader()}
[junit4] 2> 291089 T331 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 291090 T335 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:48768/zp_tvz/hq collection:control_collection shard:shard1
[junit4] 2> 291090 T304 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4] 2> 291090 T304 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 291093 T304 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 291094 T335 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 291096 T338 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7845eeb name:ZooKeeperConnection Watcher:127.0.0.1:54428/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 291097 T304 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 291098 T304 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 291101 T304 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 291101 T335 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 291103 T328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 291103 T335 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 291104 T335 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C405 name=collection1 org.apache.solr.core.SolrCore@6cc70897 url=http://127.0.0.1:48768/zp_tvz/hq/collection1 node=127.0.0.1:48768_zp_tvz%2Fhq C405_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:48768/zp_tvz/hq, core=collection1, node_name=127.0.0.1:48768_zp_tvz%2Fhq}
[junit4] 2> 291104 T335 C405 P48768 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:48768/zp_tvz/hq/collection1/
[junit4] 2> 291104 T335 C405 P48768 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 291104 T329 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 291104 T335 C405 P48768 oasc.SyncStrategy.syncToMe http://127.0.0.1:48768/zp_tvz/hq/collection1/ has no replicas
[junit4] 2> 291105 T335 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:48768/zp_tvz/hq/collection1/ shard1
[junit4] 2> 291105 T335 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 291107 T328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 291115 T328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 291117 T328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 291219 T328 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 291220 T338 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 291266 T335 oasc.ZkController.register We are http://127.0.0.1:48768/zp_tvz/hq/collection1/ and leader is http://127.0.0.1:48768/zp_tvz/hq/collection1/
[junit4] 2> 291266 T335 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:48768/zp_tvz/hq
[junit4] 2> 291266 T335 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 291266 T335 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
[junit4] 2> 291267 T335 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 291269 T335 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 291269 T328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 291270 T328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 291270 T328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 291271 T329 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 291271 T329 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:48768/zp_tvz/hq",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:48768_zp_tvz%2Fhq",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"2",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 291273 T328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 291375 T328 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 291375 T338 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 291381 T304 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
[junit4] 2> 291382 T304 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 291384 T304 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:60479
[junit4] 2> 291385 T304 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 291386 T304 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 291386 T304 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-003
[junit4] 2> 291387 T304 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-003/'
[junit4] 2> 291401 T304 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-003/solr.xml
[junit4] 2> 291453 T304 oasc.CoreContainer.<init> New CoreContainer 338505595
[junit4] 2> 291454 T304 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-003/]
[junit4] 2> 291455 T304 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 291455 T304 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 291455 T304 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 291455 T304 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 291456 T304 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 291456 T304 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 291456 T304 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 291457 T304 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 291457 T304 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 291457 T304 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 291458 T304 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 291458 T304 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 291458 T304 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54428/solr
[junit4] 2> 291459 T304 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 291459 T304 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 291461 T349 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@22883e96 name:ZooKeeperConnection Watcher:127.0.0.1:54428 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 291462 T304 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 291469 T304 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 291470 T351 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@76541506 name:ZooKeeperConnection Watcher:127.0.0.1:54428/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 291471 T304 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 291477 T304 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 292479 T304 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38340_zp_tvz%2Fhq
[junit4] 2> 292481 T304 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:38340_zp_tvz%2Fhq
[junit4] 2> 292651 T352 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 292651 T352 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 292653 T328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 292654 T328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 292653 T352 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 292654 T328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 292655 T329 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 292656 T329 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:38340/zp_tvz/hq",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:38340_zp_tvz%2Fhq",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"2",
[junit4] 2> "core_node_name":null}
[junit4] 2> 292656 T329 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
[junit4] 2> 292656 T329 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 292658 T328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 292761 T351 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 292761 T338 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 292761 T328 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 293654 T352 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 293654 T352 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 293655 T352 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 293655 T352 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 293656 T352 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 293657 T352 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-003/collection1/'
[junit4] 2> 293657 T352 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-003/collection1/lib/.svn/' to classloader
[junit4] 2> 293657 T352 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-003/collection1/lib/classes/' to classloader
[junit4] 2> 293658 T352 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-003/collection1/lib/README' to classloader
[junit4] 2> 293693 T352 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
[junit4] 2> 293747 T352 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 293748 T352 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 293759 T352 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 294002 T352 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 294014 T352 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 294017 T352 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 294035 T352 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 294040 T352 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 294044 T352 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 294045 T352 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 294045 T352 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 294046 T352 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 294047 T352 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 294047 T352 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 294048 T352 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 294048 T352 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 294048 T352 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-003/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-001/jetty1/
[junit4] 2> 294049 T352 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6e4bfd69
[junit4] 2> 294049 T352 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-001/jetty1
[junit4] 2> 294050 T352 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-001/jetty1/index/
[junit4] 2> 294050 T352 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
[junit4] 2> 294051 T352 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-001/jetty1/index
[junit4] 2> 294051 T352 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=32, maxMergeAtOnceExplicit=41, maxMergedSegmentMB=31.873046875, floorSegmentMB=0.3115234375, forceMergeDeletesPctAllowed=5.064936651571199, segmentsPerTier=43.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3054563728191859
[junit4] 2> 294052 T352 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RateLimitedDirectoryWrapper(RAMDirectory@1a9009bb lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7361089b)),segFN=segments_1,generation=1}
[junit4] 2> 294052 T352 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 294056 T352 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 294057 T352 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 294057 T352 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 294057 T352 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 294057 T352 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 294058 T352 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 294058 T352 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 294058 T352 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 294058 T352 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 294059 T352 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 294059 T352 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 294060 T352 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 294060 T352 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 294060 T352 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 294061 T352 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 294061 T352 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 294062 T352 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 294064 T352 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 294066 T352 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 294066 T352 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 294066 T352 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=21, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7367731281060768]
[junit4] 2> 294067 T352 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RateLimitedDirectoryWrapper(RAMDirectory@1a9009bb lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7361089b)),segFN=segments_1,generation=1}
[junit4] 2> 294067 T352 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 294068 T352 oass.SolrIndexSearcher.<init> Opening Searcher@65e92309[collection1] main
[junit4] 2> 294068 T352 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 294069 T352 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 294069 T352 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 294070 T352 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 294070 T352 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 294070 T352 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 294071 T352 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 294071 T352 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 294071 T352 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
[junit4] 2> 294071 T352 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 294074 T353 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@65e92309[collection1] main{UninvertingDirectoryReader()}
[junit4] 2> 294076 T352 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 294077 T356 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:38340/zp_tvz/hq collection:collection1 shard:shard2
[junit4] 2> 294078 T304 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4] 2> 294078 T304 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 294078 T356 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 294087 T356 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
[junit4] 2> 294089 T328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 294089 T328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 294089 T356 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 294089 T356 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 294089 T328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> ASYNC NEW_CORE C406 name=collection1 org.apache.solr.core.SolrCore@77515c0f url=http://127.0.0.1:38340/zp_tvz/hq/collection1 node=127.0.0.1:38340_zp_tvz%2Fhq C406_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:38340/zp_tvz/hq, core=collection1, node_name=127.0.0.1:38340_zp_tvz%2Fhq}
[junit4] 2> 294090 T356 C406 P38340 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:38340/zp_tvz/hq/collection1/
[junit4] 2> 294090 T356 C406 P38340 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 294091 T356 C406 P38340 oasc.SyncStrategy.syncToMe http://127.0.0.1:38340/zp_tvz/hq/collection1/ has no replicas
[junit4] 2> 294091 T356 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:38340/zp_tvz/hq/collection1/ shard2
[junit4] 2> 294091 T356 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4] 2> 294092 T329 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 294095 T328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 294097 T328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 294101 T328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 294203 T328 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 294203 T338 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 294203 T351 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 294248 T356 oasc.ZkController.register We are http://127.0.0.1:38340/zp_tvz/hq/collection1/ and leader is http://127.0.0.1:38340/zp_tvz/hq/collection1/
[junit4] 2> 294248 T356 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:38340/zp_tvz/hq
[junit4] 2> 294249 T356 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 294249 T356 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 294249 T356 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 294251 T328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 294252 T328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 294252 T356 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 294252 T328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 294255 T329 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 294256 T329 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:38340/zp_tvz/hq",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:38340_zp_tvz%2Fhq",
[junit4] 2> "shard":"shard2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"2",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 294258 T328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 294344 T304 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
[junit4] 2> 294344 T304 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 294346 T304 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:34468
[junit4] 2> 294348 T304 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 294348 T304 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 294348 T304 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-004
[junit4] 2> 294349 T304 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-004/'
[junit4] 2> 294360 T328 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 294360 T351 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 294360 T338 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 294363 T304 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-004/solr.xml
[junit4] 2> 294406 T304 oasc.CoreContainer.<init> New CoreContainer 434989175
[junit4] 2> 294407 T304 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-004/]
[junit4] 2> 294408 T304 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 294409 T304 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 294409 T304 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 294409 T304 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 294410 T304 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 294410 T304 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 294411 T304 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 294411 T304 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 294412 T304 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 294413 T304 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 294413 T304 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 294414 T304 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 294414 T304 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54428/solr
[junit4] 2> 294415 T304 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 294416 T304 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 294417 T368 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@316569ed name:ZooKeeperConnection Watcher:127.0.0.1:54428 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 294418 T304 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 294420 T304 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 294422 T370 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2628245c name:ZooKeeperConnection Watcher:127.0.0.1:54428/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 294422 T304 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 294431 T304 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 295433 T304 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39069_zp_tvz%2Fhq
[junit4] 2> 295435 T304 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:39069_zp_tvz%2Fhq
[junit4] 2> 295628 T371 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 295628 T371 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 295630 T328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 295630 T371 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 295630 T328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 295630 T328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 295631 T329 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 295634 T329 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:39069/zp_tvz/hq",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:39069_zp_tvz%2Fhq",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"2",
[junit4] 2> "core_node_name":null}
[junit4] 2> 295635 T329 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4] 2> 295635 T329 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 295637 T328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 295739 T338 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 295739 T370 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 295739 T351 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 295739 T328 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 296630 T371 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 296630 T371 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 296631 T371 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 296631 T371 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 296632 T371 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 296633 T371 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-004/collection1/'
[junit4] 2> 296633 T371 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-004/collection1/lib/.svn/' to classloader
[junit4] 2> 296633 T371 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-004/collection1/lib/classes/' to classloader
[junit4] 2> 296634 T371 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-004/collection1/lib/README' to classloader
[junit4] 2> 296665 T371 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
[junit4] 2> 296704 T371 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 296705 T371 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 296713 T371 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 296898 T371 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 296907 T371 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 296909 T371 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 296924 T371 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 296927 T371 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 296930 T371 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 296931 T371 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 296932 T371 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 296932 T371 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 296933 T371 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 296933 T371 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 296933 T371 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 296934 T371 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 296934 T371 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-004/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-001/jetty2/
[junit4] 2> 296934 T371 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6e4bfd69
[junit4] 2> 296935 T371 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-001/jetty2
[junit4] 2> 296935 T371 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-001/jetty2/index/
[junit4] 2> 296935 T371 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
[junit4] 2> 296936 T371 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-001/jetty2/index
[junit4] 2> 296936 T371 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=32, maxMergeAtOnceExplicit=41, maxMergedSegmentMB=31.873046875, floorSegmentMB=0.3115234375, forceMergeDeletesPctAllowed=5.064936651571199, segmentsPerTier=43.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3054563728191859
[junit4] 2> 296937 T371 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RateLimitedDirectoryWrapper(RAMDirectory@55feabfc lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@328b7fd3)),segFN=segments_1,generation=1}
[junit4] 2> 296938 T371 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 296943 T371 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 296943 T371 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 296943 T371 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 296944 T371 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 296944 T371 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 296944 T371 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 296944 T371 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 296945 T371 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 296945 T371 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 296945 T371 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 296946 T371 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 296946 T371 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 296946 T371 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 296947 T371 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 296947 T371 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 296948 T371 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 296948 T371 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 296950 T371 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 296952 T371 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 296952 T371 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 296953 T371 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=21, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7367731281060768]
[junit4] 2> 296954 T371 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RateLimitedDirectoryWrapper(RAMDirectory@55feabfc lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@328b7fd3)),segFN=segments_1,generation=1}
[junit4] 2> 296954 T371 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 296954 T371 oass.SolrIndexSearcher.<init> Opening Searcher@5fa403ef[collection1] main
[junit4] 2> 296954 T371 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 296955 T371 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 296956 T371 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 296956 T371 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 296956 T371 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 296957 T371 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 296957 T371 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 296957 T371 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 296958 T371 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
[junit4] 2> 296958 T371 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 296961 T372 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5fa403ef[collection1] main{UninvertingDirectoryReader()}
[junit4] 2> 296963 T371 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 296964 T375 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:39069/zp_tvz/hq collection:collection1 shard:shard1
[junit4] 2> 296964 T304 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4] 2> 296965 T304 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 296965 T375 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 296966 T304 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 296967 T304 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 30000 for each attempt
[junit4] 2> 296967 T304 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):30000
[junit4] 2> 296968 T304 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 296970 T375 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 296972 T328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 296972 T328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 296972 T375 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 296972 T328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 2
[...truncated too long message...]
OServerCnxn.java:1081)
[junit4] 2> at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:404)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:200)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:131)
[junit4] 2>
[junit4] 2> 1100083 T5458 oazs.NIOServerCnxn.sendBuffer ERROR Unexpected Exception: java.nio.channels.CancelledKeyException
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1081)
[junit4] 2> at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:170)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:169)
[junit4] 2>
[junit4] 2> 1100085 T5400 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:54600 54600
[junit4] 2> 1100827 T5455 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:54600 54600
[junit4] 2> 1100829 T5400 oas.SolrTestCaseJ4.tearDown ###Ending testReconnect
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=ZkSolrClientTest -Dtests.method=testReconnect -Dtests.seed=BD05FD5B9D125215 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=zh_HK -Dtests.timezone=Asia/Urumqi -Dtests.file.encoding=UTF-8
[junit4] ERROR 16.3s J0 | ZkSolrClientTest.testReconnect <<<
[junit4] > Throwable #1: org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /collections/collection3
[junit4] > at __randomizedtesting.SeedInfo.seed([BD05FD5B9D125215:64267429DA8B9F6A]:0)
[junit4] > at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
[junit4] > at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] > at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
[junit4] > at org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:432)
[junit4] > at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] > at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:429)
[junit4] > at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:386)
[junit4] > at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:373)
[junit4] > at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:329)
[junit4] > at org.apache.solr.cloud.ZkSolrClientTest.testReconnect(ZkSolrClientTest.java:136)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 1102835 T5400 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4] 2> 1102835 T5400 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=424248011
[junit4] 2> 1102835 T5400 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@4cb07900
[junit4] 2> 1102852 T5400 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0,transaction_logs_total_size=0,transaction_logs_total_number=0}
[junit4] 2> 1102852 T5400 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4] 2> 1102853 T5400 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4] 2> 1102853 T5400 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4] 2> 1102853 T5400 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4] 2> 1102853 T5400 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4] 2> 1102854 T5400 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ZkSolrClientTest-BD05FD5B9D125215-001/init-core-data-001/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ZkSolrClientTest-BD05FD5B9D125215-001/init-core-data-001/index;done=false>>]
[junit4] 2> 1102854 T5400 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ZkSolrClientTest-BD05FD5B9D125215-001/init-core-data-001/index
[junit4] 2> 1102854 T5400 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ZkSolrClientTest-BD05FD5B9D125215-001/init-core-data-001 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ZkSolrClientTest-BD05FD5B9D125215-001/init-core-data-001;done=false>>]
[junit4] 2> 1102854 T5400 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ZkSolrClientTest-BD05FD5B9D125215-001/init-core-data-001
[junit4] 2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ZkSolrClientTest-BD05FD5B9D125215-001
[junit4] 2> NOTE: test params are: codec=Lucene42, sim=RandomSimilarityProvider(queryNorm=true,coord=yes): {}, locale=zh_HK, timezone=Asia/Urumqi
[junit4] 2> NOTE: Linux 3.8.0-41-generic amd64/Oracle Corporation 1.7.0_60-ea (64-bit)/cpus=8,threads=1,free=178295544,total=341049344
[junit4] 2> NOTE: All tests run in this JVM: [TermsComponentTest, EchoParamsTest, SampleTest, TestFileDictionaryLookup, TestWriterPerf, TestCursorMarkWithoutUniqueKey, SOLR749Test, TestReplicationHandler, TestQueryUtils, StressHdfsTest, FieldMutatingUpdateProcessorTest, UpdateRequestProcessorFactoryTest, ScriptEngineTest, TestFieldSortValues, UnloadDistributedZkTest, SoftAutoCommitTest, TestHighlightDedupGrouping, AliasIntegrationTest, TestCharFilters, TestDefaultSimilarityFactory, TestCloudManagedSchemaAddField, DistributedExpandComponentTest, RecoveryZkTest, TestSort, TestSolrXmlPersistor, DirectUpdateHandlerOptimizeTest, TestSimpleQParserPlugin, CursorPagingTest, TestBlendedInfixSuggestions, DistributedQueryComponentCustomSortTest, TestJmxMonitoredMap, BasicFunctionalityTest, SuggesterFSTTest, TestConfig, EnumFieldTest, QueryEqualityTest, OverseerRolesTest, TestNonDefinedSimilarityFactory, DistributedQueryElevationComponentTest, TestRemoteStreaming, UpdateParamsTest, TestSolrIndexConfig, TestNonNRTOpen, MultiThreadedOCPTest, TestLuceneMatchVersion, ReturnFieldsTest, TestStressVersions, TestExceedMaxTermLength, TestDistributedMissingSort, TestSearchPerf, RAMDirectoryFactoryTest, SyncSliceTest, InfoHandlerTest, SolrCoreTest, FastVectorHighlighterTest, DistributedSuggestComponentTest, TestElisionMultitermQuery, ShardRoutingTest, TestCollationFieldDocValues, SearchHandlerTest, TestReversedWildcardFilterFactory, TestDocSet, TestLFUCache, BJQParserTest, ReplicationFactorTest, HdfsChaosMonkeySafeLeaderTest, TestJoin, TestRandomMergePolicy, TestUtils, OverseerCollectionProcessorTest, TestInitQParser, TestGroupingSearch, QueryElevationComponentTest, TestLMJelinekMercerSimilarityFactory, DistribDocExpirationUpdateProcessorTest, TestFastLRUCache, TestSolrDeletionPolicy1, HdfsRecoveryZkTest, ClusterStateUpdateTest, TermVectorComponentTest, AbstractAnalyticsFacetTest, TestFieldResource, CSVRequestHandlerTest, TestUniqueKeyFieldResource, StatelessScriptUpdateProcessorFactoryTest, TimeZoneUtilsTest, TestOrdValues, TestShortCircuitedRequests, RegexBoostProcessorTest, TestPseudoReturnFields, AutoCommitTest, DocValuesTest, OverseerTest, TestSystemIdResolver, DistributedTermsComponentTest, SuggesterTest, SpatialFilterTest, UUIDFieldTest, BasicDistributedZkTest, OpenCloseCoreStressTest, LeaderElectionIntegrationTest, FullSolrCloudDistribCmdsTest, LeaderElectionTest, ZkSolrClientTest]
[junit4] Completed on J0 in 23.31s, 5 tests, 1 error <<< FAILURES!
[...truncated 633 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1296: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:920: There were test failures: 406 suites, 1688 tests, 2 errors, 1 failure, 36 ignored (15 assumptions)
Total time: 61 minutes 38 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_60-ea-b15 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_55) - Build #
10427 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/10427/
Java: 32bit/jdk1.7.0_55 -server -XX:+UseParallelGC
2 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.ReplicationFactorTest
Error Message:
Suite timeout exceeded (>= 7200000 msec).
Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
at __randomizedtesting.SeedInfo.seed([511A59F9A569CC43]:0)
REGRESSION: org.apache.solr.cloud.ReplicationFactorTest.testDistribSearch
Error Message:
Test abandoned because suite timeout was reached.
Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
at __randomizedtesting.SeedInfo.seed([511A59F9A569CC43]:0)
Build Log:
[...truncated 11798 lines...]
[junit4] Suite: org.apache.solr.cloud.ReplicationFactorTest
[junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/init-core-data-001
[junit4] 2> 756406 T3082 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /w/s
[junit4] 2> 756411 T3082 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> 756411 T3082 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 756412 T3083 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 756512 T3082 oasc.ZkTestServer.run start zk server on port:57711
[junit4] 2> 756513 T3082 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 756583 T3089 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a6031e name:ZooKeeperConnection Watcher:127.0.0.1:57711 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 756583 T3082 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 756584 T3082 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 756590 T3082 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 756591 T3091 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@b48f7b name:ZooKeeperConnection Watcher:127.0.0.1:57711/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 756592 T3082 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 756592 T3082 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 756596 T3082 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 756597 T3082 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 756599 T3082 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 756606 T3082 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 756607 T3082 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 756613 T3082 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4] 2> 756613 T3082 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 756615 T3082 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 756616 T3082 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 756618 T3082 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 756618 T3082 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 756622 T3082 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 756622 T3082 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 756624 T3082 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 756625 T3082 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 756628 T3082 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 756628 T3082 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
[junit4] 2> 756631 T3082 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 756631 T3082 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 756633 T3082 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 756633 T3082 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 756635 T3082 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 756636 T3082 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 756637 T3082 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 756637 T3082 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 756878 T3082 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 756880 T3082 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:37150
[junit4] 2> 756880 T3082 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 756881 T3082 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 756881 T3082 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-002
[junit4] 2> 756881 T3082 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-002/'
[junit4] 2> 756895 T3082 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-002/solr.xml
[junit4] 2> 756931 T3082 oasc.CoreContainer.<init> New CoreContainer 2867872
[junit4] 2> 756931 T3082 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-002/]
[junit4] 2> 756932 T3082 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 756933 T3082 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 756933 T3082 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 756933 T3082 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 756934 T3082 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 756934 T3082 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 756934 T3082 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 756934 T3082 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 756935 T3082 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 756935 T3082 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 756936 T3082 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 756936 T3082 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 756936 T3082 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:57711/solr
[junit4] 2> 756937 T3082 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 756937 T3082 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 756940 T3102 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f6b228 name:ZooKeeperConnection Watcher:127.0.0.1:57711 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 756940 T3082 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 756943 T3082 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 756944 T3104 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@dba1b7 name:ZooKeeperConnection Watcher:127.0.0.1:57711/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 756945 T3082 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 756946 T3082 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 756950 T3082 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 756953 T3082 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 756956 T3082 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 756959 T3082 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 756963 T3082 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 756965 T3082 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57811_w%2Fs
[junit4] 2> 756967 T3082 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:57811_w%2Fs
[junit4] 2> 756971 T3082 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 756978 T3082 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 756984 T3082 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:57811_w%2Fs
[junit4] 2> 756984 T3082 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 756987 T3082 oasc.Overseer.start Overseer (id=91845329425006595-127.0.0.1:57811_w%2Fs-n_0000000000) starting
[junit4] 2> 756994 T3082 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 757000 T3106 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 757000 T3082 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 757002 T3082 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 757004 T3082 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 757005 T3105 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 757008 T3107 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
[junit4] 2> 757009 T3107 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 757010 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 757010 T3107 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 757012 T3105 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 757013 T3105 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:57811/w/s",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:57811_w%2Fs",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":null}
[junit4] 2> 757014 T3105 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
[junit4] 2> 757014 T3105 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 757017 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 757017 T3104 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 758011 T3107 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 758012 T3107 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 758013 T3107 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 758013 T3107 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 758014 T3107 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 758015 T3107 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-002/collection1/'
[junit4] 2> 758016 T3107 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-002/collection1/lib/.svn/' to classloader
[junit4] 2> 758016 T3107 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-002/collection1/lib/classes/' to classloader
[junit4] 2> 758017 T3107 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-002/collection1/lib/README' to classloader
[junit4] 2> 758058 T3107 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
[junit4] 2> 758091 T3107 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 758093 T3107 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 758099 T3107 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 758378 T3107 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 758383 T3107 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 758385 T3107 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 758399 T3107 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 758402 T3107 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 758404 T3107 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 758405 T3107 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 758405 T3107 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 758405 T3107 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 758406 T3107 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 758406 T3107 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 758407 T3107 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
[junit4] 2> 758407 T3107 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 758408 T3107 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-002/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-001/control/data/
[junit4] 2> 758408 T3107 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@139ecdf
[junit4] 2> 758409 T3107 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-001/control/data
[junit4] 2> 758409 T3107 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-001/control/data/index/
[junit4] 2> 758410 T3107 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
[junit4] 2> 758410 T3107 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-001/control/data/index
[junit4] 2> 758411 T3107 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=38, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=0.5703125, noCFSRatio=1.0]
[junit4] 2> 758412 T3107 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@10642b8 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@10a2ca6),segFN=segments_1,generation=1}
[junit4] 2> 758412 T3107 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 758415 T3107 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 758416 T3107 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 758416 T3107 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 758416 T3107 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 758416 T3107 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 758417 T3107 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 758417 T3107 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 758417 T3107 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 758418 T3107 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 758418 T3107 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 758418 T3107 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 758419 T3107 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 758419 T3107 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 758419 T3107 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 758419 T3107 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 758420 T3107 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 758420 T3107 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 758422 T3107 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 758425 T3107 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 758425 T3107 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 758425 T3107 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=27, maxMergeAtOnceExplicit=33, maxMergedSegmentMB=15.2119140625, floorSegmentMB=0.314453125, forceMergeDeletesPctAllowed=29.847777813798476, segmentsPerTier=47.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 758426 T3107 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@10642b8 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@10a2ca6),segFN=segments_1,generation=1}
[junit4] 2> 758426 T3107 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 758426 T3107 oass.SolrIndexSearcher.<init> Opening Searcher@701950[collection1] main
[junit4] 2> 758427 T3107 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 758427 T3107 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 758428 T3107 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 758428 T3107 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 758428 T3107 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 758428 T3107 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 758429 T3107 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 758429 T3107 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 758429 T3107 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
[junit4] 2> 758429 T3107 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 758431 T3108 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@701950[collection1] main{UninvertingDirectoryReader()}
[junit4] 2> 758434 T3107 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 758435 T3111 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:57811/w/s collection:control_collection shard:shard1
[junit4] 2> 758435 T3082 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4] 2> 758435 T3082 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 758435 T3111 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 758438 T3082 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 758439 T3114 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@382715 name:ZooKeeperConnection Watcher:127.0.0.1:57711/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 758440 T3082 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 758441 T3082 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 758441 T3111 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 758442 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 758442 T3111 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 758442 T3111 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 758442 T3082 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 758443 T3105 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> ASYNC NEW_CORE C513 name=collection1 org.apache.solr.core.SolrCore@193da52 url=http://127.0.0.1:57811/w/s/collection1 node=127.0.0.1:57811_w%2Fs C513_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:57811/w/s, core=collection1, node_name=127.0.0.1:57811_w%2Fs}
[junit4] 2> 758443 T3111 C513 P57811 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:57811/w/s/collection1/
[junit4] 2> 758443 T3111 C513 P57811 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 758444 T3111 C513 P57811 oasc.SyncStrategy.syncToMe http://127.0.0.1:57811/w/s/collection1/ has no replicas
[junit4] 2> 758444 T3111 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:57811/w/s/collection1/ shard1
[junit4] 2> 758444 T3111 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 758444 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 758453 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 758455 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 758557 T3104 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 758557 T3114 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 758604 T3111 oasc.ZkController.register We are http://127.0.0.1:57811/w/s/collection1/ and leader is http://127.0.0.1:57811/w/s/collection1/
[junit4] 2> 758604 T3111 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:57811/w/s
[junit4] 2> 758605 T3111 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 758605 T3111 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
[junit4] 2> 758605 T3111 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 758606 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 758606 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 758606 T3111 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 758606 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 758607 T3105 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 758608 T3105 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:57811/w/s",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:57811_w%2Fs",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 758610 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 758675 T3082 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
[junit4] 2> 758676 T3082 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 758679 T3082 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:52139
[junit4] 2> 758679 T3082 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 758680 T3082 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 758680 T3082 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-003
[junit4] 2> 758680 T3082 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-003/'
[junit4] 2> 758691 T3082 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-003/solr.xml
[junit4] 2> 758711 T3104 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 758711 T3114 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 758735 T3082 oasc.CoreContainer.<init> New CoreContainer 7724478
[junit4] 2> 758735 T3082 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-003/]
[junit4] 2> 758736 T3082 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 758736 T3082 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 758737 T3082 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 758737 T3082 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 758737 T3082 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 758738 T3082 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 758738 T3082 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 758738 T3082 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 758738 T3082 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 758739 T3082 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 758739 T3082 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 758740 T3082 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 758740 T3082 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:57711/solr
[junit4] 2> 758740 T3082 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 758741 T3082 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 758743 T3125 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@310d30 name:ZooKeeperConnection Watcher:127.0.0.1:57711 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 758743 T3082 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 758746 T3082 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 758748 T3127 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1352c71 name:ZooKeeperConnection Watcher:127.0.0.1:57711/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 758748 T3082 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 758756 T3082 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 759758 T3082 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51545_w%2Fs
[junit4] 2> 759759 T3082 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:51545_w%2Fs
[junit4] 2> 759768 T3128 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 759768 T3128 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 759769 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 759769 T3128 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 759770 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 759770 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 759771 T3105 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 759772 T3105 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:51545/w/s",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:51545_w%2Fs",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":null}
[junit4] 2> 759772 T3105 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
[junit4] 2> 759772 T3105 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
[junit4] 2> 759774 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 759876 T3114 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 759876 T3127 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 759876 T3104 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 760770 T3128 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 760770 T3128 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 760771 T3128 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 760771 T3128 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 760772 T3128 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 760772 T3128 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-003/collection1/'
[junit4] 2> 760773 T3128 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-003/collection1/lib/.svn/' to classloader
[junit4] 2> 760773 T3128 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-003/collection1/lib/classes/' to classloader
[junit4] 2> 760773 T3128 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-003/collection1/lib/README' to classloader
[junit4] 2> 760805 T3128 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
[junit4] 2> 760839 T3128 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 760840 T3128 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 760845 T3128 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 761114 T3128 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 761119 T3128 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 761121 T3128 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 761131 T3128 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 761133 T3128 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 761136 T3128 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 761136 T3128 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 761137 T3128 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 761137 T3128 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 761138 T3128 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 761138 T3128 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 761138 T3128 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 761138 T3128 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 761138 T3128 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-003/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-001/jetty1/
[junit4] 2> 761139 T3128 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@139ecdf
[junit4] 2> 761139 T3128 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-001/jetty1
[junit4] 2> 761140 T3128 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-001/jetty1/index/
[junit4] 2> 761140 T3128 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
[junit4] 2> 761140 T3128 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-001/jetty1/index
[junit4] 2> 761140 T3128 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=38, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=0.5703125, noCFSRatio=1.0]
[junit4] 2> 761141 T3128 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@118564e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7843f0),segFN=segments_1,generation=1}
[junit4] 2> 761141 T3128 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 761145 T3128 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 761146 T3128 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 761146 T3128 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 761146 T3128 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 761146 T3128 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 761147 T3128 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 761147 T3128 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 761147 T3128 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 761147 T3128 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 761148 T3128 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 761148 T3128 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 761149 T3128 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 761149 T3128 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 761149 T3128 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 761150 T3128 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 761150 T3128 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 761151 T3128 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 761153 T3128 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 761155 T3128 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 761155 T3128 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 761156 T3128 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=27, maxMergeAtOnceExplicit=33, maxMergedSegmentMB=15.2119140625, floorSegmentMB=0.314453125, forceMergeDeletesPctAllowed=29.847777813798476, segmentsPerTier=47.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 761156 T3128 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@118564e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7843f0),segFN=segments_1,generation=1}
[junit4] 2> 761157 T3128 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 761157 T3128 oass.SolrIndexSearcher.<init> Opening Searcher@1846c78[collection1] main
[junit4] 2> 761157 T3128 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 761158 T3128 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 761158 T3128 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 761158 T3128 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 761159 T3128 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 761159 T3128 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 761159 T3128 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 761159 T3128 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 761159 T3128 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
[junit4] 2> 761160 T3128 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 761162 T3129 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1846c78[collection1] main{UninvertingDirectoryReader()}
[junit4] 2> 761164 T3128 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 761165 T3132 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:51545/w/s collection:collection1 shard:shard3
[junit4] 2> 761165 T3082 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4] 2> 761165 T3132 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4] 2> 761165 T3082 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 761169 T3132 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard3
[junit4] 2> 761170 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 761171 T3132 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 761171 T3132 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 761171 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 761171 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> ASYNC NEW_CORE C514 name=collection1 org.apache.solr.core.SolrCore@f6c6db url=http://127.0.0.1:51545/w/s/collection1 node=127.0.0.1:51545_w%2Fs C514_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:51545/w/s, core=collection1, node_name=127.0.0.1:51545_w%2Fs}
[junit4] 2> 761171 T3132 C514 P51545 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:51545/w/s/collection1/
[junit4] 2> 761171 T3105 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 761171 T3132 C514 P51545 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 761172 T3132 C514 P51545 oasc.SyncStrategy.syncToMe http://127.0.0.1:51545/w/s/collection1/ has no replicas
[junit4] 2> 761172 T3132 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:51545/w/s/collection1/ shard3
[junit4] 2> 761172 T3132 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
[junit4] 2> 761173 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 761175 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 761179 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 761280 T3104 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 761280 T3127 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 761280 T3114 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 761326 T3132 oasc.ZkController.register We are http://127.0.0.1:51545/w/s/collection1/ and leader is http://127.0.0.1:51545/w/s/collection1/
[junit4] 2> 761326 T3132 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:51545/w/s
[junit4] 2> 761326 T3132 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 761326 T3132 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 761327 T3132 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 761328 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 761328 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 761328 T3132 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 761328 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 761329 T3105 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 761329 T3105 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:51545/w/s",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:51545_w%2Fs",
[junit4] 2> "shard":"shard3",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 761331 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 761388 T3082 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
[junit4] 2> 761389 T3082 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 761391 T3082 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:44300
[junit4] 2> 761391 T3082 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 761392 T3082 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 761392 T3082 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-004
[junit4] 2> 761392 T3082 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-004/'
[junit4] 2> 761403 T3082 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-004/solr.xml
[junit4] 2> 761432 T3114 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 761433 T3104 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 761433 T3127 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 761437 T3082 oasc.CoreContainer.<init> New CoreContainer 27163750
[junit4] 2> 761437 T3082 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-004/]
[junit4] 2> 761438 T3082 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 761438 T3082 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 761439 T3082 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 761439 T3082 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 761439 T3082 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 761439 T3082 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 761440 T3082 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 761440 T3082 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 761440 T3082 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 761441 T3082 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 761441 T3082 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 761441 T3082 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 761442 T3082 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:57711/solr
[junit4] 2> 761442 T3082 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 761443 T3082 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 761444 T3144 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1f9324c name:ZooKeeperConnection Watcher:127.0.0.1:57711 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 761445 T3082 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 761447 T3082 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 761448 T3146 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@25a255 name:ZooKeeperConnection Watcher:127.0.0.1:57711/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 761448 T3082 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 761456 T3082 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 762458 T3082 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40648_w%2Fs
[junit4] 2> 762467 T3082 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:40648_w%2Fs
[junit4] 2> 762482 T3147 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 762482 T3147 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 762484 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 762484 T3147 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 762484 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 762485 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 762487 T3105 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 762488 T3105 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:40648/w/s",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:40648_w%2Fs",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":null}
[junit4] 2> 762489 T3105 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 762489 T3105 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 762492 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 762595 T3114 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 762595 T3127 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 762595 T3146 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 762595 T3104 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 763484 T3147 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 763485 T3147 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 763485 T3147 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 763485 T3147 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 763487 T3147 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 763487 T3147 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-004/collection1/'
[junit4] 2> 763488 T3147 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-004/collection1/lib/.svn/' to classloader
[junit4] 2> 763488 T3147 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-004/collection1/lib/classes/' to classloader
[junit4] 2> 763488 T3147 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-004/collection1/lib/README' to classloader
[junit4] 2> 763523 T3147 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
[junit4] 2> 763574 T3147 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 763575 T3147 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 763581 T3147 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 763862 T3147 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 763867 T3147 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 763869 T3147 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 763881 T3147 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 763884 T3147 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 763887 T3147 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 763888 T3147 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 763888 T3147 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 763888 T3147 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 763889 T3147 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 763889 T3147 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 763890 T3147 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 763890 T3147 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 763890 T3147 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-004/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-001/jetty2/
[junit4] 2> 763890 T3147 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@139ecdf
[junit4] 2> 763891 T3147 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-001/jetty2
[junit4] 2> 763891 T3147 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-001/jetty2/index/
[junit4] 2> 763891 T3147 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
[junit4] 2> 763891 T3147 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-001/jetty2/index
[junit4] 2> 763892 T3147 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=38, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=0.5703125, noCFSRatio=1.0]
[junit4] 2> 763892 T3147 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@d70b6d lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@b71128),segFN=segments_1,generation=1}
[junit4] 2> 763892 T3147 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 763897 T3147 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 763898 T3147 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 763898 T3147 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 763898 T3147 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 763899 T3147 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 763899 T3147 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 763900 T3147 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 763900 T3147 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 763901 T3147 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 763901 T3147 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 763902 T3147 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 763902 T3147 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 763903 T3147 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 763903 T3147 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 763904 T3147 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 763904 T3147 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 763905 T3147 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 763909 T3147 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 763913 T3147 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 763913 T3147 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 763914 T3147 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=27, maxMergeAtOnceExplicit=33, maxMergedSegmentMB=15.2119140625, floorSegmentMB=0.314453125, forceMergeDeletesPctAllowed=29.847777813798476, segmentsPerTier=47.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 763914 T3147 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@d70b6d lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@b71128),segFN=segments_1,generation=1}
[junit4] 2> 763914 T3147 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 763914 T3147 oass.SolrIndexSearcher.<init> Opening Searcher@480d2b[collection1] main
[junit4] 2> 763915 T3147 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 763916 T3147 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 763916 T3147 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 763916 T3147 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 763917 T3147 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 763917 T3147 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 763917 T3147 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 763917 T3147 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 763918 T3147 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
[junit4] 2> 763918 T3147 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 763920 T3148 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@480d2b[collection1] main{UninvertingDirectoryReader()}
[junit4] 2> 763922 T3147 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 763923 T3151 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:40648/w/s collection:collection1 shard:shard1
[junit4] 2> 763924 T3082 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4] 2> 763924 T3151 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 763924 T3082 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 763928 T3151 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 763929 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 763929 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 763929 T3151 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 763930 T3151 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 763929 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 763930 T3105 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> ASYNC NEW_CORE C515 name=collection1 org.apache.solr.core.SolrCore@aba5c6 url=http://127.0.0.1:40648/w/s/collection1 node=127.0.0.1:40648_w%2Fs C515_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:40648/w/s, core=collection1, node_name=127.0.0.1:40648_w%2Fs}
[junit4] 2> 763930 T3151 C515 P40648 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:40648/w/s/collection1/
[junit4] 2> 763930 T3151 C515 P40648 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 763931 T3151 C515 P40648 oasc.SyncStrategy.syncToMe http://127.0.0.1:40648/w/s/collection1/ has no replicas
[junit4] 2> 763931 T3151 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:40648/w/s/collection1/ shard1
[junit4] 2> 763931 T3151 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 763932 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 763941 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 763944 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 764046 T3114 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 764046 T3104 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 764046 T3146 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 764046 T3127 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 764092 T3151 oasc.ZkController.register We are http://127.0.0.1:40648/w/s/collection1/ and leader is http://127.0.0.1:40648/w/s/collection1/
[junit4] 2> 764092 T3151 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:40648/w/s
[junit4] 2> 764092 T3151 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 764092 T3151 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 764092 T3151 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 764094 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 764094 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 764094 T3151 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 764094 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 764095 T3105 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 764095 T3105 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:40648/w/s",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:40648_w%2Fs",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node2"}
[junit4] 2> 764097 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 764152 T3082 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
[junit4] 2> 764153 T3082 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 764155 T3082 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:60612
[junit4] 2> 764156 T3082 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 764156 T3082 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 764156 T3082 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-005
[junit4] 2> 764157 T3082 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-005/'
[junit4] 2> 764168 T3082 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-005/solr.xml
[junit4] 2> 764199 T3114 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 764199 T3146 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 764199 T3127 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 764199 T3104 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 764202 T3082 oasc.CoreContainer.<init> New CoreContainer 5535998
[junit4] 2> 764202 T3082 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-005/]
[junit4] 2> 764203 T3082 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 764203 T3082 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 764204 T3082 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 764204 T3082 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 764204 T3082 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 764204 T3082 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 764205 T3082 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 764205 T3082 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 764205 T3082 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 764206 T3082 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 764206 T3082 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 764206 T3082 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 764207 T3082 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:57711/solr
[junit4] 2> 764207 T3082 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 764208 T3082 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 764209 T3163 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d39f27 name:ZooKeeperConnection Watcher:127.0.0.1:57711 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 764210 T3082 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 764212 T3082 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 764213 T3165 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@193ab2e name:ZooKeeperConnection Watcher:127.0.0.1:57711/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 764213 T3082 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 764220 T3082 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 765224 T3082 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:58297_w%2Fs
[junit4] 2> 765226 T3082 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:58297_w%2Fs
[junit4] 2> 765240 T3166 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 765241 T3166 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 765242 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 765243 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 765242 T3166 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 765243 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 765245 T3105 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 765246 T3105 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:58297/w/s",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:58297_w%2Fs",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":null}
[junit4] 2> 765246 T3105 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 765246 T3105 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 765249 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 765352 T3127 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 765352 T3114 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 765352 T3146 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 765352 T3104 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 765352 T3165 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 766243 T3166 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 766244 T3166 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 766244 T3166 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 766244 T3166 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 766245 T3166 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 766246 T3166 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-005/collection1/'
[junit4] 2> 766246 T3166 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-005/collection1/lib/.svn/' to classloader
[junit4] 2> 766246 T3166 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-005/collection1/lib/classes/' to classloader
[junit4] 2> 766247 T3166 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-005/collection1/lib/README' to classloader
[junit4] 2> 766278 T3166 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
[junit4] 2> 766313 T3166 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 766314 T3166 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 766319 T3166 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 766592 T3166 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, re
[...truncated too long message...]
Socket.<init>(Socket.java:208)
[junit4] 2> at org.apache.solr.cloud.ZkTestServer.send4LetterWord(ZkTestServer.java:314)
[junit4] 2> at org.apache.solr.cloud.ZkTestServer.waitForServerDown(ZkTestServer.java:275)
[junit4] 2> at org.apache.solr.cloud.ZkTestServer$ZKServerMain.shutdown(ZkTestServer.java:126)
[junit4] 2> at org.apache.solr.cloud.ZkTestServer.shutdown(ZkTestServer.java:262)
[junit4] 2> at org.apache.solr.cloud.AbstractDistribZkTestBase.tearDown(AbstractDistribZkTestBase.java:232)
[junit4] 2> at org.apache.solr.cloud.AbstractFullDistribZkTestBase.tearDown(AbstractFullDistribZkTestBase.java:1427)
[junit4] 2> at org.apache.solr.cloud.ReplicationFactorTest.tearDown(ReplicationFactorTest.java:86)
[junit4] 2> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit4] 2> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
[junit4] 2> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit4] 2> at java.lang.reflect.Method.invoke(Method.java:606)
[junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
[junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:885)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
[junit4] 2> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
[junit4] 2> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4] 2> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
[junit4] 2> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
[junit4] 2> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
[junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
[junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
[junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
[junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
[junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
[junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
[junit4] 2> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
[junit4] 2> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4] 2> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
[junit4] 2> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4] 2> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
[junit4] 2> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 2) Thread[id=3164, name=TEST-ReplicationFactorTest.testDistribSearch-seed#[511A59F9A569CC43]-SendThread(127.0.0.1:57711), state=TIMED_WAITING, group=TGRP-ReplicationFactorTest]
[junit4] 2> at java.lang.Thread.sleep(Native Method)
[junit4] 2> at org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:230)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1185)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1122)
[junit4] 2> 7994083 T3082 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4] 2> NOTE: test params are: codec=Lucene42, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=in, timezone=America/Chihuahua
[junit4] 2> NOTE: Linux 3.8.0-41-generic i386/Oracle Corporation 1.7.0_55 (32-bit)/cpus=8,threads=3,free=144297616,total=306970624
[junit4] 2> NOTE: All tests run in this JVM: [ExternalFileFieldSortTest, SliceStateTest, ScriptEngineTest, TestFunctionQuery, TestSchemaVersionResource, BadComponentTest, DeleteShardTest, TestRandomDVFaceting, BlockDirectoryTest, TestJmxMonitoredMap, TestLuceneMatchVersion, TestRemoteStreaming, UnloadDistributedZkTest, SuggesterFSTTest, DistributedExpandComponentTest, OverseerTest, MergeStrategyTest, TestUniqueKeyFieldResource, TestRequestStatusCollectionAPI, TestReplicationHandler, TestCursorMarkWithoutUniqueKey, DistributedQueryComponentCustomSortTest, TestCoreContainer, ZkControllerTest, TestFieldSortValues, InfoHandlerTest, TestSchemaNameResource, DistributedTermsComponentTest, TestBM25SimilarityFactory, HdfsBasicDistributedZk2Test, TestExceedMaxTermLength, ClusterStateTest, TestExtendedDismaxParser, TestFreeTextSuggestions, TestNonNRTOpen, TestOrdValues, PreAnalyzedUpdateProcessorTest, TestRealTimeGet, MBeansHandlerTest, HdfsChaosMonkeySafeLeaderTest, TestFieldResource, TestReversedWildcardFilterFactory, CircularListTest, TestPseudoReturnFields, SpatialFilterTest, WordBreakSolrSpellCheckerTest, JsonLoaderTest, TestSurroundQueryParser, TestLFUCache, FileBasedSpellCheckerTest, BasicDistributedZkTest, SyncSliceTest, SchemaVersionSpecificBehaviorTest, TestFaceting, TestPostingsSolrHighlighter, TestUtils, TestCharFilters, ResponseLogComponentTest, SpellPossibilityIteratorTest, UpdateParamsTest, TestSolrDeletionPolicy1, TestCollationFieldDocValues, ShowFileRequestHandlerTest, QueryParsingTest, TestFastLRUCache, TestLMJelinekMercerSimilarityFactory, SearchHandlerTest, ReturnFieldsTest, ReplicationFactorTest]
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=ReplicationFactorTest -Dtests.seed=511A59F9A569CC43 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=in -Dtests.timezone=America/Chihuahua -Dtests.file.encoding=ISO-8859-1
[junit4] ERROR 0.00s J0 | ReplicationFactorTest (suite) <<<
[junit4] > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
[junit4] > at __randomizedtesting.SeedInfo.seed([511A59F9A569CC43]:0)
[junit4] Completed on J0 in 7237.73s, 1 test, 2 errors <<< FAILURES!
[...truncated 97 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1296: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:920: There were test failures: 406 suites, 1688 tests, 1 suite-level error, 1 error, 148 ignored (15 assumptions)
Total time: 172 minutes 15 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_55 -server -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure