You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2013/03/13 06:30:34 UTC
[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.8.0-ea-b79) - Build #
4688 - Still Failing!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/4688/
Java: 32bit/jdk1.8.0-ea-b79 -client -XX:+UseSerialGC
2 tests failed.
REGRESSION: org.apache.solr.cloud.BasicDistributedZk2Test.testDistribSearch
Error Message:
Server at http://127.0.0.1:41037/onenodecollectioncore returned non ok status:404, message:Can not find: /onenodecollectioncore/update
Stack Trace:
org.apache.solr.common.SolrException: Server at http://127.0.0.1:41037/onenodecollectioncore returned non ok status:404, message:Can not find: /onenodecollectioncore/update
at __randomizedtesting.SeedInfo.seed([1763FA0A2E88829F:9685741259D7E2A3]:0)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:375)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:181)
at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:117)
at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:116)
at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:102)
at org.apache.solr.cloud.BasicDistributedZk2Test.testNodeWithoutCollectionForwarding(BasicDistributedZk2Test.java:196)
at org.apache.solr.cloud.BasicDistributedZk2Test.doTest(BasicDistributedZk2Test.java:89)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:806)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:487)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:722)
REGRESSION: org.apache.solr.cloud.UnloadDistributedZkTest.testDistribSearch
Error Message:
Still found shard
Stack Trace:
java.lang.AssertionError: Still found shard
at __randomizedtesting.SeedInfo.seed([1763FA0A2E88829F:9685741259D7E2A3]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.UnloadDistributedZkTest.testUnloadShardAndCollection(UnloadDistributedZkTest.java:124)
at org.apache.solr.cloud.UnloadDistributedZkTest.doTest(UnloadDistributedZkTest.java:77)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:806)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:487)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:722)
Build Log:
[...truncated 9249 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.BasicDistributedZk2Test
[junit4:junit4] 2> 0 T1451 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4:junit4] 2> 3 T1451 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1363152020061
[junit4:junit4] 2> 3 T1451 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 4 T1452 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 4 T1452 oazs.ZooKeeperServer.setTickTime tickTime set to 1000
[junit4:junit4] 2> 5 T1452 oazs.ZooKeeperServer.setMinSessionTimeout minSessionTimeout set to -1
[junit4:junit4] 2> 5 T1452 oazs.ZooKeeperServer.setMaxSessionTimeout maxSessionTimeout set to -1
[junit4:junit4] 2> 5 T1452 oazs.NIOServerCnxnFactory.configure binding to port 0.0.0.0/0.0.0.0:0
[junit4:junit4] 2> 6 T1452 oazsp.FileTxnSnapLog.save Snapshotting: 0x0 to /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-1363152020061/zookeeper/server1/data/version-2/snapshot.0
[junit4:junit4] 2> 104 T1451 oasc.ZkTestServer.run start zk server on port:49444
[junit4:junit4] 2> 105 T1451 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:49444 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@deae68
[junit4:junit4] 2> 107 T1457 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:49444. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 107 T1451 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 107 T1457 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:49444, initiating session
[junit4:junit4] 2> 107 T1453 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:37884
[junit4:junit4] 2> 108 T1453 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:37884
[junit4:junit4] 2> 109 T1455 oazsp.FileTxnLog.append Creating new log file: log.1
[junit4:junit4] 2> 112 T1455 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d623122600000 with negotiated timeout 10000 for client /127.0.0.1:37884
[junit4:junit4] 2> 112 T1457 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:49444, sessionid = 0x13d623122600000, negotiated timeout = 10000
[junit4:junit4] 2> 113 T1458 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@deae68 name:ZooKeeperConnection Watcher:127.0.0.1:49444 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 114 T1451 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 114 T1451 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 118 T1456 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d623122600000
[junit4:junit4] 2> 119 T1458 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 120 T1453 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:37884 which had sessionid 0x13d623122600000
[junit4:junit4] 2> 119 T1451 oaz.ZooKeeper.close Session: 0x13d623122600000 closed
[junit4:junit4] 2> 121 T1451 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:49444/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@1953c3
[junit4:junit4] 2> 123 T1459 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:49444. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 122 T1451 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 123 T1459 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:49444, initiating session
[junit4:junit4] 2> 123 T1453 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:37885
[junit4:junit4] 2> 124 T1453 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:37885
[junit4:junit4] 2> 125 T1455 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d623122600001 with negotiated timeout 10000 for client /127.0.0.1:37885
[junit4:junit4] 2> 125 T1459 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:49444, sessionid = 0x13d623122600001, negotiated timeout = 10000
[junit4:junit4] 2> 126 T1460 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1953c3 name:ZooKeeperConnection Watcher:127.0.0.1:49444/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 126 T1451 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 127 T1451 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 132 T1451 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 135 T1451 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 138 T1451 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 142 T1451 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 143 T1451 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 148 T1451 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 150 T1451 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 155 T1451 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 155 T1451 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 159 T1451 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 160 T1451 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 163 T1451 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 164 T1451 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 167 T1451 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 168 T1451 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 172 T1451 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 173 T1451 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 175 T1451 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 175 T1451 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 177 T1451 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 178 T1451 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 181 T1456 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d623122600001
[junit4:junit4] 2> 181 T1460 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 181 T1451 oaz.ZooKeeper.close Session: 0x13d623122600001 closed
[junit4:junit4] 2> 181 T1453 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:37885 which had sessionid 0x13d623122600001
[junit4:junit4] 2> 237 T1451 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4] 2> 249 T1451 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:55548
[junit4:junit4] 2> 250 T1451 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 250 T1451 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 251 T1451 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1363152020240
[junit4:junit4] 2> 251 T1451 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1363152020240/solr.xml
[junit4:junit4] 2> 252 T1451 oasc.CoreContainer.<init> New CoreContainer 21041180
[junit4:junit4] 2> 253 T1451 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1363152020240/'
[junit4:junit4] 2> 253 T1451 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1363152020240/'
[junit4:junit4] 2> 272 T1451 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 273 T1451 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 273 T1451 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 273 T1451 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 274 T1451 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 274 T1451 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 274 T1451 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 275 T1451 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 275 T1451 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 276 T1451 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 283 T1451 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4] 2> 296 T1451 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:49444/solr
[junit4:junit4] 2> 297 T1451 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 297 T1451 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:49444 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@277fe0
[junit4:junit4] 2> 298 T1451 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 298 T1470 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:49444. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 298 T1470 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:49444, initiating session
[junit4:junit4] 2> 299 T1453 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:37886
[junit4:junit4] 2> 299 T1453 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:37886
[junit4:junit4] 2> 300 T1455 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d623122600002 with negotiated timeout 20000 for client /127.0.0.1:37886
[junit4:junit4] 2> 300 T1470 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:49444, sessionid = 0x13d623122600002, negotiated timeout = 20000
[junit4:junit4] 2> 300 T1471 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@277fe0 name:ZooKeeperConnection Watcher:127.0.0.1:49444 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 300 T1451 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 302 T1456 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d623122600002
[junit4:junit4] 2> 302 T1471 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 302 T1453 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:37886 which had sessionid 0x13d623122600002
[junit4:junit4] 2> 302 T1451 oaz.ZooKeeper.close Session: 0x13d623122600002 closed
[junit4:junit4] 2> 303 T1451 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 305 T1451 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:49444/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@4147e4
[junit4:junit4] 2> 306 T1451 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 306 T1472 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:49444. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 307 T1472 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:49444, initiating session
[junit4:junit4] 2> 307 T1453 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:37887
[junit4:junit4] 2> 307 T1453 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:37887
[junit4:junit4] 2> 308 T1455 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d623122600003 with negotiated timeout 20000 for client /127.0.0.1:37887
[junit4:junit4] 2> 308 T1472 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:49444, sessionid = 0x13d623122600003, negotiated timeout = 20000
[junit4:junit4] 2> 309 T1473 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4147e4 name:ZooKeeperConnection Watcher:127.0.0.1:49444/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 309 T1451 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 310 T1456 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d623122600003 type:create cxid:0x2 zxid:0x1a txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 311 T1451 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 312 T1451 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55548_
[junit4:junit4] 2> 313 T1456 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d623122600003 type:delete cxid:0x8 zxid:0x1c txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:55548_ Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:55548_
[junit4:junit4] 2> 313 T1451 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:55548_
[junit4:junit4] 2> 316 T1451 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 319 T1456 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d623122600003 type:delete cxid:0x16 zxid:0x21 txntype:-1 reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4] 2> 319 T1451 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 322 T1451 oasc.Overseer.start Overseer (id=89335530786914307-127.0.0.1:55548_-n_0000000000) starting
[junit4:junit4] 2> 322 T1456 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d623122600003 type:create cxid:0x1b zxid:0x23 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 323 T1456 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d623122600003 type:create cxid:0x1c zxid:0x24 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 324 T1456 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d623122600003 type:create cxid:0x1d zxid:0x25 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 324 T1456 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d623122600003 type:create cxid:0x1e zxid:0x26 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 325 T1475 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 326 T1451 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 327 T1451 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 329 T1451 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 331 T1474 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 335 T1476 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1363152020240/collection1
[junit4:junit4] 2> 335 T1476 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 336 T1476 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 336 T1476 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 338 T1476 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1363152020240/collection1/'
[junit4:junit4] 2> 339 T1476 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1363152020240/collection1/lib/README' to classloader
[junit4:junit4] 2> 339 T1476 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1363152020240/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 383 T1476 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 427 T1476 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 428 T1476 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4] 2> 433 T1476 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4] 2> 841 T1476 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 847 T1476 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 850 T1476 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 859 T1476 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 862 T1476 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 865 T1476 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 866 T1476 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 866 T1476 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 867 T1476 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 867 T1476 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 868 T1476 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 868 T1476 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 868 T1476 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1363152020240/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363152020061/control/data/
[junit4:junit4] 2> 868 T1476 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6197d7
[junit4:junit4] 2> 869 T1476 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 869 T1476 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363152020061/control/data forceNew: false
[junit4:junit4] 2> 869 T1476 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1363152020061/control/data/index/
[junit4:junit4] 2> 870 T1476 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1363152020061/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 870 T1476 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363152020061/control/data/index forceNew: false
[junit4:junit4] 2> 871 T1476 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@13ca0cf lockFactory=org.apache.lucene.store.NativeFSLockFactory@1f30bdd),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> 871 T1476 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4] 2> 873 T1476 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 873 T1476 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 873 T1476 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 874 T1476 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 874 T1476 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 874 T1476 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 875 T1476 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 875 T1476 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 875 T1476 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 885 T1476 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 888 T1476 oass.SolrIndexSearcher.<init> Opening Searcher@1f4b7f3 main
[junit4:junit4] 2> 888 T1476 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363152020061/control/data/tlog
[junit4:junit4] 2> 889 T1476 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 889 T1476 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 892 T1477 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1f4b7f3 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 892 T1476 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 893 T1476 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 893 T1456 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d623122600003 type:create cxid:0x55 zxid:0x2a txntype:-1 reqpath:n/a Error Path:/solr/overseer/queue Error:KeeperErrorCode = NoNode for /solr/overseer/queue
[junit4:junit4] 2> 1834 T1474 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1834 T1474 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:55548_",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:55548",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "numShards":"1"}
[junit4:junit4] 2> 1834 T1474 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 1835 T1474 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4] 2> 1835 T1456 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d623122600003 type:create cxid:0x5d zxid:0x2d txntype:-1 reqpath:n/a Error Path:/solr/overseer/queue-work Error:KeeperErrorCode = NoNode for /solr/overseer/queue-work
[junit4:junit4] 2> 1838 T1473 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 1898 T1476 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 1898 T1476 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:55548 collection:control_collection shard:shard1
[junit4:junit4] 2> 1899 T1476 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 1908 T1456 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d623122600003 type:delete cxid:0x77 zxid:0x37 txntype:-1 reqpath:n/a Error Path:/solr/collections/control_collection/leaders Error:KeeperErrorCode = NoNode for /solr/collections/control_collection/leaders
[junit4:junit4] 2> 1909 T1476 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 1910 T1456 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d623122600003 type:create cxid:0x78 zxid:0x38 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 1912 T1476 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 1912 T1476 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 1913 T1476 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:55548/collection1/
[junit4:junit4] 2> 1913 T1476 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 1913 T1476 oasc.SyncStrategy.syncToMe http://127.0.0.1:55548/collection1/ has no replicas
[junit4:junit4] 2> 1914 T1476 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:55548/collection1/
[junit4:junit4] 2> 1914 T1476 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 1920 T1456 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d623122600003 type:create cxid:0x82 zxid:0x3c txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 3342 T1474 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 3353 T1473 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 3376 T1476 oasc.ZkController.register We are http://127.0.0.1:55548/collection1/ and leader is http://127.0.0.1:55548/collection1/
[junit4:junit4] 2> 3377 T1476 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:55548
[junit4:junit4] 2> 3377 T1476 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 3377 T1476 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 3378 T1476 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 3380 T1476 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 3381 T1451 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 3382 T1451 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 3383 T1451 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 3389 T1451 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 3391 T1451 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:49444/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@cc94a8
[junit4:junit4] 2> 3392 T1451 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 3392 T1479 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:49444. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 3393 T1479 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:49444, initiating session
[junit4:junit4] 2> 3393 T1453 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:37889
[junit4:junit4] 2> 3394 T1453 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:37889
[junit4:junit4] 2> 3395 T1455 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d623122600004 with negotiated timeout 10000 for client /127.0.0.1:37889
[junit4:junit4] 2> 3395 T1479 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:49444, sessionid = 0x13d623122600004, negotiated timeout = 10000
[junit4:junit4] 2> 3395 T1480 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@cc94a8 name:ZooKeeperConnection Watcher:127.0.0.1:49444/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 3396 T1451 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 3397 T1451 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 3399 T1451 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 3456 T1451 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4] 2> 3458 T1451 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:32962
[junit4:junit4] 2> 3459 T1451 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 3459 T1451 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 3459 T1451 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363152023458
[junit4:junit4] 2> 3460 T1451 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363152023458/solr.xml
[junit4:junit4] 2> 3460 T1451 oasc.CoreContainer.<init> New CoreContainer 33445359
[junit4:junit4] 2> 3460 T1451 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363152023458/'
[junit4:junit4] 2> 3461 T1451 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363152023458/'
[junit4:junit4] 2> 3478 T1451 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 3478 T1451 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 3479 T1451 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 3479 T1451 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 3479 T1451 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 3480 T1451 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 3480 T1451 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 3480 T1451 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 3481 T1451 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 3481 T1451 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 3487 T1451 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4] 2> 3498 T1451 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:49444/solr
[junit4:junit4] 2> 3499 T1451 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 3499 T1451 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:49444 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@743621
[junit4:junit4] 2> 3500 T1451 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 3500 T1490 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:49444. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 3501 T1490 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:49444, initiating session
[junit4:junit4] 2> 3501 T1453 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:37890
[junit4:junit4] 2> 3502 T1453 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:37890
[junit4:junit4] 2> 3503 T1455 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d623122600005 with negotiated timeout 20000 for client /127.0.0.1:37890
[junit4:junit4] 2> 3503 T1490 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:49444, sessionid = 0x13d623122600005, negotiated timeout = 20000
[junit4:junit4] 2> 3503 T1491 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@743621 name:ZooKeeperConnection Watcher:127.0.0.1:49444 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 3504 T1451 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 3505 T1456 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d623122600005
[junit4:junit4] 2> 3506 T1453 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:37890 which had sessionid 0x13d623122600005
[junit4:junit4] 2> 3506 T1491 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 3506 T1451 oaz.ZooKeeper.close Session: 0x13d623122600005 closed
[junit4:junit4] 2> 3507 T1451 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 3510 T1451 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:49444/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@34ae52
[junit4:junit4] 2> 3510 T1492 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:49444. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 3510 T1451 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 3511 T1492 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:49444, initiating session
[junit4:junit4] 2> 3511 T1453 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:37891
[junit4:junit4] 2> 3511 T1453 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:37891
[junit4:junit4] 2> 3512 T1455 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d623122600006 with negotiated timeout 20000 for client /127.0.0.1:37891
[junit4:junit4] 2> 3512 T1492 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:49444, sessionid = 0x13d623122600006, negotiated timeout = 20000
[junit4:junit4] 2> 3512 T1493 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@34ae52 name:ZooKeeperConnection Watcher:127.0.0.1:49444/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 3513 T1451 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 3514 T1456 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d623122600006 type:create cxid:0x1 zxid:0x4a txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 3514 T1456 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d623122600006 type:create cxid:0x2 zxid:0x4b txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 3516 T1451 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 4519 T1451 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:32962_
[junit4:junit4] 2> 4521 T1456 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d623122600006 type:delete cxid:0xb zxid:0x4c txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:32962_ Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:32962_
[junit4:junit4] 2> 4522 T1451 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:32962_
[junit4:junit4] 2> 4526 T1473 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 4527 T1480 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 4527 T1493 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 4529 T1473 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 4537 T1494 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363152023458/collection1
[junit4:junit4] 2> 4537 T1494 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 4538 T1494 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 4539 T1494 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 4541 T1494 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363152023458/collection1/'
[junit4:junit4] 2> 4542 T1494 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363152023458/collection1/lib/README' to classloader
[junit4:junit4] 2> 4542 T1494 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363152023458/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 4576 T1494 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 4638 T1494 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 4639 T1494 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4] 2> 4645 T1494 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4] 2> 4858 T1474 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 4859 T1474 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:55548_",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:55548",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "numShards":"1"}
[junit4:junit4] 2> 4862 T1480 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 4862 T1493 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 4862 T1473 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 5079 T1494 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 5087 T1494 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 5091 T1494 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 5106 T1494 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 5109 T1494 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 5112 T1494 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 5114 T1494 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 5114 T1494 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 5115 T1494 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 5116 T1494 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 5116 T1494 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 5117 T1494 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 5117 T1494 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363152023458/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363152020061/jetty1/
[junit4:junit4] 2> 5117 T1494 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6197d7
[junit4:junit4] 2> 5118 T1494 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 5119 T1494 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363152020061/jetty1 forceNew: false
[junit4:junit4] 2> 5119 T1494 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1363152020061/jetty1/index/
[junit4:junit4] 2> 5119 T1494 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1363152020061/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 5120 T1494 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363152020061/jetty1/index forceNew: false
[junit4:junit4] 2> 5122 T1494 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@171782c lockFactory=org.apache.lucene.store.NativeFSLockFactory@1f6686),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> 5122 T1494 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4] 2> 5125 T1494 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 5126 T1494 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 5126 T1494 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 5127 T1494 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 5128 T1494 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 5129 T1494 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 5129 T1494 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 5130 T1494 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 5131 T1494 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 5134 T1494 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 5136 T1494 oass.SolrIndexSearcher.<init> Opening Searcher@1ba6f6f main
[junit4:junit4] 2> 5137 T1494 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363152020061/jetty1/tlog
[junit4:junit4] 2> 5137 T1494 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 5137 T1494 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 5141 T1495 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1ba6f6f main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 5143 T1494 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 5143 T1494 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 6366 T1474 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 6367 T1474 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:32962_",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:32962",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "numShards":"2"}
[junit4:junit4] 2> 6368 T1474 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 2
[junit4:junit4] 2> 6368 T1474 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4:junit4] 2> 6374 T1493 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 6374 T1480 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 6374 T1473 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 7144 T1494 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 7145 T1494 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:32962 collection:collection1 shard:shard2
[junit4:junit4] 2> 7146 T1494 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4:junit4] 2> 7155 T1456 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d623122600006 type:delete cxid:0x48 zxid:0x5c txntype:-1 reqpath:n/a Error Path:/solr/collections/collection1/leaders Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders
[junit4:junit4] 2> 7156 T1494 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 7157 T1456 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d623122600006 type:create cxid:0x49 zxid:0x5d txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 7160 T1494 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 7160 T1494 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 7161 T1494 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:32962/collection1/
[junit4:junit4] 2> 7161 T1494 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 7162 T1494 oasc.SyncStrategy.syncToMe http://127.0.0.1:32962/collection1/ has no replicas
[junit4:junit4] 2> 7162 T1494 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:32962/collection1/
[junit4:junit4] 2> 7162 T1494 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4] 2> 7168 T1456 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d623122600006 type:create cxid:0x53 zxid:0x61 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 7880 T1474 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 7891 T1493 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 7891 T1473 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 7891 T1480 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 7922 T1494 oasc.ZkController.register We are http://127.0.0.1:32962/collection1/ and leader is http://127.0.0.1:32962/collection1/
[junit4:junit4] 2> 7922 T1494 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:32962
[junit4:junit4] 2> 7923 T1494 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 7923 T1494 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 7924 T1494 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 7926 T1494 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 7927 T1451 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 7928 T1451 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 7929 T1451 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 7991 T1451 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4] 2> 7993 T1451 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:41037
[junit4:junit4] 2> 7994 T1451 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 7994 T1451 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 7994 T1451 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1363152027992
[junit4:junit4] 2> 7995 T1451 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1363152027992/solr.xml
[junit4:junit4] 2> 7995 T1451 oasc.CoreContainer.<init> New CoreContainer 1980529
[junit4:junit4] 2> 7996 T1451 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1363152027992/'
[junit4:junit4] 2> 7996 T1451 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1363152027992/'
[junit4:junit4] 2> 8014 T1451 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 8015 T1451 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 8015 T1451 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 8015 T1451 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 8016 T1451 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 8016 T1451 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 8016 T1451 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 8017 T1451 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 8017 T1451 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 8018 T1451 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 8024 T1451 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4] 2> 8037 T1451 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:49444/solr
[junit4:junit4] 2> 8037 T1451 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 8037 T1451 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:49444 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@1e5b2b4
[junit4:junit4] 2> 8038 T1506 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:49444. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 8038 T1451 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 8039 T1506 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:49444, initiating session
[junit4:junit4] 2> 8039 T1453 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:37892
[junit4:junit4] 2> 8039 T1453 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:37892
[junit4:junit4] 2> 8040 T1455 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d623122600007 with negotiated timeout 20000 for client /127.0.0.1:37892
[junit4:junit4] 2> 8040 T1506 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:49444, sessionid = 0x13d623122600007, negotiated timeout = 20000
[junit4:junit4] 2> 8041 T1507 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1e5b2b4 name:ZooKeeperConnection Watcher:127.0.0.1:49444 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 8041 T1451 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 8042 T1456 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d623122600007
[junit4:junit4] 2> 8043 T1507 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 8043 T1453 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:37892 which had sessionid 0x13d623122600007
[junit4:junit4] 2> 8043 T1451 oaz.ZooKeeper.close Session: 0x13d623122600007 closed
[junit4:junit4] 2> 8044 T1451 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 8046 T1451 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:49444/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@738b04
[junit4:junit4] 2> 8047 T1508 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:49444. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 8047 T1451 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 8047 T1508 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:49444, initiating session
[junit4:junit4] 2> 8048 T1453 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:37893
[junit4:junit4] 2> 8048 T1453 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:37893
[junit4:junit4] 2> 8049 T1455 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d623122600008 with negotiated timeout 20000 for client /127.0.0.1:37893
[junit4:junit4] 2> 8049 T1508 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:49444, sessionid = 0x13d623122600008, negotiated timeout = 20000
[junit4:junit4] 2> 8049 T1509 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@738b04 name:ZooKeeperConnection Watcher:127.0.0.1:49444/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 8050 T1451 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 8051 T1456 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d623122600008 type:create cxid:0x1 zxid:0x6e txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 8052 T1456 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d623122600008 type:create cxid:0x2 zxid:0x6f txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 8054 T1451 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 9057 T1451 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41037_
[junit4:junit4] 2> 9059 T1456 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d623122600008 type:delete cxid:0xb zxid:0x70 txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:41037_ Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:41037_
[junit4:junit4] 2> 9060 T1451 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:41037_
[junit4:junit4] 2> 9063 T1473 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 9063 T1480 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 9063 T1493 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 9065 T1509 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 9066 T1473 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 9066 T1480 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 9067 T1493 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 9074 T1510 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1363152027992/collection1
[junit4:junit4] 2> 9074 T1510 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 9075 T1510 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 9076 T1510 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 9078 T1510 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1363152027992/collection1/'
[junit4:junit4] 2> 9078 T1510 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1363152027992/collection1/lib/README' to classloader
[junit4:junit4] 2> 9079 T1510 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1363152027992/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 9114 T1510 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 9176 T1510 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 9177 T1510 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4] 2> 9181 T1510 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4] 2> 9397 T1474 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 9398 T1474 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:32962_",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:32962",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "numShards":"2"}
[junit4:junit4] 2> 9400 T1509 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 9400 T1473 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 9400 T1480 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 9400 T1493 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 9622 T1510 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 9629 T1510 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 9632 T1510 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 9641 T1510 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 9644 T1510 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 9647 T1510 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 9648 T1510 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 9648 T1510 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 9648 T1510 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 9649 T1510 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 9649 T1510 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 9650 T1510 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 9650 T1510 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1363152027992/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363152020061/jetty2/
[junit4:junit4] 2> 9650 T1510 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6197d7
[junit4:junit4] 2> 9651 T1510 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 9651 T1510 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363152020061/jetty2 forceNew: false
[junit4:junit4] 2> 9651 T1510 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1363152020061/jetty2/index/
[junit4:junit4] 2> 9651 T1510 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1363152020061/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 9652 T1510 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363152020061/jetty2/index forceNew: false
[junit4:junit4] 2> 9653 T1510 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1b190f2 lockFactory=org.apache.lucene.store.NativeFSLockFactory@81c3be),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> 9653 T1510 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4] 2> 9655 T1510 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 9655 T1510 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 9655 T1510 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 9656 T1510 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 9656 T1510 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 9657 T1510 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 9657 T1510 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 9657 T1510 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 9658 T1510 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 9660 T1510 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 9663 T1510 oass.SolrIndexSearcher.<init> Opening Searcher@379218 main
[junit4:junit4] 2> 9663 T1510 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363152020061/jetty2/tlog
[junit4:junit4] 2> 9663 T1510 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 9664 T1510 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 9668 T1511 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@379218 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 9669 T1510 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 9669 T1510 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 10905 T1474 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 10906 T1474 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:41037_",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:41037",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "numShards":"2"}
[junit4:junit4] 2> 10906 T1474 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4] 2> 10907 T1474 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4] 2> 10913 T1493 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 10913 T1473 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 10913 T1509 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 10913 T1480 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 11670 T1510 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 11671 T1510 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:41037 collection:collection1 shard:shard1
[junit4:junit4] 2> 11672 T1510 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 11681 T1456 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d623122600008 type:delete cxid:0x47 zxid:0x7f txntype:-1 reqpath:n/a Error Path:/solr/collections/collection1/leaders/shard1 Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders/shard1
[junit4:junit4] 2> 11682 T1510 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 11683 T1456 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d623122600008 type:create cxid:0x48 zxid:0x80 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 11686 T1510 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 11686 T1510 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 11687 T1510 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:41037/collection1/
[junit4:junit4] 2> 11687 T1510 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 11687 T1510 oasc.SyncStrategy.syncToMe http://127.0.0.1:41037/collection1/ has no replicas
[junit4:junit4] 2> 11688 T1510 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:41037/collection1/
[junit4:junit4] 2> 11688 T1510 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 11693 T1456 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d623122600008 type:create cxid:0x51 zxid:0x83 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 12419 T1474 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 12430 T1493 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 12430 T1473 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 12430 T1480 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 12430 T1509 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 12447 T1510 oasc.ZkController.register We are http://127.0.0.1:41037/collection1/ and leader is http://127.0.0.1:41037/collection1/
[junit4:junit4] 2> 12448 T1510 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:41037
[junit4:junit4] 2> 12448 T1510 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 12448 T1510 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 12449 T1510 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 12452 T1510 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 12454 T1451 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 12454 T1451 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 12455 T1451 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 12527 T1451 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4] 2> 12530 T1451 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:46523
[junit4:junit4] 2> 12532 T1451 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 12532 T1451 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 12533 T1451 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1363152032520
[junit4:junit4] 2> 12534 T1451 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1363152032520/solr.xml
[junit4:junit4] 2> 12535 T1451 oasc.CoreContainer.<init> New CoreContainer 1262515
[junit4:junit4] 2> 12536 T1451 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1363152032520/'
[junit4:junit4] 2> 12536 T1451 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1363152032520/'
[junit4:junit4] 2> 12562 T1451 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 12562 T1451 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 12563 T1451 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 12563 T1451 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 12563 T1451 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 12564 T1451 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 12564 T1451 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 12565 T1451 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 12565 T1451 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 12565 T1451 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 12573 T1451 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4] 2> 12585 T1451 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:49444/solr
[junit4:junit4] 2> 12586 T1451 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 12586 T1451 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:49444 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@1672d32
[junit4:junit4] 2> 12587 T1451 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 12587 T1522 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:49444. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 12588 T1522 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:49444, initiating session
[junit4:junit4] 2> 12588 T1453 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:37895
[junit4:junit4] 2> 12588 T1453 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:37895
[junit4:junit4] 2> 12589 T1455 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d623122600009 with negotiated timeout 20000 for client /127.0.0.1:37895
[junit4:junit4] 2> 12589 T1522 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:49444, sessionid = 0x13d623122600009, negotiated timeout = 20000
[junit4:junit4] 2> 12590 T1523 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1672d32 name:ZooKeeperConnection Watcher:127.0.0.1:49444 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 12590 T1451 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 12592 T1456 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d623122600009
[junit4:junit4] 2> 12593 T1523 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 12593 T1451 oaz.ZooKeeper.close Session: 0x13d623122600009 closed
[junit4:junit4] 2> 12593 T1453 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:37895 which had sessionid 0x13d623122600009
[junit4:junit4] 2> 12594 T1451 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 12597 T1451 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:49444/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@1aeea0c
[junit4:junit4] 2> 12598 T1524 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:49444. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 12598 T1451 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 12598 T1524 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:49444, initiating session
[junit4:junit4] 2> 12598 T1453 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:37896
[junit4:junit4] 2> 12599 T1453 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:37896
[junit4:junit4] 2> 12599 T1455 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d62312260000a with negotiated timeout 20000 for client /127.0.0.1:37896
[junit4:junit4] 2> 12599 T1524 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:49444, sessionid = 0x13d62312260000a, negotiated timeout = 20000
[junit4:junit4] 2> 12600 T1525 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1aeea0c name:ZooKeeperConnection Watcher:127.0.0.1:49444/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 12600 T1451 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 12601 T1456 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d62312260000a type:create cxid:0x1 zxid:0x90 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 12602 T1456 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d62312260000a type:create cxid:0x2 zxid:0x91 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 12603 T1451 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 13606 T1451 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46523_
[junit4:junit4] 2> 13607 T1456 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d62312260000a type:delete cxid:0xb zxid:0x92 txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:46523_ Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:46523_
[junit4:junit4] 2> 13608 T1451 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:46523_
[junit4:junit4] 2> 13613 T1480 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 13613 T1493 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 13613 T1473 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 13613 T1480 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 13614 T1509 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 13614 T1473 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 13613 T1493 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 13614 T1509 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 13614 T1525 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 13623 T1526 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1363152032520/collection1
[junit4:junit4] 2> 13623 T1526 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 13624 T1526 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 13625 T1526 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 13627 T1526 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1363152032520/collection1/'
[junit4:junit4] 2> 13627 T1526 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1363152032520/collection1/lib/README' to classloader
[junit4:junit4] 2> 13628 T1526 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1363152032520/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 13686 T1526 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 13737 T1526 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 13738 T1526 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4] 2> 13743 T1526 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4] 2> 13939 T1474 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 13940 T1474 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:41037_",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:41037",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "numShards":"2"}
[junit4:junit4] 2> 13945 T1509 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 13945 T1525 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 13945 T1473 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 13945 T1493 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 13945 T1480 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 14224 T1526 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 14230 T1526 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 14233 T1526 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 14248 T1526 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 14252 T1526 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 14258 T1526 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 14260 T1526 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 14260 T1526 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 14261 T1526 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 14262 T1526 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 14263 T1526 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 14263 T1526 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 14264 T1526 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1363152032520/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363152020061/jetty3/
[junit4:junit4] 2> 14264 T1526 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6197d7
[junit4:junit4] 2> 14265 T1526 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 14266 T1526 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363152020061/jetty3 forceNew: false
[junit4:junit4] 2> 14266 T1526 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1363152020061/jetty3/index/
[junit4:junit4] 2> 14266 T1526 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1363152020061/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 14267 T1526 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363152020061/jetty3/index forceNew: false
[junit4:junit4] 2> 14269 T1526 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@74edd1 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1bc29bd),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> 14269 T1526 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4] 2> 14273 T1526 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 14273 T1526 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 14274 T1526 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 14275 T1526 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 14276 T1526 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 14276 T1526 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 14277 T1526 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 14278 T1526 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 14278 T1526 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 14283 T1526 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 14288 T1526 oass.SolrIndexSearcher.<init> Opening Searcher@b70671 main
[junit4:junit4] 2> 14289 T1526 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363152020061/jetty3/tlog
[junit4:junit4] 2> 14290 T1526 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 14290 T1526 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 14298 T1527 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@b70671 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 14301 T1526 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 14301 T1526 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 15452 T1474 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 15453 T1474 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:46523_",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:46523",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "numShards":"2"}
[junit4:junit4] 2> 15454 T1474 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4] 2> 15454 T1474 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4:junit4] 2> 15460 T1480 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 15461 T1525 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 15461 T1473 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 15461 T1509 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 15460 T1493 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 16303 T1526 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 16303 T1526 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:46523 collection:collection1 shard:shard2
[junit4:junit4] 2> 16307 T1526 oasc.ZkController.register We are http://127.0.0.1:46523/collection1/ and leader is http://127.0.0.1:32962/collection1/
[junit4:junit4] 2> 16307 T1526 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:46523
[junit4:junit4] 2> 16307 T1526 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 16308 T1526 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C107 name=collection1 org.apache.solr.core.SolrCore@709f0e url=http://127.0.0.1:46523/collection1 node=127.0.0.1:46523_ C107_STATE=coll:collection1 core:collection1 props:{core=collection1, node_name=127.0.0.1:46523_, state=down, base_url=http://127.0.0.1:46523, collection=collection1}
[junit4:junit4] 2> 16308 T1528 C107 P46523 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 16309 T1528 C107 P46523 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 16309 T1526 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 16309 T1528 C107 P46523 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 16310 T1528 C107 P46523 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 16310 T1451 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 16310 T1528 C107 P46523 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 16311 T1451 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 16311 T1451 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 16370 T1451 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4] 2> 16372 T1451 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:55276
[junit4:junit4] 2> 16372 T1451 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 16373 T1451 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 16373 T1451 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1363152036373
[junit4:junit4] 2> 16373 T1451 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1363152036373/solr.xml
[junit4:junit4] 2> 16374 T1451 oasc.CoreContainer.<init> New CoreContainer 17736600
[junit4:junit4] 2> 16374 T1451 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1363152036373/'
[junit4:junit4] 2> 16374 T1451 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1363152036373/'
[junit4:junit4] 2> 16391 T1451 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 16392 T1451 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 16392 T1451 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 16392 T1451 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 16393 T1451 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 16393 T1451 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 16393 T1451 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 16394 T1451 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 16394 T1451 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 16394 T1451 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 16400 T1451 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4] 2> 16411 T1451 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:49444/solr
[junit4:junit4] 2> 16412 T1451 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 16412 T1451 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:49444 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@161ad72
[junit4:junit4] 2> 16413 T1451 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 16414 T1539 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:49444. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 16414 T1539 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:49444, initiating session
[junit4:junit4] 2> 16414 T1453 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:37898
[junit4:junit4] 2> 16415 T1453 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:37898
[junit4:junit4] 2> 16416 T1455 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d62312260000b with negotiated timeout 20000 for client /127.0.0.1:37898
[junit4:junit4] 2> 16416 T1539 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:49444, sessionid = 0x13d62312260000b, negotiated timeout = 20000
[junit4:junit4] 2> 16416 T1540 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@161ad72 name:ZooKeeperConnection Watcher:127.0.0.1:49444 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 16417 T1451 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 16418 T1456 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d62312260000b
[junit4:junit4] 2> 16421 T1453 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:37898 which had sessionid 0x13d62312260000b
[junit4:junit4] 2> 16421 T1540 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 16421 T1451 oaz.ZooKeeper.close Session: 0x13d62312260000b closed
[junit4:junit4] 2> 16422 T1451 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 16428 T1451 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:49444/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@1d6a8e9
[junit4:junit4] 2> 16429 T1451 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 16430 T1541 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:49444. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 16430 T1541 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:49444, initiating session
[junit4:junit4] 2> 16431 T1453 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:37899
[junit4:junit4] 2> 16431 T1453 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:37899
[junit4:junit4] 2> 16432 T1455 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d62312260000c with negotiated timeout 20000 for client /127.0.0.1:37899
[junit4:junit4] 2> 16432 T1541 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:49444, sessionid = 0x13d62312260000c, negotiated timeout = 20000
[junit4:junit4] 2> 16432 T1542 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d6a8e9 name:ZooKeeperConnection Watcher:127.0.0.1:49444/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 16433 T1451 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 16435 T1456 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d62312260000c type:create cxid:0x1 zxid:0xa3 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 16436 T1456 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d62312260000c type:create cxid:0x2 zxid:0xa4 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 16439 T1451 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 16968 T1474 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 16969 T1474 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:46523_",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:46523",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "numShards":"2"}
[junit4:junit4] 2> 16975 T1509 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 16975 T1493 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 16975 T1542 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 16975 T1473 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 16975 T1525 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 16975 T1480 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 17316 T1487 oass.SolrDispatchFilter.handleAdminRequest /admin/cores {onlyIfLeader=true&nodeName=127.0.0.1:46523_&core=collection1&version=2&coreNodeName=4&action=PREPRECOVERY&state=recovering&wt=javabin&checkLive=true} status=0 QTime=1000
[junit4:junit4] 2> 17443 T1451 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55276_
[junit4:junit4] 2> 17444 T1456 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d62312260000c type:delete cxid:0xd zxid:0xa9 txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:55276_ Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:55276_
[junit4:junit4] 2> 17446 T1451 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:55276_
[junit4:junit4] 2> 17447 T1525 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 17448 T1480 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 17448 T1542 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 17448 T1509 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 17448 T1473 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 17449 T1473 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 17448 T1493 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 17449 T1480 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 17449 T1525 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 17449 T1509 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 17450 T1542 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 17450 T1493 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 17454 T1543 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1363152036373/collection1
[junit4:junit4] 2> 17454 T1543 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 17454 T1543 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 17455 T1543 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 17456 T1543 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1363152036373/collection1/'
[junit4:junit4] 2> 17456 T1543 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1363152036373/collection1/lib/README' to classloader
[junit4:junit4] 2> 17456 T1543 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1363152036373/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 17493 T1543 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 17530 T1543 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 17531 T1543 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4] 2> 17535 T1543 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4] 2> 17950 T1543 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 17956 T1543 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 17959 T1543 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 17968 T1543 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 17971 T1543 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 17974 T1543 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 17974 T1543 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 17975 T1543 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 17975 T1543 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 17975 T1543 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 17976 T1543 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 17976 T1543 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 17976 T1543 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1363152036373/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363152020061/jetty4/
[junit4:junit4] 2> 17976 T1543 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6197d7
[junit4:junit4] 2> 17977 T1543 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 17977 T1543 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363152020061/jetty4 forceNew: false
[junit4:junit4] 2> 17977 T1543 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1363152020061/jetty4/index/
[junit4:junit4] 2> 17977 T1543 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1363152020061/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 17978 T1543 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363152020061/jetty4/index forceNew: false
[junit4:junit4] 2> 17979 T1543 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@fb09fc lockFactory=org.apache.lucene.store.NativeFSLockFactory@a1f39d),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> 17979 T1543 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4] 2> 17980 T1543 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 17981 T1543 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 17981 T1543 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 17981 T1543 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 17982 T1543 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 17982 T1543 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 17982 T1543 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 17983 T1543 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 17983 T1543 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 17985 T1543 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 17988 T1543 oass.SolrIndexSearcher.<init> Opening Searcher@1620b8f main
[junit4:junit4] 2> 17988 T1543 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363152020061/jetty4/tlog
[junit4:junit4] 2> 17989 T1543 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 17989 T1543 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 17992 T1544 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1620b8f main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 17994 T1543 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 17994 T1543 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 18480 T1474 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 18482 T1474 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:55276_",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:55276",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "numShards":"2"}
[junit4:junit4] 2> 18482 T1474 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4] 2> 18483 T1474 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4] 2> 18487 T1480 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 18487 T1493 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 18488 T1542 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 18488 T1525 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 18487 T1473 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 18487 T1509 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 18995 T1543 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 18996 T1543 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:55276 collection:collection1 shard:shard1
[junit4:junit4] 2> 19000 T1543 oasc.ZkController.register We are http://127.0.0.1:55276/collection1/ and leader is http://127.0.0.1:41037/collection1/
[junit4:junit4] 2> 19000 T1543 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:55276
[junit4:junit4] 2> 19001 T1543 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 19001 T1543 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C108 name=collection1 org.apache.solr.core.SolrCore@1551867 url=http://127.0.0.1:55276/collection1 node=127.0.0.1:55276_ C108_STATE=coll:collection1 core:collection1 props:{core=collection1, node_name=127.0.0.1:55276_, state=down, base_url=http://127.0.0.1:55276, collection=collection1}
[junit4:junit4] 2> 19002 T1545 C108 P55276 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 19003 T1545 C108 P55276 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 19003 T1543 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 19003 T1545 C108 P55276 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 19004 T1545 C108 P55276 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 19005 T1451 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 19005 T1545 C108 P55276 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 19006 T1451 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 19007 T1451 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 19015 T1451 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 19016 T1451 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 19024 T1487 oasc.CoreContainer.create Creating SolrCore 'onenodecollectioncore' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363152023458/onenodecollectioncore
[junit4:junit4] 2> 19024 T1487 oasc.ZkController.createCollectionZkNode Check for collection zkNode:onenodecollection
[junit4:junit4] 2> 19025 T1487 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:onenodecollection
[junit4:junit4] 2> 19026 T1487 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4] 2> 19028 T1487 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
[junit4:junit4] 2> 19029 T1487 oascc.SolrZkClient.makePath makePath: /collections/onenodecollection
[junit4:junit4] 2> 19032 T1487 oasc.ZkController.readConfigName Load collection config from:/collections/onenodecollection
[junit4:junit4] 2> 19033 T1487 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363152023458/onenodecollectioncore/'
[junit4:junit4] 2> 19074 T1487 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 19135 T1487 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 19136 T1487 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4] 2> 19140 T1487 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4] 2> C107_STATE=coll:collection1 core:collection1 props:{core=collection1, node_name=127.0.0.1:46523_, shard=shard2, state=recovering, base_url=http://127.0.0.1:46523, collection=collection1}
[junit4:junit4] 2> 19317 T1528 C107 P46523 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:32962/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 19318 T1528 C107 P46523 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:46523 START replicas=[http://127.0.0.1:32962/collection1/] nUpdates=100
[junit4:junit4] 2> 19318 T1528 C107 P46523 oasu.PeerSync.sync WARNING no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 19318 T1528 C107 P46523 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 19319 T1528 C107 P46523 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 19319 T1528 C107 P46523 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 19319 T1528 C107 P46523 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 19319 T1528 C107 P46523 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:32962/collection1/. core=collection1
[junit4:junit4] 2> 19320 T1528 C107 P46523 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C109 name=collection1 org.apache.solr.core.SolrCore@bede74 url=http://127.0.0.1:32962/collection1 node=127.0.0.1:32962_ C109_STATE=coll:collection1 core:collection1 props:{core=collection1, node_name=127.0.0.1:32962_, shard=shard2, state=active, base_url=http://127.0.0.1:32962, collection=collection1, leader=true}
[junit4:junit4] 2> 19323 T1486 C109 P32962 REQ /get {version=2&distrib=false&getVersions=100&qt=/get&wt=javabin} status=0 QTime=1
[junit4:junit4] 2> 19324 T1486 C109 P32962 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 19327 T1486 C109 P32962 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@171782c lockFactory=org.apache.lucene.store.NativeFSLockFactory@1f6686),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> 19328 T1486 C109 P32962 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4] 2> 19328 T1486 C109 P32962 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@171782c lockFactory=org.apache.lucene.store.NativeFSLockFactory@1f6686),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@171782c lockFactory=org.apache.lucene.store.NativeFSLockFactory@1f6686),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4] 2> 19329 T1486 C109 P32962 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[segments_2]
[junit4:junit4] 2> 19329 T1486 C109 P32962 oass.SolrIndexSearcher.<init> Opening Searcher@18cf09e realtime
[junit4:junit4] 2> 19329 T1486 C109 P32962 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 19330 T1486 C109 P32962 /update {version=2&waitSearcher=true&softCommit=false&commit_end_point=true&commit=true&wt=javabin&openSearcher=false} {commit=} 0 6
[junit4:junit4] 2> 19330 T1528 C107 P46523 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 19331 T1528 C107 P46523 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 19333 T1489 C109 P32962 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 19334 T1489 C109 P32962 REQ /replication {command=indexversion&version=2&qt=/replication&wt=javabin} status=0 QTime=2
[junit4:junit4] 2> 19334 T1528 C107 P46523 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 19334 T1528 C107 P46523 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 19334 T1528 C107 P46523 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 19350 T1486 C109 P32962 REQ /replication {command=filelist&version=2&generation=2&qt=/replication&wt=javabin} status=0 QTime=0
[junit4:junit4] 2> 19351 T1528 C107 P46523 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 19352 T1528 C107 P46523 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363152020061/jetty3/index.20130312222039409 forceNew: false
[junit4:junit4] 2> 19352 T1528 C107 P46523 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@40628e lockFactory=org.apache.lucene.store.NativeFSLockFactory@3ac012) fullCopy=false
[junit4:junit4] 2> 19355 T1488 C109 P32962 REQ /replication {command=filecontent&generation=2&checksum=true&qt=/replication&wt=filestream&file=segments_2} status=0 QTime=0
[junit4:junit4] 2> 19356 T1528 C107 P46523 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 19358 T1528 C107 P46523 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 19358 T1528 C107 P46523 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 19359 T1528 C107 P46523 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@74edd1 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1bc29bd),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@74edd1 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1bc29bd),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4] 2> 19360 T1528 C107 P46523 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[segments_2]
[junit4:junit4] 2> 19360 T1528 C107 P46523 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 19361 T1528 C107 P46523 oass.SolrIndexSearcher.<init> Opening Searcher@10d4361 main
[junit4:junit4] 2> 19362 T1527 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@10d4361 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 19363 T1528 C107 P46523 oash.SnapPuller.fetchLatestIndex removing temporary index download directory files MockDirWrapper(org.apache.lucene.store.RAMDirectory@40628e lockFactory=org.apache.lucene.store.NativeFSLockFactory@3ac012)
[junit4:junit4] 2> 19363 T1528 C107 P46523 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 19364 T1528 C107 P46523 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 19364 T1528 C107 P46523 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 19364 T1528 C107 P46523 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 19366 T1528 C107 P46523 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 19616 T1487 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 19622 T1487 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 19625 T1487 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 19638 T1487 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 19642 T1487 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 19647 T1487 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 19648 T1487 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 19649 T1487 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 19649 T1487 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 19651 T1487 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 19651 T1487 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 19652 T1487 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 19652 T1487 oasc.SolrCore.<init> [onenodecollectioncore] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363152023458/onenodecollectioncore/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1363152020061/onenodecollection/
[junit4:junit4] 2> 19653 T1487 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6197d7
[junit4:junit4] 2> 19653 T1487 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 19654 T1487 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1363152020061/onenodecollection forceNew: false
[junit4:junit4] 2> 19655 T1487 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1363152020061/onenodecollection/index/
[junit4:junit4] 2> 19655 T1487 oasc.SolrCore.initIndex WARNING [onenodecollectioncore] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1363152020061/onenodecollection/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 19656 T1487 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1363152020061/onenodecollection/index forceNew: false
[junit4:junit4] 2> 19657 T1487 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@13e9469 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1f8cbe),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> 19658 T1487 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4] 2> 19660 T1487 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 19661 T1487 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 19662 T1487 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 19663 T1487 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 19663 T1487 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 19664 T1487 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 19664 T1487 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 19665 T1487 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 19666 T1487 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 19670 T1487 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 19674 T1487 oass.SolrIndexSearcher.<init> Opening Searcher@1e692a3 main
[junit4:junit4] 2> 19675 T1487 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1363152020061/onenodecollection/tlog
[junit4:junit4] 2> 19675 T1487 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 19676 T1487 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 19681 T1549 oasc.SolrCore.registerSearcher [onenodecollectioncore] Registered new searcher Searcher@1e692a3 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 19683 T1487 oasc.ZkController.publish publishing core=onenodecollectioncore state=down
[junit4:junit4] 2> 19993 T1474 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 19995 T1474 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:55276_",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:55276",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "numShards":"2"}
[junit4:junit4] 2> 19999 T1474 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:46523_",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:46523",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "numShards":"2"}
[junit4:junit4] 2> 20004 T1474 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "core":"onenodecollectioncore",
[junit4:junit4] 2> "node_name":"127.0.0.1:32962_",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:32962",
[junit4:junit4] 2> "roles":"none",
[junit4:junit4] 2> "collection":"onenodecollection",
[junit4:junit4] 2> "numShards":"1"}
[junit4:junit4] 2> 20004 T1474 oasc.Overseer$ClusterStateUpdater.createCollection Create collection onenodecollection with numShards 1
[junit4:junit4] 2> 20005 T1474 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4] 2> 20010 T1480 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 20010 T1493 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 20011 T1525 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 20010 T1542 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 20010 T1509 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 20010 T1473 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 20015 T1504 oass.SolrDispatchFilter.handleAdminRequest /admin/cores {onlyIfLeader=true&nodeName=127.0.0.1:55276_&core=collection1&version=2&coreNodeName=5&action=PREPRECOVERY&state=recovering&wt=javabin&checkLive=true} status=0 QTime=1000
[junit4:junit4] 2> 20685 T1487 oasc.CoreContainer.registerCore registering core: onenodecollectioncore
[junit4:junit4] 2> 20685 T1487 oasc.ZkController.register Register replica - core:onenodecollectioncore address:http://127.0.0.1:32962 collection:onenodecollection shard:shard1
[junit4:junit4] 2> 20686 T1487 oascc.SolrZkClient.makePath makePath: /collections/onenodecollection/leader_elect/shard1/election
[junit4:junit4] 2> 20691 T1456 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d623122600006 type:delete cxid:0xa8 zxid:0xc4 txntype:-1 reqpath:n/a Error Path:/solr/collections/onenodecollection/leaders Error:KeeperErrorCode = NoNode for /solr/collections/onenodecollection/leaders
[junit4:junit4] 2> 20692 T1487 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 20692 T1456 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d623122600006 type:create cxid:0xa9 zxid:0xc5 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 20693 T1487 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 20693 T1487 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 20694 T1487 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:32962/onenodecollectioncore/
[junit4:junit4] 2> 20694 T1487 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 20694 T1487 oasc.SyncStrategy.syncToMe http://127.0.0.1:32962/onenodecollectioncore/ has no replicas
[junit4:junit4] 2> 20695 T1487 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:32962/onenodecollectioncore/
[junit4:junit4] 2> 20695 T1487 oascc.SolrZkClient.makePath makePath: /collections/onenodecollection/leaders/shard1
[junit4:junit4] 2> 20697 T1456 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d623122600006 type:create cxid:0xb3 zxid:0xc9 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 21522 T1474 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 21532 T1542 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 21532 T1509 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 21533 T1480 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 21533 T1473 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 21532 T1493 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 21532 T1525 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 21551 T1487 oasc.ZkController.register We are http://127.0.0.1:32962/onenodecollectioncore/ and leader is http://127.0.0.1:32962/onenodecollectioncore/
[junit4:junit4] 2> 21552 T1487 oasc.ZkController.register No LogReplay needed for core=onenodecollectioncore baseURL=http://127.0.0.1:32962
[junit4:junit4] 2> 21552 T1487 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 21553 T1487 oasc.ZkController.publish publishing core=onenodecollectioncore state=active
[junit4:junit4] 2> 21555 T1487 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 21556 T1487 oass.SolrDispatchFilter.handleAdminRequest /admin/cores {name=onenodecollectioncore&version=2&action=CREATE&roles=none&collection=onenodecollection&wt=javabin&numShards=1&dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1363152020061/onenodecollection} status=0 QTime=2533
[junit4:junit4] 2> 21557 T1451 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: onenodecollection failOnTimeout:true timeout (sec):230
[junit4:junit4] 2> 21559 T1451 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> C108_STATE=coll:collection1 core:collection1 props:{core=collection1, node_name=127.0.0.1:55276_, shard=shard1, state=recovering, base_url=http://127.0.0.1:55276, collection=collection1}
[junit4:junit4] 2> 22017 T1545 C108 P55276 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:41037/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 22017 T1545 C108 P55276 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:55276 START replicas=[http://127.0.0.1:41037/collection1/] nUpdates=100
[junit4:junit4] 2> 22018 T1545 C108 P55276 oasu.PeerSync.sync WARNING no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 22018 T1545 C108 P55276 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 22019 T1545 C108 P55276 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 22019 T1545 C108 P55276 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 22019 T1545 C108 P55276 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 22020 T1545 C108 P55276 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:41037/collection1/. core=collection1
[junit4:junit4] 2> 22020 T1545 C108 P55276 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C110 name=collection1 org.apache.solr.core.SolrCore@1a3651e url=http://127.0.0.1:41037/collection1 node=127.0.0.1:41037_ C110_STATE=coll:collection1 core:collection1 props:{core=collection1, node_name=127.0.0.1:41037_, shard=shard1, state=active, base_url=http://127.0.0.1:41037, collection=collection1, leader=true}
[junit4:junit4] 2> 22027 T1504 C110 P41037 REQ /get {version=2&distrib=false&getVersions=100&qt=/get&wt=javabin} status=0 QTime=1
[junit4:junit4] 2> 22029 T1503 C110 P41037 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 22031 T1503 C110 P41037 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1b190f2 lockFactory=org.apache.lucene.store.NativeFSLockFactory@81c3be),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> 22032 T1503 C110 P41037 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4] 2> 22033 T1503 C110 P41037 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1b190f2 lockFactory=org.apache.lucene.store.NativeFSLockFactory@81c3be),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1b190f2 lockFactory=org.apache.lucene.store.NativeFSLockFactory@81c3be),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4] 2> 22033 T1503 C110 P41037 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[segments_2]
[junit4:junit4] 2> 22034 T1503 C110 P41037 oass.SolrIndexSearcher.<init> Opening Searcher@1189a66 realtime
[junit4:junit4] 2> 22035 T1503 C110 P41037 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 22035 T1503 C110 P41037 /update {version=2&waitSearcher=true&softCommit=false&commit_end_point=true&commit=true&wt=javabin&openSearcher=false} {commit=} 0 7
[junit4:junit4] 2> 22036 T1545 C108 P55276 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 22037 T1545 C108 P55276 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 22039 T1502 C110 P41037 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 22039 T1502 C110 P41037 REQ /replication {command=indexversion&version=2&qt=/replication&wt=javabin} status=0 QTime=1
[junit4:junit4] 2> 22040 T1545 C108 P55276 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 22040 T1545 C108 P55276 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 22041 T1545 C108 P55276 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 22043 T1504 C110 P41037 REQ /replication {command=filelist&version=2&generation=2&qt=/replication&wt=javabin} status=0 QTime=0
[junit4:junit4] 2> 22044 T1545 C108 P55276 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 22045 T1545 C108 P55276 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363152020061/jetty4/index.20130312222042102 forceNew: false
[junit4:junit4] 2> 22045 T1545 C108 P55276 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@1538ee7 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1a770e4) fullCopy=false
[junit4:junit4] 2> 22048 T1503 C110 P41037 REQ /replication {command=filecontent&generation=2&checksum=true&qt=/replication&wt=filestream&file=segments_2} status=0 QTime=0
[junit4:junit4] 2> 22050 T1545 C108 P55276 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 22051 T1545 C108 P55276 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 22052 T1545 C108 P55276 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 22053 T1545 C108 P55276 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@fb09fc lockFactory=org.apache.lucene.store.NativeFSLockFactory@a1f39d),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@fb09fc lockFactory=org.apache.lucene.store.NativeFSLockFactory@a1f39d),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4] 2> 22054 T1545 C108 P55276 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[segments_2]
[junit4:junit4] 2> 22055 T1545 C108 P55276 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 22055 T1545 C108 P55276 oass.SolrIndexSearcher.<init> Opening Searcher@ff9d49 main
[junit4:junit4] 2> 22057 T1544 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@ff9d49 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 22058 T1545 C108 P55276 oash.SnapPuller.fetchLatestIndex removing temporary index download directory files MockDirWrapper(org.apache.lucene.store.RAMDirectory@1538ee7 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1a770e4)
[junit4:junit4] 2> 22058 T1545 C108 P55276 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 22059 T1545 C108 P55276 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 22059 T1545 C108 P55276 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 22060 T1545 C108 P55276 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 22061 T1545 C108 P55276 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 22561 T1451 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 23040 T1474 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 23042 T1474 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "core":"onenodecollectioncore",
[junit4:junit4] 2> "node_name":"127.0.0.1:32962_",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:32962",
[junit4:junit4] 2> "roles":"none",
[junit4:junit4] 2> "collection":"onenodecollection",
[junit4:junit4] 2> "numShards":"1"}
[junit4:junit4] 2> 23046 T1474 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:55276_",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:55276",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "numShards":"2"}
[junit4:junit4] 2> 23051 T1525 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 23051 T1509 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 23052 T1493 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 23051 T1480 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 23051 T1542 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 23052 T1473 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 23562 T1451 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 23564 T1451 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: onenodecollection
[junit4:junit4] 2> 23565 T1451 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 23581 T1505 oejs.Response.sendError WARNING Committed before 404 Can not find: /onenodecollectioncore/update
[junit4:junit4] 2> 23582 T1451 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:49444 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@69602a
[junit4:junit4] 2> 23582 T1505 oejs.ServletHandler.doHandle WARNING /onenodecollectioncore/update java.lang.IllegalStateException: Committed
[junit4:junit4] 2> at org.eclipse.jetty.server.Response.resetBuffer(Response.java:1136)
[junit4:junit4] 2> at org.eclipse.jetty.server.Response.sendError(Response.java:314)
[junit4:junit4] 2> at org.apache.solr.client.solrj.embedded.JettySolrRunner$Servlet404.service(JettySolrRunner.java:465)
[junit4:junit4] 2> at javax.servlet.http.HttpServlet.service(HttpServlet.java:770)
[junit4:junit4] 2> at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:669)
[junit4:junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1336)
[junit4:junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:379)
[junit4:junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:141)
[junit4:junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
[junit4:junit4] 2> at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:135)
[junit4:junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
[junit4:junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453)
[junit4:junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.GzipHandler.handle(GzipHandler.java:275)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1074)
[junit4:junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:382)
[junit4:junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1006)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
[junit4:junit4] 2> at org.eclipse.jetty.server.Server.handle(Server.java:365)
[junit4:junit4] 2> at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:485)
[junit4:junit4] 2> at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:937)
[junit4:junit4] 2> at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:998)
[junit4:junit4] 2> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:948)
[junit4:junit4] 2> at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
[junit4:junit4] 2> at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
[junit4:junit4] 2> at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:628)
[junit4:junit4] 2> at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
[junit4:junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4:junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:722)
[junit4:junit4] 2>
[junit4:junit4] 2> 23583 T1552 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:49444. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 23583 T1451 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 23585 T1552 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:49444, initiating session
[junit4:junit4] 2> 23585 T1453 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:37910
[junit4:junit4] 2> 23586 T1453 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:37910
[junit4:junit4] 2> 23587 T1455 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d62312260000d with negotiated timeout 10000 for client /127.0.0.1:37910
[junit4:junit4] 2> 23587 T1552 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:49444, sessionid = 0x13d62312260000d, negotiated timeout = 10000
[junit4:junit4] 2> 23587 T1553 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@69602a name:ZooKeeperConnection Watcher:127.0.0.1:49444 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 23588 T1451 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 23641 T1456 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d62312260000d
[junit4:junit4] 2> 23642 T1553 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 23642 T1453 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:37910 which had sessionid 0x13d62312260000d
[junit4:junit4] 2> 23642 T1451 oaz.ZooKeeper.close Session: 0x13d62312260000d closed
[junit4:junit4] 2> 23643 T1456 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d623122600004
[junit4:junit4] 2> 23644 T1453 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:37889 which had sessionid 0x13d623122600004
[junit4:junit4] 2> 23644 T1480 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 23644 T1451 oaz.ZooKeeper.close Session: 0x13d623122600004 closed
[junit4:junit4] 2> 23648 T1451 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 55548
[junit4:junit4] 2> 23649 T1451 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=21041180
[junit4:junit4] 2> 24560 T1474 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 24560 T1474 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:55548_",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:55548",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"control_collection"}
[junit4:junit4] 2> 24563 T1509 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 24564 T1525 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 24564 T1542 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 24564 T1493 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 24564 T1473 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 25651 T1451 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 25651 T1451 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 25654 T1451 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@1dc4284
[junit4:junit4] 2> 25658 T1451 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 25659 T1451 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 25659 T1451 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 25659 T1451 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 25660 T1451 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 25660 T1451 oasc.CachingDirectoryFactory.close Closing directory when closing factory: ./org.apache.solr.cloud.BasicDistributedZk2Test-1363152020061/control/data/index
[junit4:junit4] 2> 25661 T1451 oasc.CachingDirectoryFactory.closeDirectory Closing directory: ./org.apache.solr.cloud.BasicDistributedZk2Test-1363152020061/control/data/index
[junit4:junit4] 2> 25661 T1451 oasc.CachingDirectoryFactory.close Closing directory when closing factory: ./org.apache.solr.cloud.BasicDistributedZk2Test-1363152020061/control/data
[junit4:junit4] 2> 25661 T1451 oasc.CachingDirectoryFactory.closeDirectory Closing directory: ./org.apache.solr.cloud.BasicDistributedZk2Test-1363152020061/control/data
[junit4:junit4] 2> 25662 T1474 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=89335530786914307-127.0.0.1:55548_-n_0000000000) am no longer a leader.
[junit4:junit4] 2> 25663 T1456 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d623122600003
[junit4:junit4] 2> 25664 T1542 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 25664 T1525 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 25664 T1453 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:37887 which had sessionid 0x13d623122600003
[junit4:junit4] 2> 25665 T1509 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 25664 T1451 oaz.ZooKeeper.close Session: 0x13d623122600003 closed
[junit4:junit4] 2> 25665 T1509 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 25665 T1456 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d623122600006 type:delete cxid:0xc0 zxid:0xe5 txntype:-1 reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4] 2> 25666 T1525 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 25666 T1493 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 25666 T1542 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 25668 T1493 oasc.Overseer.start Overseer (id=89335530786914310-127.0.0.1:32962_-n_0000000001) starting
[junit4:junit4] 2> 25669 T1456 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d623122600006 type:create cxid:0xc5 zxid:0xe7 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 25669 T1456 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d623122600006 type:create cxid:0xc6 zxid:0xe8 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 25670 T1456 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d623122600006 type:create cxid:0xc7 zxid:0xe9 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 25671 T1456 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d623122600006 type:create cxid:0xc8 zxid:0xea txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 25671 T1555 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 25671 T1493 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 25672 T1554 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 25673 T1493 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 25674 T1554 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 25675 T1554 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:55548_",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:55548",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "numShards":"2"}
[junit4:junit4] 2> 25678 T1493 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 25678 T1525 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 25678 T1509 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 25679 T1542 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 25687 T1451 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
[junit4:junit4] 2> 25739 T1451 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 32962
[junit4:junit4] 2> 25740 T1451 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=33445359
[junit4:junit4] 2> 26964 T1473 oascc.ZkStateReader$3.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 26964 T1473 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 26965 T1473 oascc.ZkStateReader$2.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 26965 T1473 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 27190 T1554 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 27191 T1554 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "core":"onenodecollectioncore",
[junit4:junit4] 2> "node_name":"127.0.0.1:32962_",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:32962__onenodecollectioncore",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:32962",
[junit4:junit4] 2> "roles":"none",
[junit4:junit4] 2> "collection":"onenodecollection"}
[junit4:junit4] 2> 27193 T1554 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:32962_",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:32962",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1"}
[junit4:junit4] 2> 27197 T1493 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred
[...truncated too long message...]
ad6",
[junit4:junit4] 1> "node_name":"127.0.0.1:32880_eje",
[junit4:junit4] 1> "shard":"shard2",
[junit4:junit4] 1> "state":"recovery_failed",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:32880/eje",
[junit4:junit4] 1> "collection":"multiunload"},
[junit4:junit4] 1> "127.0.0.1:32880_eje_multiunload11":{
[junit4:junit4] 1> "core":"multiunload11",
[junit4:junit4] 1> "node_name":"127.0.0.1:32880_eje",
[junit4:junit4] 1> "shard":"shard2",
[junit4:junit4] 1> "state":"recovery_failed",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:32880/eje",
[junit4:junit4] 1> "collection":"multiunload"}}}}}}
[junit4:junit4] 1> /zookeeper (1)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1>
[junit4:junit4] 1>
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=UnloadDistributedZkTest -Dtests.method=testDistribSearch -Dtests.seed=1763FA0A2E88829F -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ar_MA -Dtests.timezone=Pacific/Noumea -Dtests.file.encoding=UTF-8
[junit4:junit4] FAILURE 135s J1 | UnloadDistributedZkTest.testDistribSearch <<<
[junit4:junit4] > Throwable #1: java.lang.AssertionError: Still found shard
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([1763FA0A2E88829F:9685741259D7E2A3]:0)
[junit4:junit4] > at org.junit.Assert.fail(Assert.java:93)
[junit4:junit4] > at org.apache.solr.cloud.UnloadDistributedZkTest.testUnloadShardAndCollection(UnloadDistributedZkTest.java:124)
[junit4:junit4] > at org.apache.solr.cloud.UnloadDistributedZkTest.doTest(UnloadDistributedZkTest.java:77)
[junit4:junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:806)
[junit4:junit4] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit4:junit4] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
[junit4:junit4] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit4:junit4] > at java.lang.reflect.Method.invoke(Method.java:487)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
[junit4:junit4] > at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
[junit4:junit4] > at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
[junit4:junit4] > at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4:junit4] > at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
[junit4:junit4] > at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
[junit4:junit4] > at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
[junit4:junit4] > at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
[junit4:junit4] > at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4] > at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
[junit4:junit4] > at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4:junit4] > at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
[junit4:junit4] > at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:722)
[junit4:junit4] 2> 135090 T1565 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4] 2> 135101 T1564 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
[junit4:junit4] 2> NOTE: test params are: codec=CheapBastard, sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=ar_MA, timezone=Pacific/Noumea
[junit4:junit4] 2> NOTE: Linux 3.2.0-38-generic i386/Oracle Corporation 1.8.0-ea (32-bit)/cpus=8,threads=2,free=43800080,total=203784192
[junit4:junit4] 2> NOTE: All tests run in this JVM: [ShardRoutingCustomTest, TestCodecSupport, TestRangeQuery, ZkNodePropsTest, TestAnalyzedSuggestions, TestSolrXMLSerializer, TestPropInject, TestDistributedGrouping, TestUtils, XsltUpdateRequestHandlerTest, NoCacheHeaderTest, FieldAnalysisRequestHandlerTest, HighlighterConfigTest, TestRecovery, TestFieldTypeResource, JSONWriterTest, ChaosMonkeySafeLeaderTest, RequiredFieldsTest, TestBinaryField, TestFiltering, TestIndexingPerformance, UpdateParamsTest, TestSolrDiscoveryProperties, TestReload, CoreContainerCoreInitFailuresTest, TestQueryTypes, OpenExchangeRatesOrgProviderTest, XmlUpdateRequestHandlerTest, ClusterStateTest, HighlighterTest, SolrCoreCheckLockOnStartupTest, TestLuceneMatchVersion, SignatureUpdateProcessorFactoryTest, TestSystemIdResolver, SpellCheckComponentTest, DisMaxRequestHandlerTest, SolrCmdDistributorTest, ZkSolrClientTest, TestSuggestSpellingConverter, SampleTest, DistanceFunctionTest, PingRequestHandlerTest, CurrencyFieldOpenExchangeTest, TestUpdate, TestRandomFaceting, TestFieldCollectionResource, IndexBasedSpellCheckerTest, CurrencyFieldXmlFileTest, SortByFunctionTest, IndexSchemaRuntimeFieldTest, ConvertedLegacyTest, TestStressReorder, TestWordDelimiterFilterFactory, BasicZkTest, BadIndexSchemaTest, SimplePostToolTest, TestArbitraryIndexDir, TestJmxMonitoredMap, TestSolrJ, TestDefaultSimilarityFactory, EchoParamsTest, TestHashPartitioner, BasicDistributedZk2Test, UnloadDistributedZkTest]
[junit4:junit4] Completed on J1 in 135.37s, 1 test, 1 failure <<< FAILURES!
[...truncated 420 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:381: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:361: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:183: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:449: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1213: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:877: There were test failures: 271 suites, 1141 tests, 1 error, 1 failure, 13 ignored (7 assumptions)
Total time: 45 minutes 57 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Description set: Java: 32bit/jdk1.8.0-ea-b79 -client -XX:+UseSerialGC
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0-ea-b79) - Build #
4689 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/4689/
Java: 64bit/jdk1.8.0-ea-b79 -XX:+UseConcMarkSweepGC
1 tests failed.
FAILED: org.apache.solr.cloud.UnloadDistributedZkTest.testDistribSearch
Error Message:
Still found shard
Stack Trace:
java.lang.AssertionError: Still found shard
at __randomizedtesting.SeedInfo.seed([8ED9F67CEAE3B8A2:F3F78649DBCD89E]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.UnloadDistributedZkTest.testUnloadShardAndCollection(UnloadDistributedZkTest.java:124)
at org.apache.solr.cloud.UnloadDistributedZkTest.doTest(UnloadDistributedZkTest.java:77)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:806)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:487)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:722)
Build Log:
[...truncated 9210 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.UnloadDistributedZkTest
[junit4:junit4] 2> 0 T1804 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /fy/td
[junit4:junit4] 2> 4 T1804 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-UnloadDistributedZkTest-1363157961938
[junit4:junit4] 2> 5 T1804 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 5 T1805 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 6 T1805 oazs.ZooKeeperServer.setTickTime tickTime set to 1000
[junit4:junit4] 2> 6 T1805 oazs.ZooKeeperServer.setMinSessionTimeout minSessionTimeout set to -1
[junit4:junit4] 2> 6 T1805 oazs.ZooKeeperServer.setMaxSessionTimeout maxSessionTimeout set to -1
[junit4:junit4] 2> 6 T1805 oazs.NIOServerCnxnFactory.configure binding to port 0.0.0.0/0.0.0.0:0
[junit4:junit4] 2> 7 T1805 oazsp.FileTxnSnapLog.save Snapshotting: 0x0 to /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.UnloadDistributedZkTest-1363157961937/zookeeper/server1/data/version-2/snapshot.0
[junit4:junit4] 2> 105 T1804 oasc.ZkTestServer.run start zk server on port:58818
[junit4:junit4] 2> 106 T1804 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:58818 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@6ea5549f
[junit4:junit4] 2> 107 T1804 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 107 T1810 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:58818. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 107 T1810 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:58818, initiating session
[junit4:junit4] 2> 107 T1806 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:40360
[junit4:junit4] 2> 107 T1806 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:40360
[junit4:junit4] 2> 108 T1808 oazsp.FileTxnLog.append Creating new log file: log.1
[junit4:junit4] 2> 224 T1808 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d628bccd50000 with negotiated timeout 10000 for client /127.0.0.1:40360
[junit4:junit4] 2> 225 T1810 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:58818, sessionid = 0x13d628bccd50000, negotiated timeout = 10000
[junit4:junit4] 2> 225 T1811 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6ea5549f name:ZooKeeperConnection Watcher:127.0.0.1:58818 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 225 T1804 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 226 T1804 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 229 T1809 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d628bccd50000
[junit4:junit4] 2> 230 T1811 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 231 T1806 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:40360 which had sessionid 0x13d628bccd50000
[junit4:junit4] 2> 230 T1804 oaz.ZooKeeper.close Session: 0x13d628bccd50000 closed
[junit4:junit4] 2> 231 T1804 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:58818/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@6062a73b
[junit4:junit4] 2> 232 T1804 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 232 T1812 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:58818. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 232 T1812 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:58818, initiating session
[junit4:junit4] 2> 233 T1806 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:40361
[junit4:junit4] 2> 233 T1806 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:40361
[junit4:junit4] 2> 233 T1808 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d628bccd50001 with negotiated timeout 10000 for client /127.0.0.1:40361
[junit4:junit4] 2> 233 T1812 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:58818, sessionid = 0x13d628bccd50001, negotiated timeout = 10000
[junit4:junit4] 2> 234 T1813 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6062a73b name:ZooKeeperConnection Watcher:127.0.0.1:58818/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 234 T1804 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 234 T1804 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 237 T1804 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 240 T1804 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 241 T1804 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 245 T1804 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 246 T1804 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 249 T1804 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 250 T1804 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 252 T1804 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 253 T1804 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 255 T1804 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 255 T1804 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 257 T1804 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 257 T1804 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 259 T1804 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 260 T1804 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 262 T1804 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 262 T1804 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 264 T1804 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 265 T1804 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 267 T1804 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 267 T1804 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 269 T1809 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d628bccd50001
[junit4:junit4] 2> 270 T1806 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:40361 which had sessionid 0x13d628bccd50001
[junit4:junit4] 2> 269 T1804 oaz.ZooKeeper.close Session: 0x13d628bccd50001 closed
[junit4:junit4] 2> 270 T1813 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 325 T1804 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4] 2> 356 T1804 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:57954
[junit4:junit4] 2> 357 T1804 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 357 T1804 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 358 T1804 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.UnloadDistributedZkTest-controljetty-1363157962203
[junit4:junit4] 2> 358 T1804 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.UnloadDistributedZkTest-controljetty-1363157962203/solr.xml
[junit4:junit4] 2> 358 T1804 oasc.CoreContainer.<init> New CoreContainer 1223258584
[junit4:junit4] 2> 359 T1804 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.UnloadDistributedZkTest-controljetty-1363157962203/'
[junit4:junit4] 2> 359 T1804 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.UnloadDistributedZkTest-controljetty-1363157962203/'
[junit4:junit4] 2> 374 T1804 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 375 T1804 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 375 T1804 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 376 T1804 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 376 T1804 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 376 T1804 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 376 T1804 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 377 T1804 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 377 T1804 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 377 T1804 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 382 T1804 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4] 2> 391 T1804 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:58818/solr
[junit4:junit4] 2> 391 T1804 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 391 T1804 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:58818 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@1d93d882
[junit4:junit4] 2> 392 T1804 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 392 T1823 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:58818. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 393 T1823 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:58818, initiating session
[junit4:junit4] 2> 393 T1806 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:40362
[junit4:junit4] 2> 393 T1806 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:40362
[junit4:junit4] 2> 394 T1808 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d628bccd50002 with negotiated timeout 20000 for client /127.0.0.1:40362
[junit4:junit4] 2> 394 T1823 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:58818, sessionid = 0x13d628bccd50002, negotiated timeout = 20000
[junit4:junit4] 2> 394 T1824 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d93d882 name:ZooKeeperConnection Watcher:127.0.0.1:58818 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 394 T1804 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 396 T1809 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d628bccd50002
[junit4:junit4] 2> 396 T1824 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 396 T1806 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:40362 which had sessionid 0x13d628bccd50002
[junit4:junit4] 2> 396 T1804 oaz.ZooKeeper.close Session: 0x13d628bccd50002 closed
[junit4:junit4] 2> 397 T1804 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 404 T1804 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:58818/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@64fd2006
[junit4:junit4] 2> 405 T1804 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 405 T1825 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:58818. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 406 T1825 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:58818, initiating session
[junit4:junit4] 2> 406 T1806 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:40363
[junit4:junit4] 2> 406 T1806 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:40363
[junit4:junit4] 2> 407 T1808 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d628bccd50003 with negotiated timeout 20000 for client /127.0.0.1:40363
[junit4:junit4] 2> 407 T1825 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:58818, sessionid = 0x13d628bccd50003, negotiated timeout = 20000
[junit4:junit4] 2> 407 T1826 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@64fd2006 name:ZooKeeperConnection Watcher:127.0.0.1:58818/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 407 T1804 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 409 T1809 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d628bccd50003 type:create cxid:0x2 zxid:0x1a txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 410 T1804 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 411 T1804 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57954_fy%2Ftd
[junit4:junit4] 2> 411 T1809 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d628bccd50003 type:delete cxid:0x8 zxid:0x1c txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:57954_fy%2Ftd Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:57954_fy%2Ftd
[junit4:junit4] 2> 412 T1804 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:57954_fy%2Ftd
[junit4:junit4] 2> 414 T1804 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 417 T1809 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d628bccd50003 type:delete cxid:0x16 zxid:0x21 txntype:-1 reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4] 2> 418 T1804 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 419 T1804 oasc.Overseer.start Overseer (id=89335920193765379-127.0.0.1:57954_fy%2Ftd-n_0000000000) starting
[junit4:junit4] 2> 420 T1809 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d628bccd50003 type:create cxid:0x1b zxid:0x23 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 420 T1809 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d628bccd50003 type:create cxid:0x1c zxid:0x24 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 421 T1809 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d628bccd50003 type:create cxid:0x1d zxid:0x25 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 422 T1809 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d628bccd50003 type:create cxid:0x1e zxid:0x26 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 422 T1828 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 423 T1804 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 425 T1804 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 426 T1804 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 427 T1827 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 429 T1804 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 429 T1804 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 429 T1804 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 432 T1804 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 433 T1804 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:58818/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@20c45bfb
[junit4:junit4] 2> 433 T1804 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 434 T1830 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:58818. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 434 T1830 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:58818, initiating session
[junit4:junit4] 2> 434 T1806 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:40364
[junit4:junit4] 2> 434 T1806 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:40364
[junit4:junit4] 2> 435 T1808 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d628bccd50004 with negotiated timeout 10000 for client /127.0.0.1:40364
[junit4:junit4] 2> 435 T1830 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:58818, sessionid = 0x13d628bccd50004, negotiated timeout = 10000
[junit4:junit4] 2> 436 T1831 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@20c45bfb name:ZooKeeperConnection Watcher:127.0.0.1:58818/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 436 T1804 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 437 T1804 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 439 T1804 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 499 T1804 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4] 2> 501 T1804 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:35631
[junit4:junit4] 2> 501 T1804 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 501 T1804 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 502 T1804 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty1-1363157962372
[junit4:junit4] 2> 502 T1804 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty1-1363157962372/solr.xml
[junit4:junit4] 2> 502 T1804 oasc.CoreContainer.<init> New CoreContainer 27059960
[junit4:junit4] 2> 503 T1804 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty1-1363157962372/'
[junit4:junit4] 2> 503 T1804 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty1-1363157962372/'
[junit4:junit4] 2> 519 T1804 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 519 T1804 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 519 T1804 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 520 T1804 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 520 T1804 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 520 T1804 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 520 T1804 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 521 T1804 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 521 T1804 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 521 T1804 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 526 T1804 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4] 2> 534 T1804 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:58818/solr
[junit4:junit4] 2> 535 T1804 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 535 T1804 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:58818 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@2bd65b2f
[junit4:junit4] 2> 536 T1804 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 536 T1841 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:58818. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 537 T1841 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:58818, initiating session
[junit4:junit4] 2> 537 T1806 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:40365
[junit4:junit4] 2> 537 T1806 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:40365
[junit4:junit4] 2> 538 T1808 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d628bccd50005 with negotiated timeout 20000 for client /127.0.0.1:40365
[junit4:junit4] 2> 538 T1841 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:58818, sessionid = 0x13d628bccd50005, negotiated timeout = 20000
[junit4:junit4] 2> 538 T1842 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2bd65b2f name:ZooKeeperConnection Watcher:127.0.0.1:58818 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 538 T1804 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 539 T1809 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d628bccd50005
[junit4:junit4] 2> 540 T1842 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 540 T1806 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:40365 which had sessionid 0x13d628bccd50005
[junit4:junit4] 2> 540 T1804 oaz.ZooKeeper.close Session: 0x13d628bccd50005 closed
[junit4:junit4] 2> 540 T1804 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 543 T1804 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:58818/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@3b8c2c69
[junit4:junit4] 2> 543 T1804 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 544 T1843 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:58818. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 544 T1843 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:58818, initiating session
[junit4:junit4] 2> 544 T1806 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:40366
[junit4:junit4] 2> 544 T1806 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:40366
[junit4:junit4] 2> 545 T1808 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d628bccd50006 with negotiated timeout 20000 for client /127.0.0.1:40366
[junit4:junit4] 2> 545 T1843 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:58818, sessionid = 0x13d628bccd50006, negotiated timeout = 20000
[junit4:junit4] 2> 545 T1844 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3b8c2c69 name:ZooKeeperConnection Watcher:127.0.0.1:58818/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 546 T1804 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 547 T1809 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d628bccd50006 type:create cxid:0x1 zxid:0x2e txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 547 T1809 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d628bccd50006 type:create cxid:0x2 zxid:0x2f txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 549 T1804 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 1551 T1804 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35631_fy%2Ftd
[junit4:junit4] 2> 1552 T1809 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d628bccd50006 type:delete cxid:0xb zxid:0x30 txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:35631_fy%2Ftd Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:35631_fy%2Ftd
[junit4:junit4] 2> 1552 T1804 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:35631_fy%2Ftd
[junit4:junit4] 2> 1555 T1826 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 1555 T1831 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 1555 T1844 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 1559 T1804 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 1559 T1804 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 1559 T1804 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 1623 T1804 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4] 2> 1625 T1804 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:33530
[junit4:junit4] 2> 1625 T1804 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 1626 T1804 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 1626 T1804 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty2-1363157963495
[junit4:junit4] 2> 1626 T1804 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty2-1363157963495/solr.xml
[junit4:junit4] 2> 1626 T1804 oasc.CoreContainer.<init> New CoreContainer 1056016891
[junit4:junit4] 2> 1627 T1804 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty2-1363157963495/'
[junit4:junit4] 2> 1627 T1804 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty2-1363157963495/'
[junit4:junit4] 2> 1644 T1804 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 1645 T1804 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 1645 T1804 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 1645 T1804 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 1645 T1804 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 1646 T1804 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 1646 T1804 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 1646 T1804 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 1647 T1804 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 1647 T1804 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 1659 T1804 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4] 2> 1667 T1804 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:58818/solr
[junit4:junit4] 2> 1668 T1804 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 1668 T1804 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:58818 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@6f52e04
[junit4:junit4] 2> 1669 T1804 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 1669 T1855 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:58818. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 1670 T1855 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:58818, initiating session
[junit4:junit4] 2> 1670 T1806 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:40367
[junit4:junit4] 2> 1670 T1806 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:40367
[junit4:junit4] 2> 1671 T1808 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d628bccd50007 with negotiated timeout 20000 for client /127.0.0.1:40367
[junit4:junit4] 2> 1671 T1855 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:58818, sessionid = 0x13d628bccd50007, negotiated timeout = 20000
[junit4:junit4] 2> 1671 T1856 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6f52e04 name:ZooKeeperConnection Watcher:127.0.0.1:58818 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 1672 T1804 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 1673 T1809 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d628bccd50007
[junit4:junit4] 2> 1673 T1856 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 1673 T1806 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:40367 which had sessionid 0x13d628bccd50007
[junit4:junit4] 2> 1673 T1804 oaz.ZooKeeper.close Session: 0x13d628bccd50007 closed
[junit4:junit4] 2> 1674 T1804 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 1676 T1804 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:58818/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@49aafca1
[junit4:junit4] 2> 1677 T1804 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 1677 T1857 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:58818. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 1678 T1857 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:58818, initiating session
[junit4:junit4] 2> 1678 T1806 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:40368
[junit4:junit4] 2> 1678 T1806 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:40368
[junit4:junit4] 2> 1679 T1808 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d628bccd50008 with negotiated timeout 20000 for client /127.0.0.1:40368
[junit4:junit4] 2> 1679 T1857 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:58818, sessionid = 0x13d628bccd50008, negotiated timeout = 20000
[junit4:junit4] 2> 1679 T1858 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@49aafca1 name:ZooKeeperConnection Watcher:127.0.0.1:58818/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 1679 T1804 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 1680 T1809 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d628bccd50008 type:create cxid:0x1 zxid:0x36 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 1681 T1809 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d628bccd50008 type:create cxid:0x2 zxid:0x37 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 1682 T1804 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 2684 T1804 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33530_fy%2Ftd
[junit4:junit4] 2> 2685 T1809 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d628bccd50008 type:delete cxid:0xb zxid:0x38 txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:33530_fy%2Ftd Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:33530_fy%2Ftd
[junit4:junit4] 2> 2686 T1804 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:33530_fy%2Ftd
[junit4:junit4] 2> 2689 T1826 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 2689 T1831 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 2689 T1858 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 2689 T1844 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 2693 T1804 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 2693 T1804 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 2693 T1804 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 2753 T1804 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4] 2> 2755 T1804 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:35038
[junit4:junit4] 2> 2756 T1804 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 2756 T1804 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 2756 T1804 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty3-1363157964628
[junit4:junit4] 2> 2757 T1804 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty3-1363157964628/solr.xml
[junit4:junit4] 2> 2757 T1804 oasc.CoreContainer.<init> New CoreContainer 1957048194
[junit4:junit4] 2> 2757 T1804 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty3-1363157964628/'
[junit4:junit4] 2> 2757 T1804 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty3-1363157964628/'
[junit4:junit4] 2> 2773 T1804 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 2774 T1804 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 2774 T1804 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 2774 T1804 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 2775 T1804 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 2775 T1804 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 2775 T1804 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 2775 T1804 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 2776 T1804 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 2776 T1804 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 2781 T1804 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4] 2> 2789 T1804 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:58818/solr
[junit4:junit4] 2> 2790 T1804 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 2790 T1804 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:58818 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@2d983280
[junit4:junit4] 2> 2791 T1804 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 2791 T1869 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:58818. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 2792 T1869 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:58818, initiating session
[junit4:junit4] 2> 2792 T1806 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:40370
[junit4:junit4] 2> 2792 T1806 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:40370
[junit4:junit4] 2> 2793 T1808 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d628bccd50009 with negotiated timeout 20000 for client /127.0.0.1:40370
[junit4:junit4] 2> 2793 T1869 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:58818, sessionid = 0x13d628bccd50009, negotiated timeout = 20000
[junit4:junit4] 2> 2793 T1870 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2d983280 name:ZooKeeperConnection Watcher:127.0.0.1:58818 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 2793 T1804 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 2794 T1809 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d628bccd50009
[junit4:junit4] 2> 2795 T1806 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:40370 which had sessionid 0x13d628bccd50009
[junit4:junit4] 2> 2795 T1870 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 2795 T1804 oaz.ZooKeeper.close Session: 0x13d628bccd50009 closed
[junit4:junit4] 2> 2795 T1804 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 2798 T1804 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:58818/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@72afcec4
[junit4:junit4] 2> 2799 T1804 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 2799 T1871 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:58818. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 2799 T1871 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:58818, initiating session
[junit4:junit4] 2> 2800 T1806 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:40371
[junit4:junit4] 2> 2800 T1806 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:40371
[junit4:junit4] 2> 2801 T1808 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d628bccd5000a with negotiated timeout 20000 for client /127.0.0.1:40371
[junit4:junit4] 2> 2801 T1871 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:58818, sessionid = 0x13d628bccd5000a, negotiated timeout = 20000
[junit4:junit4] 2> 2801 T1872 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@72afcec4 name:ZooKeeperConnection Watcher:127.0.0.1:58818/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 2801 T1804 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 2803 T1809 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d628bccd5000a type:create cxid:0x1 zxid:0x3e txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 2804 T1809 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d628bccd5000a type:create cxid:0x2 zxid:0x3f txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 2806 T1804 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 3809 T1804 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35038_fy%2Ftd
[junit4:junit4] 2> 3810 T1809 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d628bccd5000a type:delete cxid:0xb zxid:0x40 txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:35038_fy%2Ftd Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:35038_fy%2Ftd
[junit4:junit4] 2> 3811 T1804 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:35038_fy%2Ftd
[junit4:junit4] 2> 3813 T1844 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 3814 T1826 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 3814 T1831 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 3814 T1858 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 3814 T1872 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 3818 T1804 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 3818 T1804 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 3818 T1804 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 3885 T1804 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4] 2> 3887 T1804 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:37405
[junit4:junit4] 2> 3888 T1804 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 3888 T1804 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 3889 T1804 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty4-1363157965762
[junit4:junit4] 2> 3889 T1804 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty4-1363157965762/solr.xml
[junit4:junit4] 2> 3889 T1804 oasc.CoreContainer.<init> New CoreContainer 680275124
[junit4:junit4] 2> 3890 T1804 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty4-1363157965762/'
[junit4:junit4] 2> 3890 T1804 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty4-1363157965762/'
[junit4:junit4] 2> 3905 T1804 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 3906 T1804 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 3906 T1804 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 3906 T1804 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 3906 T1804 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 3907 T1804 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 3907 T1804 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 3907 T1804 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 3908 T1804 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 3908 T1804 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 3913 T1804 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4] 2> 3921 T1804 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:58818/solr
[junit4:junit4] 2> 3921 T1804 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 3921 T1804 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:58818 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@345e29c4
[junit4:junit4] 2> 3922 T1804 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 3922 T1883 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:58818. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 3923 T1883 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:58818, initiating session
[junit4:junit4] 2> 3923 T1806 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:40372
[junit4:junit4] 2> 3924 T1806 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:40372
[junit4:junit4] 2> 3924 T1808 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d628bccd5000b with negotiated timeout 20000 for client /127.0.0.1:40372
[junit4:junit4] 2> 3924 T1883 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:58818, sessionid = 0x13d628bccd5000b, negotiated timeout = 20000
[junit4:junit4] 2> 3925 T1884 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@345e29c4 name:ZooKeeperConnection Watcher:127.0.0.1:58818 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 3925 T1804 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 3926 T1809 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d628bccd5000b
[junit4:junit4] 2> 3927 T1806 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:40372 which had sessionid 0x13d628bccd5000b
[junit4:junit4] 2> 3927 T1884 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 3927 T1804 oaz.ZooKeeper.close Session: 0x13d628bccd5000b closed
[junit4:junit4] 2> 3927 T1804 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 3930 T1804 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:58818/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@69134e9f
[junit4:junit4] 2> 3930 T1804 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 3931 T1885 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:58818. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 3931 T1885 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:58818, initiating session
[junit4:junit4] 2> 3931 T1806 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:40373
[junit4:junit4] 2> 3932 T1806 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:40373
[junit4:junit4] 2> 3932 T1808 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d628bccd5000c with negotiated timeout 20000 for client /127.0.0.1:40373
[junit4:junit4] 2> 3932 T1885 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:58818, sessionid = 0x13d628bccd5000c, negotiated timeout = 20000
[junit4:junit4] 2> 3933 T1886 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@69134e9f name:ZooKeeperConnection Watcher:127.0.0.1:58818/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 3933 T1804 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 3934 T1809 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d628bccd5000c type:create cxid:0x1 zxid:0x46 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 3935 T1809 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d628bccd5000c type:create cxid:0x2 zxid:0x47 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 3937 T1804 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 4939 T1804 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37405_fy%2Ftd
[junit4:junit4] 2> 4940 T1809 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d628bccd5000c type:delete cxid:0xb zxid:0x48 txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:37405_fy%2Ftd Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:37405_fy%2Ftd
[junit4:junit4] 2> 4940 T1804 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:37405_fy%2Ftd
[junit4:junit4] 2> 4943 T1872 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 4943 T1844 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 4943 T1826 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 4943 T1831 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 4943 T1886 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 4943 T1858 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 4947 T1804 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 4947 T1804 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 4948 T1804 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 4950 T1804 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 4953 T1838 oasc.CoreContainer.create Creating SolrCore 'unloadcollection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty1-1363157962372/unloadcollection1
[junit4:junit4] 2> 4953 T1838 oasc.ZkController.createCollectionZkNode Check for collection zkNode:unloadcollection
[junit4:junit4] 2> 4954 T1838 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:unloadcollection
[junit4:junit4] 2> 4954 T1838 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4] 2> 4955 T1838 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
[junit4:junit4] 2> 4956 T1838 oascc.SolrZkClient.makePath makePath: /collections/unloadcollection
[junit4:junit4] 2> 4957 T1838 oasc.ZkController.readConfigName Load collection config from:/collections/unloadcollection
[junit4:junit4] 2> 4959 T1838 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty1-1363157962372/unloadcollection1/'
[junit4:junit4] 2> 4980 T1838 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 5021 T1838 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 5022 T1838 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4] 2> 5026 T1838 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4] 2> 5362 T1838 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 5367 T1838 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 5369 T1838 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 5380 T1838 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 5384 T1838 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 5388 T1838 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 5389 T1838 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 5389 T1838 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 5389 T1838 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 5390 T1838 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 5391 T1838 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 5391 T1838 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 5391 T1838 oasc.SolrCore.<init> [unloadcollection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty1-1363157962372/unloadcollection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-UnloadDistributedZkTest-1363157961938/1363157966885unloadcollection1_1n/
[junit4:junit4] 2> 5391 T1838 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@50b4ffce
[junit4:junit4] 2> 5392 T1838 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 5393 T1838 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-UnloadDistributedZkTest-1363157961938/1363157966885unloadcollection1_1n forceNew: false
[junit4:junit4] 2> 5393 T1838 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-UnloadDistributedZkTest-1363157961938/1363157966885unloadcollection1_1n/index/
[junit4:junit4] 2> 5393 T1838 oasc.SolrCore.initIndex WARNING [unloadcollection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-UnloadDistributedZkTest-1363157961938/1363157966885unloadcollection1_1n/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 5393 T1838 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-UnloadDistributedZkTest-1363157961938/1363157966885unloadcollection1_1n/index forceNew: false
[junit4:junit4] 2> 5395 T1838 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5ba44f5c lockFactory=org.apache.lucene.store.NativeFSLockFactory@396cdfd5),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> 5395 T1838 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4] 2> 5397 T1838 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 5397 T1838 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 5398 T1838 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 5398 T1838 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 5399 T1838 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 5399 T1838 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 5400 T1838 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 5400 T1838 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 5401 T1838 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 5404 T1838 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 5407 T1838 oass.SolrIndexSearcher.<init> Opening Searcher@6fd2130f main
[junit4:junit4] 2> 5408 T1838 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-UnloadDistributedZkTest-1363157961938/1363157966885unloadcollection1_1n/tlog
[junit4:junit4] 2> 5408 T1838 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 5409 T1838 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 5412 T1888 oasc.SolrCore.registerSearcher [unloadcollection1] Registered new searcher Searcher@6fd2130f main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 5412 T1838 oasc.ZkController.publish publishing core=unloadcollection1 state=down
[junit4:junit4] 2> 5413 T1809 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d628bccd50006 type:create cxid:0x40 zxid:0x4c txntype:-1 reqpath:n/a Error Path:/solr/overseer/queue Error:KeeperErrorCode = NoNode for /solr/overseer/queue
[junit4:junit4] 2> 6438 T1827 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 6438 T1827 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:35631/fy/td",
[junit4:junit4] 2> "collection":"unloadcollection",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:35631_fy%2Ftd",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "core":"unloadcollection1",
[junit4:junit4] 2> "numShards":"1"}
[junit4:junit4] 2> 6439 T1827 oasc.Overseer$ClusterStateUpdater.createCollection Create collection unloadcollection with numShards 1
[junit4:junit4] 2> 6439 T1827 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4] 2> 6439 T1809 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d628bccd50003 type:create cxid:0x46 zxid:0x4f txntype:-1 reqpath:n/a Error Path:/solr/overseer/queue-work Error:KeeperErrorCode = NoNode for /solr/overseer/queue-work
[junit4:junit4] 2> 6444 T1872 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 6444 T1826 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 6444 T1886 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 6444 T1858 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 6444 T1844 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 6444 T1831 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 7417 T1838 oasc.CoreContainer.registerCore registering core: unloadcollection1
[junit4:junit4] 2> 7418 T1838 oasc.ZkController.register Register replica - core:unloadcollection1 address:http://127.0.0.1:35631/fy/td collection:unloadcollection shard:shard1
[junit4:junit4] 2> 7419 T1838 oascc.SolrZkClient.makePath makePath: /collections/unloadcollection/leader_elect/shard1/election
[junit4:junit4] 2> 7424 T1809 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d628bccd50006 type:delete cxid:0x51 zxid:0x59 txntype:-1 reqpath:n/a Error Path:/solr/collections/unloadcollection/leaders Error:KeeperErrorCode = NoNode for /solr/collections/unloadcollection/leaders
[junit4:junit4] 2> 7424 T1838 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 7425 T1809 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d628bccd50006 type:create cxid:0x52 zxid:0x5a txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 7426 T1838 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 7426 T1838 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 7426 T1838 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:35631/fy/td/unloadcollection1/
[junit4:junit4] 2> 7427 T1838 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 7427 T1838 oasc.SyncStrategy.syncToMe http://127.0.0.1:35631/fy/td/unloadcollection1/ has no replicas
[junit4:junit4] 2> 7427 T1838 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:35631/fy/td/unloadcollection1/
[junit4:junit4] 2> 7427 T1838 oascc.SolrZkClient.makePath makePath: /collections/unloadcollection/leaders/shard1
[junit4:junit4] 2> 7430 T1809 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d628bccd50006 type:create cxid:0x5c zxid:0x5e txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 7948 T1827 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 7957 T1831 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 7957 T1826 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 7957 T1872 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 7957 T1858 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 7957 T1886 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 7957 T1844 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 7983 T1838 oasc.ZkController.register We are http://127.0.0.1:35631/fy/td/unloadcollection1/ and leader is http://127.0.0.1:35631/fy/td/unloadcollection1/
[junit4:junit4] 2> 7983 T1838 oasc.ZkController.register No LogReplay needed for core=unloadcollection1 baseURL=http://127.0.0.1:35631/fy/td
[junit4:junit4] 2> 7984 T1838 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 7984 T1838 oasc.ZkController.publish publishing core=unloadcollection1 state=active
[junit4:junit4] 2> 7992 T1838 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 7993 T1838 oasc.CoreContainer.persistFile Persisting cores config to /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty1-1363157962372/solr.xml
[junit4:junit4] 2> 7997 T1838 oasc.SolrXMLSerializer.persistFile Persisting cores config to /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty1-1363157962372/solr.xml
[junit4:junit4] 2> 7998 T1838 oass.SolrDispatchFilter.handleAdminRequest /admin/cores {collection=unloadcollection&action=CREATE&dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-UnloadDistributedZkTest-1363157961938/1363157966885unloadcollection1_1n&version=2&name=unloadcollection1&wt=javabin&numShards=1} status=0 QTime=3045
[junit4:junit4] 2> 7999 T1804 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 8000 T1804 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:58818/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@7412f01
[junit4:junit4] 2> 8001 T1804 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 8001 T1889 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:58818. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 8001 T1889 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:58818, initiating session
[junit4:junit4] 2> 8001 T1806 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:40375
[junit4:junit4] 2> 8002 T1806 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:40375
[junit4:junit4] 2> 8003 T1808 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d628bccd5000d with negotiated timeout 10000 for client /127.0.0.1:40375
[junit4:junit4] 2> 8003 T1889 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:58818, sessionid = 0x13d628bccd5000d, negotiated timeout = 10000
[junit4:junit4] 2> 8003 T1890 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7412f01 name:ZooKeeperConnection Watcher:127.0.0.1:58818/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 8003 T1804 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 8005 T1804 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 8007 T1804 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 8008 T1804 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 8011 T1854 oasc.CoreContainer.create Creating SolrCore 'unloadcollection2' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty2-1363157963495/unloadcollection2
[junit4:junit4] 2> 8012 T1854 oasc.ZkController.createCollectionZkNode Check for collection zkNode:unloadcollection
[junit4:junit4] 2> 8012 T1854 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 8013 T1854 oasc.ZkController.readConfigName Load collection config from:/collections/unloadcollection
[junit4:junit4] 2> 8014 T1854 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty2-1363157963495/unloadcollection2/'
[junit4:junit4] 2> 8060 T1854 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 8092 T1854 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 8093 T1854 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4] 2> 8096 T1854 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4] 2> 8430 T1854 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 8436 T1854 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 8438 T1854 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 8450 T1854 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 8453 T1854 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 8456 T1854 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 8457 T1854 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 8457 T1854 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 8457 T1854 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 8458 T1854 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 8458 T1854 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 8459 T1854 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 8459 T1854 oasc.SolrCore.<init> [unloadcollection2] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty2-1363157963495/unloadcollection2/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-UnloadDistributedZkTest-1363157961938/1363157969943unloadcollection1_2n/
[junit4:junit4] 2> 8459 T1854 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@50b4ffce
[junit4:junit4] 2> 8460 T1854 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 8460 T1854 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-UnloadDistributedZkTest-1363157961938/1363157969943unloadcollection1_2n forceNew: false
[junit4:junit4] 2> 8460 T1854 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-UnloadDistributedZkTest-1363157961938/1363157969943unloadcollection1_2n/index/
[junit4:junit4] 2> 8461 T1854 oasc.SolrCore.initIndex WARNING [unloadcollection2] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-UnloadDistributedZkTest-1363157961938/1363157969943unloadcollection1_2n/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 8461 T1854 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-UnloadDistributedZkTest-1363157961938/1363157969943unloadcollection1_2n/index forceNew: false
[junit4:junit4] 2> 8462 T1854 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5df92342 lockFactory=org.apache.lucene.store.NativeFSLockFactory@535ab1e9),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> 8463 T1854 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4] 2> 8464 T1854 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 8465 T1854 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 8465 T1854 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 8466 T1854 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 8466 T1854 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 8466 T1854 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 8466 T1854 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 8467 T1854 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 8467 T1854 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 8470 T1854 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 8472 T1854 oass.SolrIndexSearcher.<init> Opening Searcher@1adfbd67 main
[junit4:junit4] 2> 8472 T1854 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-UnloadDistributedZkTest-1363157961938/1363157969943unloadcollection1_2n/tlog
[junit4:junit4] 2> 8473 T1854 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 8473 T1854 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 8476 T1891 oasc.SolrCore.registerSearcher [unloadcollection2] Registered new searcher Searcher@1adfbd67 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 8476 T1854 oasc.ZkController.publish publishing core=unloadcollection2 state=down
[junit4:junit4] 2> 8477 T1854 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 9477 T1827 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 9478 T1827 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:35631/fy/td",
[junit4:junit4] 2> "collection":"unloadcollection",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:35631_fy%2Ftd",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "core":"unloadcollection1",
[junit4:junit4] 2> "numShards":"1"}
[junit4:junit4] 2> 9481 T1827 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:33530/fy/td",
[junit4:junit4] 2> "collection":"unloadcollection",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:33530_fy%2Ftd",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "core":"unloadcollection2",
[junit4:junit4] 2> "numShards":"2"}
[junit4:junit4] 2> 9481 T1827 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 9481 T1827 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4] 2> 9484 T1890 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 9484 T1826 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 9484 T1831 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 9484 T1844 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 9484 T1858 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 9484 T1872 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 9484 T1886 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 10478 T1854 oasc.CoreContainer.registerCore registering core: unloadcollection2
[junit4:junit4] 2> 10478 T1854 oasc.ZkController.register Register replica - core:unloadcollection2 address:http://127.0.0.1:33530/fy/td collection:unloadcollection shard:shard1
[junit4:junit4] 2> 10481 T1854 oasc.ZkController.register We are http://127.0.0.1:33530/fy/td/unloadcollection2/ and leader is http://127.0.0.1:35631/fy/td/unloadcollection1/
[junit4:junit4] 2> 10481 T1854 oasc.ZkController.register No LogReplay needed for core=unloadcollection2 baseURL=http://127.0.0.1:33530/fy/td
[junit4:junit4] 2> 10481 T1854 oasc.ZkController.checkRecovery Core needs to recover:unloadcollection2
[junit4:junit4] 2> 10482 T1854 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C97 name=unloadcollection2 org.apache.solr.core.SolrCore@626a95f7 url=http://127.0.0.1:33530/fy/td/unloadcollection2 node=127.0.0.1:33530_fy%2Ftd C97_STATE=coll:unloadcollection core:unloadcollection2 props:{state=down, base_url=http://127.0.0.1:33530/fy/td, collection=unloadcollection, node_name=127.0.0.1:33530_fy%2Ftd, core=unloadcollection2}
[junit4:junit4] 2> 10482 T1892 C97 P33530 oasc.RecoveryStrategy.run Starting recovery process. core=unloadcollection2 recoveringAfterStartup=true
[junit4:junit4] 2> 10483 T1892 C97 P33530 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 10483 T1892 C97 P33530 oasc.ZkController.publish publishing core=unloadcollection2 state=recovering
[junit4:junit4] 2> 10483 T1854 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 10483 T1892 C97 P33530 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 10484 T1854 oasc.CoreContainer.persistFile Persisting cores config to /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty2-1363157963495/solr.xml
[junit4:junit4] 2> 10484 T1892 C97 P33530 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 10490 T1854 oasc.SolrXMLSerializer.persistFile Persisting cores config to /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty2-1363157963495/solr.xml
[junit4:junit4] 2> 10492 T1854 oass.SolrDispatchFilter.handleAdminRequest /admin/cores {collection=unloadcollection&action=CREATE&dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-UnloadDistributedZkTest-1363157961938/1363157969943unloadcollection1_2n&version=2&name=unloadcollection2&wt=javabin} status=0 QTime=2481
[junit4:junit4] 2> 10493 T1804 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 10494 T1804 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: unloadcollection failOnTimeout:true timeout (sec):230
[junit4:junit4] 2> 10495 T1804 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 10990 T1827 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 10991 T1827 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:33530/fy/td",
[junit4:junit4] 2> "collection":"unloadcollection",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:33530_fy%2Ftd",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "core":"unloadcollection2",
[junit4:junit4] 2> "numShards":"2"}
[junit4:junit4] 2> 11014 T1886 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 11014 T1844 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 11014 T1872 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 11014 T1826 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 11014 T1831 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 11014 T1890 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 11014 T1858 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 11490 T1838 oass.SolrDispatchFilter.handleAdminRequest /admin/cores {state=recovering&action=PREPRECOVERY&version=2&coreNodeName=127.0.0.1:33530_fy%252Ftd_unloadcollection2&wt=javabin&checkLive=true&onlyIfLeader=true&nodeName=127.0.0.1:33530_fy%252Ftd&core=unloadcollection1} status=0 QTime=1001
[junit4:junit4] 2> 11496 T1804 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 12498 T1804 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> C97_STATE=coll:unloadcollection core:unloadcollection2 props:{state=recovering, base_url=http://127.0.0.1:33530/fy/td, collection=unloadcollection, node_name=127.0.0.1:33530_fy%2Ftd, shard=shard1, core=unloadcollection2}
[junit4:junit4] 2> 13490 T1892 C97 P33530 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:35631/fy/td/unloadcollection1/ core=unloadcollection2 - recoveringAfterStartup=true
[junit4:junit4] 2> 13491 T1892 C97 P33530 oasu.PeerSync.sync PeerSync: core=unloadcollection2 url=http://127.0.0.1:33530/fy/td START replicas=[http://127.0.0.1:35631/fy/td/unloadcollection1/] nUpdates=100
[junit4:junit4] 2> 13491 T1892 C97 P33530 oasu.PeerSync.sync WARNING no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 13491 T1892 C97 P33530 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=unloadcollection2
[junit4:junit4] 2> 13492 T1892 C97 P33530 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=unloadcollection2
[junit4:junit4] 2> 13492 T1892 C97 P33530 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=unloadcollection2
[junit4:junit4] 2> 13492 T1892 C97 P33530 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 13492 T1892 C97 P33530 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:35631/fy/td/unloadcollection1/. core=unloadcollection2
[junit4:junit4] 2> 13492 T1892 C97 P33530 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C98 name=unloadcollection1 org.apache.solr.core.SolrCore@1c62774c url=http://127.0.0.1:35631/fy/td/unloadcollection1 node=127.0.0.1:35631_fy%2Ftd C98_STATE=coll:unloadcollection core:unloadcollection1 props:{state=active, base_url=http://127.0.0.1:35631/fy/td, collection=unloadcollection, node_name=127.0.0.1:35631_fy%2Ftd, shard=shard1, core=unloadcollection1, leader=true}
[junit4:junit4] 2> 13495 T1838 C98 P35631 REQ /get {version=2&wt=javabin&getVersions=100&distrib=false&qt=/get} status=0 QTime=0
[junit4:junit4] 2> 13497 T1839 C98 P35631 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 13499 T1839 C98 P35631 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5ba44f5c lockFactory=org.apache.lucene.store.NativeFSLockFactory@396cdfd5),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> 13499 T1839 C98 P35631 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4] 2> 13500 T1804 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 13500 T1839 C98 P35631 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5ba44f5c lockFactory=org.apache.lucene.store.NativeFSLockFactory@396cdfd5),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5ba44f5c lockFactory=org.apache.lucene.store.NativeFSLockFactory@396cdfd5),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4] 2> 13500 T1839 C98 P35631 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[segments_2]
[junit4:junit4] 2> 13500 T1839 C98 P35631 oass.SolrIndexSearcher.<init> Opening Searcher@23e8c8 realtime
[junit4:junit4] 2> 13501 T1839 C98 P35631 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 13501 T1839 C98 P35631 /update {openSearcher=false&commit_end_point=true&version=2&waitSearcher=true&wt=javabin&commit=true&softCommit=false} {commit=} 0 4
[junit4:junit4] 2> 13502 T1892 C97 P33530 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 13502 T1892 C97 P33530 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 13544 T1837 C98 P35631 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 13544 T1837 C98 P35631 REQ /replication {version=2&wt=javabin&command=indexversion&qt=/replication} status=0 QTime=41
[junit4:junit4] 2> 13545 T1892 C97 P33530 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 13545 T1892 C97 P33530 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 13545 T1892 C97 P33530 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 13547 T1838 C98 P35631 REQ /replication {version=2&generation=2&wt=javabin&command=filelist&qt=/replication} status=0 QTime=0
[junit4:junit4] 2> 13547 T1892 C97 P33530 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 13548 T1892 C97 P33530 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-UnloadDistributedZkTest-1363157961938/1363157969943unloadcollection1_2n/index.20130313135935480 forceNew: false
[junit4:junit4] 2> 13548 T1892 C97 P33530 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@44c69480 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2cb18f19) fullCopy=false
[junit4:junit4] 2> 13550 T1839 C98 P35631 REQ /replication {checksum=true&generation=2&wt=filestream&file=segments_2&command=filecontent&qt=/replication} status=0 QTime=0
[junit4:junit4] 2> 13551 T1892 C97 P33530 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 13552 T1892 C97 P33530 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 13552 T1892 C97 P33530 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=unloadcollection2
[junit4:junit4] 2> 13553 T1892 C97 P33530 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5df92342 lockFactory=org.apache.lucene.store.NativeFSLockFactory@535ab1e9),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5df92342 lockFactory=org.apache.lucene.store.NativeFSLockFactory@535ab1e9),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4] 2> 13553 T1892 C97 P33530 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[segments_2]
[junit4:junit4] 2> 13554 T1892 C97 P33530 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 13554 T1892 C97 P33530 oass.SolrIndexSearcher.<init> Opening Searcher@57676c6 main
[junit4:junit4] 2> 13555 T1891 oasc.SolrCore.registerSearcher [unloadcollection2] Registered new searcher Searcher@57676c6 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 13555 T1892 C97 P33530 oash.SnapPuller.fetchLatestIndex removing temporary index download directory files MockDirWrapper(org.apache.lucene.store.RAMDirectory@44c69480 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2cb18f19)
[junit4:junit4] 2> 13555 T1892 C97 P33530 oasc.RecoveryStrategy.replay No replay needed. core=unloadcollection2
[junit4:junit4] 2> 13555 T1892 C97 P33530 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=unloadcollection2
[junit4:junit4] 2> 13555 T1892 C97 P33530 oasc.ZkController.publish publishing core=unloadcollection2 state=active
[junit4:junit4] 2> 13555 T1892 C97 P33530 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 13557 T1892 C97 P33530 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=unloadcollection2
[junit4:junit4] 2> 14021 T1827 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 14022 T1827 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:33530/fy/td",
[junit4:junit4] 2> "collection":"unloadcollection",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:33530_fy%2Ftd",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "core":"unloadcollection2",
[junit4:junit4] 2> "numShards":"2"}
[junit4:junit4] 2> 14025 T1890 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 14025 T1886 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 14025 T1844 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 14025 T1872 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 14025 T1831 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 14025 T1858 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 14025 T1826 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 14501 T1804 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 14502 T1804 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: unloadcollection
[junit4:junit4] 2> 14503 T1804 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 14506 T1868 oasc.CoreContainer.create Creating SolrCore 'unloadcollection3' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty3-1363157964628/unloadcollection3
[junit4:junit4] 2> 14506 T1868 oasc.ZkController.createCollectionZkNode Check for collection zkNode:unloadcollection
[junit4:junit4] 2> 14507 T1868 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 14507 T1868 oasc.ZkController.readConfigName Load collection config from:/collections/unloadcollection
[junit4:junit4] 2> 14509 T1868 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty3-1363157964628/unloadcollection3/'
[junit4:junit4] 2> 14534 T1868 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 14565 T1868 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 14566 T1868 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4] 2> 14570 T1868 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4] 2> 14874 T1868 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 14879 T1868 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 14881 T1868 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 14894 T1868 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 14897 T1868 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 14900 T1868 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 14901 T1868 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 14901 T1868 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 14901 T1868 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 14902 T1868 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 14903 T1868 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 14903 T1868 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 14903 T1868 oasc.SolrCore.<init> [unloadcollection3] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty3-1363157964628/unloadcollection3/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-UnloadDistributedZkTest-1363157961938/1363157976438unloadcollection_3n/
[junit4:junit4] 2> 14903 T1868 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@50b4ffce
[junit4:junit4] 2> 14904 T1868 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 14905 T1868 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-UnloadDistributedZkTest-1363157961938/1363157976438unloadcollection_3n forceNew: false
[junit4:junit4] 2> 14905 T1868 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-UnloadDistributedZkTest-1363157961938/1363157976438unloadcollection_3n/index/
[junit4:junit4] 2> 14905 T1868 oasc.SolrCore.initIndex WARNING [unloadcollection3] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-UnloadDistributedZkTest-1363157961938/1363157976438unloadcollection_3n/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 14905 T1868 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-UnloadDistributedZkTest-1363157961938/1363157976438unloadcollection_3n/index forceNew: false
[junit4:junit4] 2> 14907 T1868 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@651e2ddd lockFactory=org.apache.lucene.store.NativeFSLockFactory@75246b27),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> 14907 T1868 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4] 2> 14909 T1868 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 14909 T1868 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 14910 T1868 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 14910 T1868 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 14911 T1868 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 14911 T1868 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 14911 T1868 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 14912 T1868 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 14912 T1868 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 14914 T1868 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 14917 T1868 oass.SolrIndexSearcher.<init> Opening Searcher@5b924486 main
[junit4:junit4] 2> 14917 T1868 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-UnloadDistributedZkTest-1363157961938/1363157976438unloadcollection_3n/tlog
[junit4:junit4] 2> 14917 T1868 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 14918 T1868 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 14921 T1895 oasc.SolrCore.registerSearcher [unloadcollection3] Registered new searcher Searcher@5b924486 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 14922 T1868 oasc.ZkController.publish publishing core=unloadcollection3 state=down
[junit4:junit4] 2> 14923 T1868 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 15529 T1827 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 15530 T1827 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:35038/fy/td",
[junit4:junit4] 2> "collection":"unloadcollection",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:35038_fy%2Ftd",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "core":"unloadcollection3",
[junit4:junit4] 2> "numShards":"2"}
[junit4:junit4] 2> 15530 T1827 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 15530 T1827 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4] 2> 16724 T1808 oazsp.FileTxnLog.commit WARNING fsync-ing the write ahead log in SyncThread:0 took 1190ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
[junit4:junit4] 2> 16725 T1826 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 16725 T1831 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 16725 T1858 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 16725 T1844 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 16725 T1890 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 16725 T1886 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 16725 T1872 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 16925 T1868 oasc.CoreContainer.registerCore registering core: unloadcollection3
[junit4:junit4] 2> 16925 T1868 oasc.ZkController.register Register replica - core:unloadcollection3 address:http://127.0.0.1:35038/fy/td collection:unloadcollection shard:shard1
[junit4:junit4] 2> 16928 T1868 oasc.ZkController.register We are http://127.0.0.1:35038/fy/td/unloadcollection3/ and leader is http://127.0.0.1:35631/fy/td/unloadcollection1/
[junit4:junit4] 2> 16929 T1868 oasc.ZkController.register No LogReplay needed for core=unloadcollection3 baseURL=http://127.0.0.1:35038/fy/td
[junit4:junit4] 2> 16929 T1868 oasc.ZkController.checkRecovery Core needs to recover:unloadcollection3
[junit4:junit4] 2> 16929 T1868 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C99 name=unloadcollection3 org.apache.solr.core.SolrCore@7de19894 url=http://127.0.0.1:35038/fy/td/unloadcollection3 node=127.0.0.1:35038_fy%2Ftd C99_STATE=coll:unloadcollection core:unloadcollection3 props:{state=down, base_url=http://127.0.0.1:35038/fy/td, collection=unloadcollection, node_name=127.0.0.1:35038_fy%2Ftd, core=unloadcollection3}
[junit4:junit4] 2> 16930 T1896 C99 P35038 oasc.RecoveryStrategy.run Starting recovery process. core=unloadcollection3 recoveringAfterStartup=true
[junit4:junit4] 2> 16931 T1896 C99 P35038 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 16931 T1896 C99 P35038 oasc.ZkController.publish publishing core=unloadcollection3 state=recovering
[junit4:junit4] 2> 16931 T1868 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 16931 T1896 C99 P35038 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 16932 T1868 oasc.CoreContainer.persistFile Persisting cores config to /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty3-1363157964628/solr.xml
[junit4:junit4] 2> 16932 T1896 C99 P35038 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 16938 T1868 oasc.SolrXMLSerializer.persistFile Persisting cores config to /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty3-1363157964628/solr.xml
[junit4:junit4] 2> 16939 T1868 oass.SolrDispatchFilter.handleAdminRequest /admin/cores {collection=unloadcollection&action=CREATE&dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-UnloadDistributedZkTest-1363157961938/1363157976438unloadcollection_3n&version=2&name=unloadcollection3&wt=javabin} status=0 QTime=2433
[junit4:junit4] 2> 16939 T1804 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: unloadcollection failOnTimeout:true timeout (sec):230
[junit4:junit4] 2> 16940 T1804 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 17942 T1804 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 18230 T1827 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 18230 T1827 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:35038/fy/td",
[junit4:junit4] 2> "collection":"unloadcollection",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:35038_fy%2Ftd",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "core":"unloadcollection3",
[junit4:junit4] 2> "numShards":"2"}
[junit4:junit4] 2> 18247 T1872 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 18247 T1826 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 18247 T1858 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 18248 T1890 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 18248 T1886 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 18247 T1844 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 18247 T1831 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 18936 T1839 oass.SolrDispatchFilter.handleAdminRequest /admin/cores {state=recovering&action=PREPRECOVERY&version=2&coreNodeName=127.0.0.1:35038_fy%252Ftd_unloadcollection3&wt=javabin&checkLive=true&onlyIfLeader=true&nodeName=127.0.0.1:35038_fy%252Ftd&core=unloadcollection1} status=0 QTime=2000
[junit4:junit4] 2> 18943 T1804 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 19945 T1804 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> C99_STATE=coll:unloadcollection core:unloadcollection3 props:{state=recovering, base_url=http://127.0.0.1:35038/fy/td, collection=unloadcollection, node_name=127.0.0.1:35038_fy%2Ftd, shard=shard1, core=unloadcollection3}
[junit4:junit4] 2> 20937 T1896 C99 P35038 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:35631/fy/td/unloadcollection1/ core=unloadcollection3 - recoveringAfterStartup=true
[junit4:junit4] 2> 20938 T1896 C99 P35038 oasu.PeerSync.sync PeerSync: core=unloadcollection3 url=http://127.0.0.1:35038/fy/td START replicas=[http://127.0.0.1:35631/fy/td/unloadcollection1/] nUpdates=100
[junit4:junit4] 2> 20938 T1896 C99 P35038 oasu.PeerSync.sync WARNING no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 20938 T1896 C99 P35038 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=unloadcollection3
[junit4:junit4] 2> 20938 T1896 C99 P35038 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=unloadcollection3
[junit4:junit4] 2> 20938 T1896 C99 P35038 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=unloadcollection3
[junit4:junit4] 2> 20938 T1896 C99 P35038 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 20939 T1896 C99 P35038 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:35631/fy/td/unloadcollection1/. core=unloadcollection3
[junit4:junit4] 2> 20939 T1896 C99 P35038 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 20940 T1837 C98 P35631 REQ /get {version=2&wt=javabin&getVersions=100&distrib=false&qt=/get} status=0 QTime=0
[junit4:junit4] 2> 20943 T1840 C98 P35631 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 20944 T1840 C98 P35631 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5ba44f5c lockFactory=org.apache.lucene.store.NativeFSLockFactory@396cdfd5),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5ba44f5c lockFactory=org.apache.lucene.store.NativeFSLockFactory@396cdfd5),segFN=segments_3,generation=3,filenames=[segments_3]
[junit4:junit4] 2> 20944 T1840 C98 P35631 oasc.SolrDeletionPolicy.updateCommits newest commit = 3[segments_3]
[junit4:junit4] 2> 20944 T1840 C98 P35631 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 20945 T1840 C98 P35631 /update {openSearcher=false&commit_end_point=true&version=2&waitSearcher=true&wt=javabin&commit=true&softCommit=false} {commit=} 0 3
[junit4:junit4] 2> 20945 T1896 C99 P35038 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 20945 T1896 C99 P35038 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 20947 T1804 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 20947 T1838 C98 P35631 REQ /replication {version=2&wt=javabin&command=indexversion&qt=/replication} status=0 QTime=0
[junit4:junit4] 2> 20948 T1896 C99 P35038 oash.SnapPuller.fetchLatestIndex Master's generation: 3
[junit4:junit4] 2> 20948 T1896 C99 P35038 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 20948 T1896 C99 P35038 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 20950 T1837 C98 P35631 REQ /replication {version=2&generation=3&wt=javabin&command=filelist&qt=/replication} status=0 QTime=0
[junit4:junit4] 2> 20951 T1896 C99 P35038 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 20952 T1896 C99 P35038 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-UnloadDistributedZkTest-1363157961938/1363157976438unloadcollection_3n/index.20130313135942884 forceNew: false
[junit4:junit4] 2> 20952 T1896 C99 P35038 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@720ec61f lockFactory=org.apache.lucene.store.NativeFSLockFactory@5f03d194) fullCopy=false
[junit4:junit4] 2> 20954 T1840 C98 P35631 REQ /replication {checksum=true&generation=3&wt=filestream&file=segments_3&command=filecontent&qt=/replication} status=0 QTime=0
[junit4:junit4] 2> 20955 T1896 C99 P35038 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 20956 T1896 C99 P35038 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 20956 T1896 C99 P35038 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=unloadcollection3
[junit4:junit4] 2> 20957 T1896 C99 P35038 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@651e2ddd lockFactory=org.apache.lucene.store.NativeFSLockFactory@75246b27),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@651e2ddd lockFactory=org.apache.lucene.store.NativeFSLockFactory@75246b27),segFN=segments_3,generation=3,filenames=[segments_3]
[junit4:junit4] 2> 20957 T1896 C99 P35038 oasc.SolrDeletionPolicy.updateCommits newest commit = 3[segments_3]
[junit4:junit4] 2> 20957 T1896 C99 P35038 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 20957 T1896 C99 P35038 oass.SolrIndexSearcher.<init> Opening Searcher@5cdff44f main
[junit4:junit4] 2> 20958 T1895 oasc.SolrCore.registerSearcher [unloadcollection3] Registered new searcher Searcher@5cdff44f main{StandardDirectoryReader(segments_3:1:nrt)}
[junit4:junit4] 2> 20959 T1896 C99 P35038 oash.SnapPuller.fetchLatestIndex removing temporary index download directory files MockDirWrapper(org.apache.lucene.store.RAMDirectory@720ec61f lockFactory=org.apache.lucene.store.NativeFSLockFactory@5f03d194)
[junit4:junit4] 2> 20959 T1896 C99 P35038 oasc.RecoveryStrategy.replay No replay needed. core=unloadcollection3
[junit4:junit4] 2> 20959 T1896 C99 P35038 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=unloadcollection3
[junit4:junit4] 2> 20959 T1896 C99 P35038 oasc.ZkController.publish publishing core=unloadcollection3 state=active
[junit4:junit4] 2> 20959 T1896 C99 P35038 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 20960 T1896 C99 P35038 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=unloadcollection3
[junit4:junit4] 2> 21253 T1827 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 21254 T1827 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:35038/fy/td",
[junit4:junit4] 2> "collection":"unloadcollection",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:35038_fy%2Ftd",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "core":"unloadcollection3",
[junit4:junit4] 2> "numShards":"2"}
[junit4:junit4] 2> 21274 T1858 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 21274 T1872 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 21274 T1826 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 21274 T1844 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 21274 T1886 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 21274 T1831 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 21274 T1890 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 21949 T1804 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 21950 T1804 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: unloadcollection
[junit4:junit4] 2> 21950 T1804 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> C99_STATE=coll:unloadcollection core:unloadcollection3 props:{state=active, base_url=http://127.0.0.1:35038/fy/td, collection=unloadcollection, node_name=127.0.0.1:35038_fy%2Ftd, shard=shard1, core=unloadcollection3}
[junit4:junit4] 2> 21969 T1865 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[20 (1429374746123829248)]} 0 1
[junit4:junit4] 2> C97_STATE=coll:unloadcollection core:unloadcollection2 props:{state=active, base_url=http://127.0.0.1:33530/fy/td, collection=unloadcollection, node_name=127.0.0.1:33530_fy%2Ftd, shard=shard1, core=unloadcollection2}
[junit4:junit4] 2> 21973 T1854 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[20 (1429374746123829248)]} 0 1
[junit4:junit4] 2> 21974 T1837 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[20 (1429374746123829248)]} 0 10
[junit4:junit4] 2> 21975 T1866 C99 P35038 /update {version=2&wt=javabin} {add=[20]} 0 13
[junit4:junit4] 2> 21983 T1853 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[21 (1429374746138509312)]} 0 1
[junit4:junit4] 2> 21983 T1867 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[21 (1429374746138509312)]} 0 1
[junit4:junit4] 2> 21984 T1840 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[21 (1429374746138509312)]} 0 5
[junit4:junit4] 2> 21984 T1868 C99 P35038 /update {version=2&wt=javabin} {add=[21]} 0 7
[junit4:junit4] 2> 21991 T1851 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[22 (1429374746147946496)]} 0 0
[junit4:junit4] 2> 21991 T1867 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[22 (1429374746147946496)]} 0 0
[junit4:junit4] 2> 21992 T1839 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[22 (1429374746147946496)]} 0 4
[junit4:junit4] 2> 21992 T1866 C99 P35038 /update {version=2&wt=javabin} {add=[22]} 0 6
[junit4:junit4] 2> 22000 T1865 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[23 (1429374746157383680)]} 0 1
[junit4:junit4] 2> 22000 T1852 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[23 (1429374746157383680)]} 0 1
[junit4:junit4] 2> 22001 T1838 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[23 (1429374746157383680)]} 0 4
[junit4:junit4] 2> 22001 T1868 C99 P35038 /update {version=2&wt=javabin} {add=[23]} 0 6
[junit4:junit4] 2> 22009 T1866 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[24 (1429374746166820864)]} 0 1
[junit4:junit4] 2> 22009 T1854 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[24 (1429374746166820864)]} 0 1
[junit4:junit4] 2> 22010 T1837 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[24 (1429374746166820864)]} 0 5
[junit4:junit4] 2> 22010 T1867 C99 P35038 /update {version=2&wt=javabin} {add=[24]} 0 7
[junit4:junit4] 2> 22017 T1868 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[25 (1429374746176258048)]} 0 0
[junit4:junit4] 2> 22019 T1853 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[25 (1429374746176258048)]} 0 1
[junit4:junit4] 2> 22019 T1840 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[25 (1429374746176258048)]} 0 5
[junit4:junit4] 2> 22020 T1865 C99 P35038 /update {version=2&wt=javabin} {add=[25]} 0 8
[junit4:junit4] 2> 22029 T1851 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[26 (1429374746187792384)]} 0 1
[junit4:junit4] 2> 22029 T1867 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[26 (1429374746187792384)]} 0 1
[junit4:junit4] 2> 22030 T1839 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[26 (1429374746187792384)]} 0 4
[junit4:junit4] 2> 22030 T1866 C99 P35038 /update {version=2&wt=javabin} {add=[26]} 0 6
[junit4:junit4] 2> 22037 T1852 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[27 (1429374746196180992)]} 0 0
[junit4:junit4] 2> 22037 T1865 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[27 (1429374746196180992)]} 0 0
[junit4:junit4] 2> 22038 T1838 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[27 (1429374746196180992)]} 0 4
[junit4:junit4] 2> 22039 T1868 C99 P35038 /update {version=2&wt=javabin} {add=[27]} 0 7
[junit4:junit4] 2> 22046 T1866 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[28 (1429374746205618176)]} 0 1
[junit4:junit4] 2> 22046 T1854 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[28 (1429374746205618176)]} 0 1
[junit4:junit4] 2> 22047 T1837 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[28 (1429374746205618176)]} 0 4
[junit4:junit4] 2> 22047 T1867 C99 P35038 /update {version=2&wt=javabin} {add=[28]} 0 6
[junit4:junit4] 2> 22055 T1868 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[29 (1429374746215055360)]} 0 1
[junit4:junit4] 2> 22055 T1853 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[29 (1429374746215055360)]} 0 1
[junit4:junit4] 2> 22056 T1840 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[29 (1429374746215055360)]} 0 4
[junit4:junit4] 2> 22056 T1865 C99 P35038 /update {version=2&wt=javabin} {add=[29]} 0 6
[junit4:junit4] 2> 22063 T1867 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[30 (1429374746224492544)]} 0 0
[junit4:junit4] 2> 22064 T1851 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[30 (1429374746224492544)]} 0 0
[junit4:junit4] 2> 22065 T1839 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[30 (1429374746224492544)]} 0 5
[junit4:junit4] 2> 22065 T1866 C99 P35038 /update {version=2&wt=javabin} {add=[30]} 0 6
[junit4:junit4] 2> 22073 T1852 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[31 (1429374746233929728)]} 0 1
[junit4:junit4] 2> 22073 T1865 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[31 (1429374746233929728)]} 0 1
[junit4:junit4] 2> 22073 T1838 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[31 (1429374746233929728)]} 0 4
[junit4:junit4] 2> 22074 T1868 C99 P35038 /update {version=2&wt=javabin} {add=[31]} 0 6
[junit4:junit4] 2> 22081 T1854 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[32 (1429374746242318336)]} 0 0
[junit4:junit4] 2> 22081 T1866 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[32 (1429374746242318336)]} 0 0
[junit4:junit4] 2> 22082 T1837 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[32 (1429374746242318336)]} 0 4
[junit4:junit4] 2> 22083 T1867 C99 P35038 /update {version=2&wt=javabin} {add=[32]} 0 7
[junit4:junit4] 2> 22090 T1868 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[33 (1429374746251755520)]} 0 1
[junit4:junit4] 2> 22092 T1853 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[33 (1429374746251755520)]} 0 0
[junit4:junit4] 2> 22093 T1840 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[33 (1429374746251755520)]} 0 6
[junit4:junit4] 2> 22094 T1865 C99 P35038 /update {version=2&wt=javabin} {add=[33]} 0 9
[junit4:junit4] 2> 22101 T1851 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[34 (1429374746263289856)]} 0 1
[junit4:junit4] 2> 22101 T1867 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[34 (1429374746263289856)]} 0 1
[junit4:junit4] 2> 22102 T1839 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[34 (1429374746263289856)]} 0 4
[junit4:junit4] 2> 22102 T1866 C99 P35038 /update {version=2&wt=javabin} {add=[34]} 0 6
[junit4:junit4] 2> 22109 T1852 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[35 (1429374746271678464)]} 0 0
[junit4:junit4] 2> 22109 T1865 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[35 (1429374746271678464)]} 0 0
[junit4:junit4] 2> 22110 T1838 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[35 (1429374746271678464)]} 0 4
[junit4:junit4] 2> 22110 T1868 C99 P35038 /update {version=2&wt=javabin} {add=[35]} 0 6
[junit4:junit4] 2> 22118 T1854 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[36 (1429374746281115648)]} 0 1
[junit4:junit4] 2> 22118 T1866 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[36 (1429374746281115648)]} 0 1
[junit4:junit4] 2> 22119 T1837 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[36 (1429374746281115648)]} 0 4
[junit4:junit4] 2> 22120 T1867 C99 P35038 /update {version=2&wt=javabin} {add=[36]} 0 7
[junit4:junit4] 2> 22126 T1853 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[37 (1429374746290552832)]} 0 0
[junit4:junit4] 2> 22126 T1868 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[37 (1429374746290552832)]} 0 0
[junit4:junit4] 2> 22127 T1840 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[37 (1429374746290552832)]} 0 4
[junit4:junit4] 2> 22128 T1865 C99 P35038 /update {version=2&wt=javabin} {add=[37]} 0 6
[junit4:junit4] 2> 22135 T1851 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[38 (1429374746298941440)]} 0 1
[junit4:junit4] 2> 22135 T1867 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[38 (1429374746298941440)]} 0 1
[junit4:junit4] 2> 22135 T1839 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[38 (1429374746298941440)]} 0 3
[junit4:junit4] 2> 22136 T1866 C99 P35038 /update {version=2&wt=javabin} {add=[38]} 0 6
[junit4:junit4] 2> 22143 T1852 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[39 (1429374746307330048)]} 0 1
[junit4:junit4] 2> 22143 T1865 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[39 (1429374746307330048)]} 0 1
[junit4:junit4] 2> 22144 T1838 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[39 (1429374746307330048)]} 0 4
[junit4:junit4] 2> 22144 T1868 C99 P35038 /update {version=2&wt=javabin} {add=[39]} 0 6
[junit4:junit4] 2> 22153 T1854 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[40 (1429374746317815808)]} 0 0
[junit4:junit4] 2> 22153 T1866 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[40 (1429374746317815808)]} 0 0
[junit4:junit4] 2> 22154 T1840 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[40 (1429374746317815808)]} 0 4
[junit4:junit4] 2> 22155 T1867 C99 P35038 /update {version=2&wt=javabin} {add=[40]} 0 9
[junit4:junit4] 2> 22162 T1865 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[41 (1429374746327252992)]} 0 1
[junit4:junit4] 2> 22162 T1853 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[41 (1429374746327252992)]} 0 1
[junit4:junit4] 2> 22163 T1838 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[41 (1429374746327252992)]} 0 4
[junit4:junit4] 2> 22163 T1868 C99 P35038 /update {version=2&wt=javabin} {add=[41]} 0 6
[junit4:junit4] 2> 22170 T1851 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[42 (1429374746335641600)]} 0 0
[junit4:junit4] 2> 22170 T1867 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[42 (1429374746335641600)]} 0 0
[junit4:junit4] 2> 22171 T1837 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[42 (1429374746335641600)]} 0 4
[junit4:junit4] 2> 22172 T1866 C99 P35038 /update {version=2&wt=javabin} {add=[42]} 0 7
[junit4:junit4] 2> 22179 T1852 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[43 (1429374746345078784)]} 0 1
[junit4:junit4] 2> 22179 T1868 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[43 (1429374746345078784)]} 0 1
[junit4:junit4] 2> 22180 T1840 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[43 (1429374746345078784)]} 0 4
[junit4:junit4] 2> 22180 T1865 C99 P35038 /update {version=2&wt=javabin} {add=[43]} 0 6
[junit4:junit4] 2> 22188 T1866 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[44 (1429374746353467392)]} 0 0
[junit4:junit4] 2> 22187 T1854 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[44 (1429374746353467392)]} 0 0
[junit4:junit4] 2> 22188 T1839 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[44 (1429374746353467392)]} 0 4
[junit4:junit4] 2> 22189 T1867 C99 P35038 /update {version=2&wt=javabin} {add=[44]} 0 6
[junit4:junit4] 2> 22196 T1865 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[45 (1429374746362904576)]} 0 0
[junit4:junit4] 2> 22196 T1853 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[45 (1429374746362904576)]} 0 0
[junit4:junit4] 2> 22196 T1838 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[45 (1429374746362904576)]} 0 3
[junit4:junit4] 2> 22197 T1868 C99 P35038 /update {version=2&wt=javabin} {add=[45]} 0 6
[junit4:junit4] 2> 22204 T1867 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[46 (1429374746371293184)]} 0 1
[junit4:junit4] 2> 22204 T1851 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[46 (1429374746371293184)]} 0 1
[junit4:junit4] 2> 22205 T1837 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[46 (1429374746371293184)]} 0 4
[junit4:junit4] 2> 22205 T1866 C99 P35038 /update {version=2&wt=javabin} {add=[46]} 0 6
[junit4:junit4] 2> 22213 T1868 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[47 (1429374746380730368)]} 0 0
[junit4:junit4] 2> 22213 T1852 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[47 (1429374746380730368)]} 0 0
[junit4:junit4] 2> 22214 T1840 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[47 (1429374746380730368)]} 0 4
[junit4:junit4] 2> 22215 T1865 C99 P35038 /update {version=2&wt=javabin} {add=[47]} 0 7
[junit4:junit4] 2> 22222 T1854 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[48 (1429374746390167552)]} 0 1
[junit4:junit4] 2> 22222 T1866 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[48 (1429374746390167552)]} 0 1
[junit4:junit4] 2> 22223 T1839 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[48 (1429374746390167552)]} 0 4
[junit4:junit4] 2> 22223 T1867 C99 P35038 /update {version=2&wt=javabin} {add=[48]} 0 6
[junit4:junit4] 2> 22230 T1865 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[49 (1429374746398556160)]} 0 1
[junit4:junit4] 2> 22230 T1853 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[49 (1429374746398556160)]} 0 1
[junit4:junit4] 2> 22231 T1838 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[49 (1429374746398556160)]} 0 4
[junit4:junit4] 2> 22231 T1868 C99 P35038 /update {version=2&wt=javabin} {add=[49]} 0 6
[junit4:junit4] 2> 22239 T1867 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[50 (1429374746406944768)]} 0 1
[junit4:junit4] 2> 22239 T1851 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[50 (1429374746406944768)]} 0 1
[junit4:junit4] 2> 22240 T1837 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[50 (1429374746406944768)]} 0 5
[junit4:junit4] 2> 22240 T1866 C99 P35038 /update {version=2&wt=javabin} {add=[50]} 0 7
[junit4:junit4] 2> 22247 T1868 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[51 (1429374746416381952)]} 0 0
[junit4:junit4] 2> 22247 T1852 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[51 (1429374746416381952)]} 0 0
[junit4:junit4] 2> 22248 T1840 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[51 (1429374746416381952)]} 0 4
[junit4:junit4] 2> 22248 T1865 C99 P35038 /update {version=2&wt=javabin} {add=[51]} 0 6
[junit4:junit4] 2> 22255 T1854 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[52 (1429374746424770560)]} 0 1
[junit4:junit4] 2> 22255 T1866 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[52 (1429374746424770560)]} 0 1
[junit4:junit4] 2> 22256 T1839 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[52 (1429374746424770560)]} 0 4
[junit4:junit4] 2> 22256 T1867 C99 P35038 /update {version=2&wt=javabin} {add=[52]} 0 6
[junit4:junit4] 2> 22267 T1853 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[53 (1429374746433159168)]} 0 0
[junit4:junit4] 2> 22267 T1865 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[53 (1429374746433159168)]} 0 0
[junit4:junit4] 2> 22268 T1838 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[53 (1429374746433159168)]} 0 8
[junit4:junit4] 2> 22269 T1868 C99 P35038 /update {version=2&wt=javabin} {add=[53]} 0 11
[junit4:junit4] 2> 22276 T1867 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[54 (1429374746446790656)]} 0 1
[junit4:junit4] 2> 22276 T1851 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[54 (1429374746446790656)]} 0 1
[junit4:junit4] 2> 22276 T1837 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[54 (1429374746446790656)]} 0 3
[junit4:junit4] 2> 22277 T1866 C99 P35038 /update {version=2&wt=javabin} {add=[54]} 0 6
[junit4:junit4] 2> 22284 T1868 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[55 (1429374746455179264)]} 0 1
[junit4:junit4] 2> 22284 T1852 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[55 (1429374746455179264)]} 0 1
[junit4:junit4] 2> 22285 T1840 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[55 (1429374746455179264)]} 0 4
[junit4:junit4] 2> 22285 T1865 C99 P35038 /update {version=2&wt=javabin} {add=[55]} 0 6
[junit4:junit4] 2> 22292 T1866 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[56 (1429374746463567872)]} 0 0
[junit4:junit4] 2> 22292 T1854 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[56 (1429374746463567872)]} 0 0
[junit4:junit4] 2> 22293 T1839 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[56 (1429374746463567872)]} 0 4
[junit4:junit4] 2> 22294 T1867 C99 P35038 /update {version=2&wt=javabin} {add=[56]} 0 7
[junit4:junit4] 2> 22301 T1853 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[57 (1429374746473005056)]} 0 1
[junit4:junit4] 2> 22301 T1865 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[57 (1429374746473005056)]} 0 1
[junit4:junit4] 2> 22301 T1838 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[57 (1429374746473005056)]} 0 3
[junit4:junit4] 2> 22302 T1868 C99 P35038 /update {version=2&wt=javabin} {add=[57]} 0 6
[junit4:junit4] 2> 22309 T1851 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[58 (1429374746481393664)]} 0 1
[junit4:junit4] 2> 22309 T1867 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[58 (1429374746481393664)]} 0 1
[junit4:junit4] 2> 22310 T1837 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[58 (1429374746481393664)]} 0 4
[junit4:junit4] 2> 22310 T1866 C99 P35038 /update {version=2&wt=javabin} {add=[58]} 0 6
[junit4:junit4] 2> 22317 T1868 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[59 (1429374746489782272)]} 0 0
[junit4:junit4] 2> 22317 T1852 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[59 (1429374746489782272)]} 0 0
[junit4:junit4] 2> 22318 T1840 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[59 (1429374746489782272)]} 0 4
[junit4:junit4] 2> 22318 T1865 C99 P35038 /update {version=2&wt=javabin} {add=[59]} 0 6
[junit4:junit4] 2> 22326 T1854 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[60 (1429374746499219456)]} 0 1
[junit4:junit4] 2> 22326 T1866 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[60 (1429374746499219456)]} 0 1
[junit4:junit4] 2> 22326 T1839 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[60 (1429374746499219456)]} 0 4
[junit4:junit4] 2> 22327 T1867 C99 P35038 /update {version=2&wt=javabin} {add=[60]} 0 6
[junit4:junit4] 2> 22337 T1853 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[61 (1429374746510753792)]} 0 0
[junit4:junit4] 2> 22337 T1865 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[61 (1429374746510753792)]} 0 0
[junit4:junit4] 2> 22338 T1838 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[61 (1429374746510753792)]} 0 4
[junit4:junit4] 2> 22339 T1868 C99 P35038 /update {version=2&wt=javabin} {add=[61]} 0 10
[junit4:junit4] 2> 22346 T1851 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[62 (1429374746520190976)]} 0 1
[junit4:junit4] 2> 22346 T1867 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[62 (1429374746520190976)]} 0 1
[junit4:junit4] 2> 22346 T1837 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[62 (1429374746520190976)]} 0 4
[junit4:junit4] 2> 22347 T1866 C99 P35038 /update {version=2&wt=javabin} {add=[62]} 0 6
[junit4:junit4] 2> 22354 T1852 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[63 (1429374746528579584)]} 0 1
[junit4:junit4] 2> 22354 T1868 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[63 (1429374746528579584)]} 0 1
[junit4:junit4] 2> 22355 T1840 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[63 (1429374746528579584)]} 0 4
[junit4:junit4] 2> 22355 T1865 C99 P35038 /update {version=2&wt=javabin} {add=[63]} 0 6
[junit4:junit4] 2> 22363 T1866 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[64 (1429374746538016768)]} 0 1
[junit4:junit4] 2> 22363 T1854 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[64 (1429374746538016768)]} 0 1
[junit4:junit4] 2> 22363 T1839 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[64 (1429374746538016768)]} 0 4
[junit4:junit4] 2> 22364 T1867 C99 P35038 /update {version=2&wt=javabin} {add=[64]} 0 6
[junit4:junit4] 2> 22371 T1853 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[65 (1429374746546405376)]} 0 1
[junit4:junit4] 2> 22373 T1865 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[65 (1429374746546405376)]} 0 1
[junit4:junit4] 2> 22373 T1838 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[65 (1429374746546405376)]} 0 5
[junit4:junit4] 2> 22374 T1868 C99 P35038 /update {version=2&wt=javabin} {add=[65]} 0 8
[junit4:junit4] 2> 22381 T1867 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[66 (1429374746556891136)]} 0 1
[junit4:junit4] 2> 22381 T1852 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[66 (1429374746556891136)]} 0 1
[junit4:junit4] 2> 22382 T1837 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[66 (1429374746556891136)]} 0 4
[junit4:junit4] 2> 22382 T1866 C99 P35038 /update {version=2&wt=javabin} {add=[66]} 0 6
[junit4:junit4] 2> 22389 T1868 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[67 (1429374746565279744)]} 0 0
[junit4:junit4] 2> 22389 T1854 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[67 (1429374746565279744)]} 0 0
[junit4:junit4] 2> 22390 T1840 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[67 (1429374746565279744)]} 0 4
[junit4:junit4] 2> 22391 T1865 C99 P35038 /update {version=2&wt=javabin} {add=[67]} 0 7
[junit4:junit4] 2> 22398 T1868 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[68 (1429374746574716928)]} 0 0
[junit4:junit4] 2> 22398 T1851 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[68 (1429374746574716928)]} 0 0
[junit4:junit4] 2> 22404 T1839 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[68 (1429374746574716928)]} 0 9
[junit4:junit4] 2> 22404 T1866 C99 P35038 /update {version=2&wt=javabin} {add=[68]} 0 11
[junit4:junit4] 2> 22412 T1867 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[69 (1429374746589396992)]} 0 1
[junit4:junit4] 2> 22412 T1853 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[69 (1429374746589396992)]} 0 1
[junit4:junit4] 2> 22413 T1838 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[69 (1429374746589396992)]} 0 4
[junit4:junit4] 2> 22413 T1865 C99 P35038 /update {version=2&wt=javabin} {add=[69]} 0 6
[junit4:junit4] 2> 22420 T1852 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[70 (1429374746597785600)]} 0 0
[junit4:junit4] 2> 22422 T1866 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[70 (1429374746597785600)]} 0 1
[junit4:junit4] 2> 22422 T1837 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[70 (1429374746597785600)]} 0 5
[junit4:junit4] 2> 22423 T1868 C99 P35038 /update {version=2&wt=javabin} {add=[70]} 0 8
[junit4:junit4] 2> 22430 T1854 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[71 (1429374746608271360)]} 0 1
[junit4:junit4] 2> 22430 T1865 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[71 (1429374746608271360)]} 0 1
[junit4:junit4] 2> 22431 T1840 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[71 (1429374746608271360)]} 0 4
[junit4:junit4] 2> 22432 T1867 C99 P35038 /update {version=2&wt=javabin} {add=[71]} 0 7
[junit4:junit4] 2> 22439 T1868 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[72 (1429374746617708544)]} 0 1
[junit4:junit4] 2> 22439 T1851 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[72 (1429374746617708544)]} 0 1
[junit4:junit4] 2> 22440 T1839 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[72 (1429374746617708544)]} 0 4
[junit4:junit4] 2> 22440 T1866 C99 P35038 /update {version=2&wt=javabin} {add=[72]} 0 6
[junit4:junit4] 2> 22447 T1853 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[73 (1429374746626097152)]} 0 0
[junit4:junit4] 2> 22447 T1867 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[73 (1429374746626097152)]} 0 0
[junit4:junit4] 2> 22448 T1838 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[73 (1429374746626097152)]} 0 4
[junit4:junit4] 2> 22449 T1865 C99 P35038 /update {version=2&wt=javabin} {add=[73]} 0 7
[junit4:junit4] 2> 22455 T1852 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[74 (1429374746635534336)]} 0 0
[junit4:junit4] 2> 22455 T1866 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[74 (1429374746635534336)]} 0 0
[junit4:junit4] 2> 22456 T1837 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[74 (1429374746635534336)]} 0 4
[junit4:junit4] 2> 22457 T1868 C99 P35038 /update {version=2&wt=javabin} {add=[74]} 0 6
[junit4:junit4] 2> 22464 T1865 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[75 (1429374746643922944)]} 0 1
[junit4:junit4] 2> 22464 T1854 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[75 (1429374746643922944)]} 0 1
[junit4:junit4] 2> 22465 T1840 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[75 (1429374746643922944)]} 0 4
[junit4:junit4] 2> 22465 T1867 C99 P35038 /update {version=2&wt=javabin} {add=[75]} 0 6
[junit4:junit4] 2> 22473 T1868 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[76 (1429374746653360128)]} 0 1
[junit4:junit4] 2> 22473 T1851 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[76 (1429374746653360128)]} 0 1
[junit4:junit4] 2> 22474 T1839 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[76 (1429374746653360128)]} 0 5
[junit4:junit4] 2> 22474 T1866 C99 P35038 /update {version=2&wt=javabin} {add=[76]} 0 6
[junit4:junit4] 2> 22481 T1867 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[77 (1429374746661748736)]} 0 0
[junit4:junit4] 2> 22481 T1853 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[77 (1429374746661748736)]} 0 0
[junit4:junit4] 2> 22482 T1838 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[77 (1429374746661748736)]} 0 4
[junit4:junit4] 2> 22482 T1865 C99 P35038 /update {version=2&wt=javabin} {add=[77]} 0 6
[junit4:junit4] 2> 22490 T1852 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[78 (1429374746671185920)]} 0 1
[junit4:junit4] 2> 22490 T1866 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[78 (1429374746671185920)]} 0 1
[junit4:junit4] 2> 22490 T1837 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[78 (1429374746671185920)]} 0 3
[junit4:junit4] 2> 22491 T1868 C99 P35038 /update {version=2&wt=javabin} {add=[78]} 0 6
[junit4:junit4] 2> 22498 T1865 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[79 (1429374746679574528)]} 0 1
[junit4:junit4] 2> 22498 T1854 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[79 (1429374746679574528)]} 0 1
[junit4:junit4] 2> 22498 T1840 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[79 (1429374746679574528)]} 0 3
[junit4:junit4] 2> 22499 T1867 C99 P35038 /update {version=2&wt=javabin} {add=[79]} 0 6
[junit4:junit4] 2> 22506 T1851 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[80 (1429374746687963136)]} 0 1
[junit4:junit4] 2> 22506 T1868 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[80 (1429374746687963136)]} 0 1
[junit4:junit4] 2> 22507 T1839 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[80 (1429374746687963136)]} 0 4
[junit4:junit4] 2> 22507 T1866 C99 P35038 /update {version=2&wt=javabin} {add=[80]} 0 6
[junit4:junit4] 2> 22514 T1867 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[81 (1429374746696351744)]} 0 0
[junit4:junit4] 2> 22514 T1852 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[81 (1429374746696351744)]} 0 0
[junit4:junit4] 2> 22515 T1838 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[81 (1429374746696351744)]} 0 4
[junit4:junit4] 2> 22516 T1865 C99 P35038 /update {version=2&wt=javabin} {add=[81]} 0 7
[junit4:junit4] 2> 22523 T1854 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[82 (1429374746705788928)]} 0 1
[junit4:junit4] 2> 22523 T1866 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[82 (1429374746705788928)]} 0 1
[junit4:junit4] 2> 22526 T1837 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[82 (1429374746705788928)]} 0 6
[junit4:junit4] 2> 22527 T1868 C99 P35038 /update {version=2&wt=javabin} {add=[82]} 0 9
[junit4:junit4] 2> 22534 T1853 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[83 (1429374746717323264)]} 0 0
[junit4:junit4] 2> 22534 T1865 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[83 (1429374746717323264)]} 0 0
[junit4:junit4] 2> 22535 T1840 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[83 (1429374746717323264)]} 0 4
[junit4:junit4] 2> 22536 T1867 C99 P35038 /update {version=2&wt=javabin} {add=[83]} 0 7
[junit4:junit4] 2> 22543 T1851 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[84 (1429374746726760448)]} 0 1
[junit4:junit4] 2> 22543 T1868 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[84 (1429374746726760448)]} 0 1
[junit4:junit4] 2> 22544 T1839 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[84 (1429374746726760448)]} 0 4
[junit4:junit4] 2> 22545 T1866 C99 P35038 /update {version=2&wt=javabin} {add=[84]} 0 7
[junit4:junit4] 2> 22552 T1852 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[85 (1429374746736197632)]} 0 1
[junit4:junit4] 2> 22552 T1867 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[85 (1429374746736197632)]} 0 0
[junit4:junit4] 2> 22553 T1837 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[85 (1429374746736197632)]} 0 4
[junit4:junit4] 2> 22553 T1865 C99 P35038 /update {version=2&wt=javabin} {add=[85]} 0 6
[junit4:junit4] 2> 22561 T1854 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[86 (1429374746745634816)]} 0 1
[junit4:junit4] 2> 22561 T1866 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[86 (1429374746745634816)]} 0 1
[junit4:junit4] 2> 22561 T1840 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[86 (1429374746745634816)]} 0 4
[junit4:junit4] 2> 22562 T1868 C99 P35038 /update {version=2&wt=javabin} {add=[86]} 0 6
[junit4:junit4] 2> 22569 T1867 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[87 (1429374746754023424)]} 0 1
[junit4:junit4] 2> 22569 T1853 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[87 (1429374746754023424)]} 0 1
[junit4:junit4] 2> 22570 T1839 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[87 (1429374746754023424)]} 0 4
[junit4:junit4] 2> 22570 T1865 C99 P35038 /update {version=2&wt=javabin} {add=[87]} 0 6
[junit4:junit4] 2> 22577 T1851 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[88 (1429374746762412032)]} 0 0
[junit4:junit4] 2> 22577 T1868 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[88 (1429374746762412032)]} 0 0
[junit4:junit4] 2> 22578 T1838 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[88 (1429374746762412032)]} 0 4
[junit4:junit4] 2> 22579 T1866 C99 P35038 /update {version=2&wt=javabin} {add=[88]} 0 7
[junit4:junit4] 2> 22586 T1852 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[89 (1429374746771849216)]} 0 0
[junit4:junit4] 2> 22586 T1865 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[89 (1429374746771849216)]} 0 0
[junit4:junit4] 2> 22587 T1837 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[89 (1429374746771849216)]} 0 4
[junit4:junit4] 2> 22588 T1867 C99 P35038 /update {version=2&wt=javabin} {add=[89]} 0 7
[junit4:junit4] 2> 22594 T1866 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[90 (1429374746781286400)]} 0 0
[junit4:junit4] 2> 22594 T1854 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[90 (1429374746781286400)]} 0 0
[junit4:junit4] 2> 22595 T1840 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[90 (1429374746781286400)]} 0 4
[junit4:junit4] 2> 22596 T1868 C99 P35038 /update {version=2&wt=javabin} {add=[90]} 0 6
[junit4:junit4] 2> 22606 T1867 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[91 (1429374746792820736)]} 0 1
[junit4:junit4] 2> 22606 T1853 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[91 (1429374746792820736)]} 0 1
[junit4:junit4] 2> 22607 T1839 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[91 (1429374746792820736)]} 0 4
[junit4:junit4] 2> 22607 T1865 C99 P35038 /update {version=2&wt=javabin} {add=[91]} 0 9
[junit4:junit4] 2> 22614 T1868 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[92 (1429374746801209344)]} 0 0
[junit4:junit4] 2> 22614 T1851 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[92 (1429374746801209344)]} 0 1
[junit4:junit4] 2> 22615 T1838 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[92 (1429374746801209344)]} 0 4
[junit4:junit4] 2> 22616 T1866 C99 P35038 /update {version=2&wt=javabin} {add=[92]} 0 7
[junit4:junit4] 2> 22628 T1852 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[93 (1429374746815889408)]} 0 0
[junit4:junit4] 2> 22628 T1865 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[93 (1429374746815889408)]} 0 0
[junit4:junit4] 2> 22629 T1837 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[93 (1429374746815889408)]} 0 9
[junit4:junit4] 2> 22630 T1867 C99 P35038 /update {version=2&wt=javabin} {add=[93]} 0 12
[junit4:junit4] 2> 22637 T1854 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[94 (1429374746825326592)]} 0 1
[junit4:junit4] 2> 22637 T1866 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[94 (1429374746825326592)]} 0 1
[junit4:junit4] 2> 22637 T1840 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[94 (1429374746825326592)]} 0 3
[junit4:junit4] 2> 22638 T1868 C99 P35038 /update {version=2&wt=javabin} {add=[94]} 0 6
[junit4:junit4] 2> 22645 T1867 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[95 (1429374746833715200)]} 0 1
[junit4:junit4] 2> 22645 T1853 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[95 (1429374746833715200)]} 0 1
[junit4:junit4] 2> 22646 T1839 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[95 (1429374746833715200)]} 0 4
[junit4:junit4] 2> 22646 T1865 C99 P35038 /update {version=2&wt=javabin} {add=[95]} 0 6
[junit4:junit4] 2> 22653 T1868 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[96 (1429374746843152384)]} 0 0
[junit4:junit4] 2> 22653 T1851 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[96 (1429374746843152384)]} 0 0
[junit4:junit4] 2> 22654 T1838 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[96 (1429374746843152384)]} 0 4
[junit4:junit4] 2> 22655 T1866 C99 P35038 /update {version=2&wt=javabin} {add=[96]} 0 6
[junit4:junit4] 2> 22662 T1852 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[97 (1429374746851540992)]} 0 1
[junit4:junit4] 2> 22662 T1865 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[97 (1429374746851540992)]} 0 1
[junit4:junit4] 2> 22663 T1837 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[97 (1429374746851540992)]} 0 4
[junit4:junit4] 2> 22664 T1867 C99 P35038 /update {version=2&wt=javabin} {add=[97]} 0 7
[junit4:junit4] 2> 22672 T1854 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[98 (1429374746862026752)]} 0 1
[junit4:junit4] 2> 22672 T1866 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[98 (1429374746862026752)]} 0 1
[junit4:junit4] 2> 22673 T1840 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[98 (1429374746862026752)]} 0 4
[junit4:junit4] 2> 22673 T1868 C99 P35038 /update {version=2&wt=javabin} {add=[98]} 0 6
[junit4:junit4] 2> 22680 T1853 C97 P33530 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[99 (1429374746870415360)]} 0 0
[junit4:junit4] 2> 22680 T1867 C99 P35038 /update {distrib.from=http://127.0.0.1:35631/fy/td/unloadcollection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[99 (1429374746870415360)]} 0 0
[junit4:junit4] 2> 22681 T1839 C98 P35631 /update {distrib.from=http://127.0.0.1:35038/fy/td/unloadcollection3/&version=2&wt=javabin&update.distrib=TOLEADER} {add=[99 (1429374746870415360)]} 0 4
[junit4:junit4] 2> 22681 T1865 C99 P35038 /update {version=2&wt=javabin} {add=[99]} 0 6
[junit4:junit4] 2> 22682 T1804 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 22686 T1837 oasha.CoreAdminHandler.handleUnloadAction Unregistering core unloadcollection1 from cloudstate.
[junit4:junit4] 2> 22688 T1837 oasc.SolrCore.close [unloadcollection1] CLOSING SolrCore org.apache.solr.core.SolrCore@1c62774c
[junit4:junit4] 2> 22689 T1858 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 22689 T1809 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d628bccd50008 type:create cxid:0x55 zxid:0x8f txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 22690 T1858 oasc.ShardLeaderElectionContext.shouldIBeLeader Checking if I should try and be the leader.
[junit4:junit4] 2> 22691 T1858 oasc.ShardLeaderElectionContext.shouldIBeLeader My last published State was Active, it's okay to be the leader.
[junit4:junit4] 2> 22691 T1858 oasc.ShardLeaderElectionContext.runLeaderProcess
[...truncated too long message...]
hard_and_collection",
[junit4:junit4] 1> "node_name":"127.0.0.1:35631_fy%2Ftd",
[junit4:junit4] 1> "shard":"shard2",
[junit4:junit4] 1> "core":"test_unload_shard_and_collection_1",
[junit4:junit4] 1> "leader":"true"}}}},
[junit4:junit4] 1> "router":"compositeId"}}
[junit4:junit4] 1>
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=UnloadDistributedZkTest -Dtests.method=testDistribSearch -Dtests.seed=8ED9F67CEAE3B8A2 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ms_MY -Dtests.timezone=VST -Dtests.file.encoding=ISO-8859-1
[junit4:junit4] FAILURE 140s J1 | UnloadDistributedZkTest.testDistribSearch <<<
[junit4:junit4] > Throwable #1: java.lang.AssertionError: Still found shard
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([8ED9F67CEAE3B8A2:F3F78649DBCD89E]:0)
[junit4:junit4] > at org.junit.Assert.fail(Assert.java:93)
[junit4:junit4] > at org.apache.solr.cloud.UnloadDistributedZkTest.testUnloadShardAndCollection(UnloadDistributedZkTest.java:124)
[junit4:junit4] > at org.apache.solr.cloud.UnloadDistributedZkTest.doTest(UnloadDistributedZkTest.java:77)
[junit4:junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:806)
[junit4:junit4] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit4:junit4] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
[junit4:junit4] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit4:junit4] > at java.lang.reflect.Method.invoke(Method.java:487)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
[junit4:junit4] > at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
[junit4:junit4] > at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
[junit4:junit4] > at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4:junit4] > at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
[junit4:junit4] > at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
[junit4:junit4] > at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
[junit4:junit4] > at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
[junit4:junit4] > at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4] > at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
[junit4:junit4] > at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4:junit4] > at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
[junit4:junit4] > at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:722)
[junit4:junit4] 2> 139635 T1804 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4] 2> NOTE: test params are: codec=Lucene42: {range_facet_l=PostingsFormat(name=MockSep), multiDefault=PostingsFormat(name=MockSep), timestamp=PostingsFormat(name=MockSep), a_t=PostingsFormat(name=MockSep), text=Lucene41(blocksize=128), range_facet_si=PostingsFormat(name=MockSep), range_facet_sl=PostingsFormat(name=Memory doPackFST= false), id=PostingsFormat(name=Memory doPackFST= false), _version_=Pulsing41(freqCutoff=2 minBlockSize=72 maxBlockSize=178), other_tl1=Pulsing41(freqCutoff=2 minBlockSize=72 maxBlockSize=178), a_si=Pulsing41(freqCutoff=2 minBlockSize=72 maxBlockSize=178), intDefault=Pulsing41(freqCutoff=2 minBlockSize=72 maxBlockSize=178)}, docValues:{timestamp=DocValuesFormat(name=CheapBastard)}, sim=DefaultSimilarity, locale=ms_MY, timezone=VST
[junit4:junit4] 2> NOTE: Linux 3.2.0-38-generic amd64/Oracle Corporation 1.8.0-ea (64-bit)/cpus=8,threads=2,free=72092008,total=166981632
[junit4:junit4] 2> NOTE: All tests run in this JVM: [ConvertedLegacyTest, SolrIndexConfigTest, TestFiltering, TermVectorComponentDistributedTest, TestZkChroot, RequestHandlersTest, IndexSchemaRuntimeFieldTest, DocValuesMultiTest, TestLMJelinekMercerSimilarityFactory, TestHashPartitioner, StandardRequestHandlerTest, SimplePostToolTest, UniqFieldsUpdateProcessorFactoryTest, JSONWriterTest, TestFoldingMultitermQuery, UpdateParamsTest, FullSolrCloudDistribCmdsTest, TestFieldCollectionResource, XmlUpdateRequestHandlerTest, DistanceFunctionTest, TimeZoneUtilsTest, TestCSVResponseWriter, HighlighterConfigTest, BasicDistributedZk2Test, IndexReaderFactoryTest, MultiTermTest, PingRequestHandlerTest, TestLuceneMatchVersion, SampleTest, TestPropInject, HighlighterTest, CoreContainerCoreInitFailuresTest, OutputWriterTest, TestBM25SimilarityFactory, DebugComponentTest, CurrencyFieldXmlFileTest, TestPartialUpdateDeduplication, SpellCheckComponentTest, SolrPluginUtilsTest, StatsComponentTest, NotRequiredUniqueKeyTest, SolrCmdDistributorTest, TestLazyCores, ZkNodePropsTest, HardAutoCommitTest, SortByFunctionTest, TestBinaryField, TestFieldTypeCollectionResource, TestDistributedGrouping, ChaosMonkeySafeLeaderTest, TestPostingsSolrHighlighter, TestMergePolicyConfig, TestMultiCoreConfBootstrap, DateMathParserTest, TestBinaryResponseWriter, TestRandomFaceting, CurrencyFieldOpenExchangeTest, DirectUpdateHandlerTest, TestSolrXMLSerializer, BasicZkTest, TestIndexingPerformance, SuggesterTSTTest, ShardRoutingCustomTest, AliasIntegrationTest, TestArbitraryIndexDir, UnloadDistributedZkTest]
[junit4:junit4] Completed on J1 in 139.67s, 1 test, 1 failure <<< FAILURES!
[...truncated 419 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:381: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:361: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:183: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:449: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1213: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:877: There were test failures: 271 suites, 1141 tests, 1 failure, 13 ignored (7 assumptions)
Total time: 48 minutes 33 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Description set: Java: 64bit/jdk1.8.0-ea-b79 -XX:+UseConcMarkSweepGC
Email was triggered for: Failure
Sending email for trigger: Failure