You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2013/03/13 17:57:43 UTC

[JENKINS] Lucene-Solr-Tests-trunk-Java6 - Build # 16000 - Failure

Build: https://builds.apache.org/job/Lucene-Solr-Tests-trunk-Java6/16000/

1 tests failed.
REGRESSION:  org.apache.solr.cloud.SyncSliceTest.testDistribSearch

Error Message:
Test Setup Failure: shard1 should have just been set up to be inconsistent - but it's still consistent. Leader:http://127.0.0.1:61449/collection1skip list:[CloudJettyRunner [url=http://127.0.0.1:61453/collection1], CloudJettyRunner [url=http://127.0.0.1:27180/collection1]]

Stack Trace:
java.lang.AssertionError: Test Setup Failure: shard1 should have just been set up to be inconsistent - but it's still consistent. Leader:http://127.0.0.1:61449/collection1skip list:[CloudJettyRunner [url=http://127.0.0.1:61453/collection1], CloudJettyRunner [url=http://127.0.0.1:27180/collection1]]
	at __randomizedtesting.SeedInfo.seed([8651EC74A9666928:7B7626CDE390914]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNotNull(Assert.java:526)
	at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:212)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:806)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:679)




Build Log:
[...truncated 8669 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.SyncSliceTest
[junit4:junit4]   2> 0 T1998 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4:junit4]   2> 5 T1998 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4]   2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./solrtest-SyncSliceTest-1363193302973
[junit4:junit4]   2> 6 T1998 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 6 T1999 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4]   2> 9 T1999 oazs.ZooKeeperServer.setTickTime tickTime set to 1000
[junit4:junit4]   2> 10 T1999 oazs.ZooKeeperServer.setMinSessionTimeout minSessionTimeout set to -1
[junit4:junit4]   2> 10 T1999 oazs.ZooKeeperServer.setMaxSessionTimeout maxSessionTimeout set to -1
[junit4:junit4]   2> 10 T1999 oazs.NIOServerCnxnFactory.configure binding to port 0.0.0.0/0.0.0.0:0
[junit4:junit4]   2> 11 T1999 oazsp.FileTxnSnapLog.save Snapshotting: 0x0 to /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-1363193302973/zookeeper/server1/data/version-2/snapshot.0
[junit4:junit4]   2> 107 T1998 oasc.ZkTestServer.run start zk server on port:61439
[junit4:junit4]   2> 107 T1998 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:61439 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@a933114
[junit4:junit4]   2> 108 T1998 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 108 T2004 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:61439. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 109 T2004 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:61439, initiating session
[junit4:junit4]   2> 109 T2000 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:61440
[junit4:junit4]   2> 109 T2000 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:61440
[junit4:junit4]   2> 110 T2002 oazsp.FileTxnLog.append Creating new log file: log.1
[junit4:junit4]   2> 138 T2002 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d64a70fc40000 with negotiated timeout 10000 for client /140.211.11.196:61440
[junit4:junit4]   2> 138 T2004 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:61439, sessionid = 0x13d64a70fc40000, negotiated timeout = 10000
[junit4:junit4]   2> 138 T2005 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@a933114 name:ZooKeeperConnection Watcher:127.0.0.1:61439 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 138 T1998 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 139 T1998 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4]   2> 142 T2003 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d64a70fc40000
[junit4:junit4]   2> 153 T2005 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 153 T2000 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:61440 which had sessionid 0x13d64a70fc40000
[junit4:junit4]   2> 153 T1998 oaz.ZooKeeper.close Session: 0x13d64a70fc40000 closed
[junit4:junit4]   2> 154 T1998 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:61439/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@4b17c2fa
[junit4:junit4]   2> 155 T1998 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 155 T2006 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:61439. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 155 T2006 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:61439, initiating session
[junit4:junit4]   2> 155 T2000 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:61441
[junit4:junit4]   2> 156 T2000 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:61441
[junit4:junit4]   2> 157 T2002 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d64a70fc40001 with negotiated timeout 10000 for client /140.211.11.196:61441
[junit4:junit4]   2> 157 T2006 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:61439, sessionid = 0x13d64a70fc40001, negotiated timeout = 10000
[junit4:junit4]   2> 157 T2007 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4b17c2fa name:ZooKeeperConnection Watcher:127.0.0.1:61439/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 157 T1998 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 158 T1998 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4]   2> 162 T1998 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4]   2> 172 T1998 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4]   2> 174 T1998 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4]   2> 177 T1998 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 178 T1998 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 188 T1998 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4]   2> 189 T1998 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4]   2> 293 T1998 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4]   2> 293 T1998 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4]   2> 296 T1998 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4]   2> 297 T1998 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4]   2> 307 T1998 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4]   2> 307 T1998 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4]   2> 311 T1998 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 312 T1998 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 315 T1998 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 316 T1998 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 320 T1998 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 321 T1998 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 324 T1998 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4]   2> 325 T1998 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4]   2> 328 T2003 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d64a70fc40001
[junit4:junit4]   2> 329 T2000 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:61441 which had sessionid 0x13d64a70fc40001
[junit4:junit4]   2> 333 T2007 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 333 T1998 oaz.ZooKeeper.close Session: 0x13d64a70fc40001 closed
[junit4:junit4]   2> 614 T1998 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 621 T1998 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:61442
[junit4:junit4]   2> 622 T1998 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 623 T1998 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 624 T1998 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-controljetty-1363193303302
[junit4:junit4]   2> 624 T1998 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-controljetty-1363193303302/solr.xml
[junit4:junit4]   2> 625 T1998 oasc.CoreContainer.<init> New CoreContainer 294424329
[junit4:junit4]   2> 626 T1998 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-controljetty-1363193303302/'
[junit4:junit4]   2> 627 T1998 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-controljetty-1363193303302/'
[junit4:junit4]   2> 707 T1998 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 708 T1998 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 709 T1998 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 710 T1998 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 710 T1998 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 711 T1998 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 712 T1998 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 712 T1998 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 713 T1998 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 714 T1998 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 748 T1998 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 779 T1998 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:61439/solr
[junit4:junit4]   2> 780 T1998 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 781 T1998 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:61439 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@18fa26e
[junit4:junit4]   2> 782 T1998 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 783 T2017 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:61439. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 784 T2017 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:61439, initiating session
[junit4:junit4]   2> 784 T2000 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:61443
[junit4:junit4]   2> 785 T2000 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:61443
[junit4:junit4]   2> 787 T2002 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d64a70fc40002 with negotiated timeout 20000 for client /140.211.11.196:61443
[junit4:junit4]   2> 787 T2017 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:61439, sessionid = 0x13d64a70fc40002, negotiated timeout = 20000
[junit4:junit4]   2> 787 T2018 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@18fa26e name:ZooKeeperConnection Watcher:127.0.0.1:61439 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 787 T1998 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 789 T2003 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d64a70fc40002
[junit4:junit4]   2> 790 T2018 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 790 T2000 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:61443 which had sessionid 0x13d64a70fc40002
[junit4:junit4]   2> 790 T1998 oaz.ZooKeeper.close Session: 0x13d64a70fc40002 closed
[junit4:junit4]   2> 791 T1998 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 809 T1998 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:61439/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@38da4a28
[junit4:junit4]   2> 811 T2019 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:61439. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 810 T1998 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 811 T2019 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:61439, initiating session
[junit4:junit4]   2> 811 T2000 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:61444
[junit4:junit4]   2> 811 T2000 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:61444
[junit4:junit4]   2> 813 T2002 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d64a70fc40003 with negotiated timeout 20000 for client /140.211.11.196:61444
[junit4:junit4]   2> 813 T2019 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:61439, sessionid = 0x13d64a70fc40003, negotiated timeout = 20000
[junit4:junit4]   2> 813 T2020 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@38da4a28 name:ZooKeeperConnection Watcher:127.0.0.1:61439/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 813 T1998 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 816 T2003 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d64a70fc40003 type:create cxid:0x2 zxid:0x1a txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 817 T1998 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4]   2> 820 T1998 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:61442_
[junit4:junit4]   2> 821 T2003 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d64a70fc40003 type:delete cxid:0x8 zxid:0x1c txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:61442_ Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:61442_
[junit4:junit4]   2> 822 T1998 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:61442_
[junit4:junit4]   2> 825 T1998 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4]   2> 839 T2003 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d64a70fc40003 type:delete cxid:0x16 zxid:0x21 txntype:-1 reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4]   2> 840 T1998 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 843 T1998 oasc.Overseer.start Overseer (id=89338236304097283-127.0.0.1:61442_-n_0000000000) starting
[junit4:junit4]   2> 844 T2003 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d64a70fc40003 type:create cxid:0x1b zxid:0x23 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 851 T2003 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d64a70fc40003 type:create cxid:0x1c zxid:0x24 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 852 T2003 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d64a70fc40003 type:create cxid:0x1d zxid:0x25 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 860 T2003 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d64a70fc40003 type:create cxid:0x1e zxid:0x26 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 861 T2022 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4]   2> 862 T1998 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4]   2> 872 T1998 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4]   2> 874 T1998 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 877 T2021 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 883 T2023 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-controljetty-1363193303302/collection1
[junit4:junit4]   2> 883 T2023 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4]   2> 883 T2023 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 884 T2023 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4]   2> 885 T2023 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-controljetty-1363193303302/collection1/'
[junit4:junit4]   2> 886 T2023 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-controljetty-1363193303302/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 887 T2023 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-controljetty-1363193303302/collection1/lib/README' to classloader
[junit4:junit4]   2> 942 T2023 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 1007 T2023 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 1108 T2023 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 1114 T2023 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 1742 T2023 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 1754 T2023 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 1758 T2023 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 1770 T2023 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 1775 T2023 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 1779 T2023 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 1780 T2023 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 1780 T2023 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 1781 T2023 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 1782 T2023 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 1782 T2023 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 1782 T2023 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 1782 T2023 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-controljetty-1363193303302/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1363193302973/control/data/
[junit4:junit4]   2> 1783 T2023 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3dbe8711
[junit4:junit4]   2> 1783 T2023 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 1784 T2023 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363193302973/control/data forceNew: false
[junit4:junit4]   2> 1784 T2023 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1363193302973/control/data/index/
[junit4:junit4]   2> 1784 T2023 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1363193302973/control/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 1785 T2023 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363193302973/control/data/index forceNew: false
[junit4:junit4]   2> 1788 T2023 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@60c481cc lockFactory=org.apache.lucene.store.NativeFSLockFactory@6ba8ee5d),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 1788 T2023 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 1791 T2023 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 1791 T2023 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 1791 T2023 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 1792 T2023 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 1792 T2023 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 1793 T2023 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 1793 T2023 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 1793 T2023 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 1794 T2023 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 1800 T2023 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 1806 T2023 oass.SolrIndexSearcher.<init> Opening Searcher@106022da main
[junit4:junit4]   2> 1807 T2023 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.SyncSliceTest-1363193302973/control/data/tlog
[junit4:junit4]   2> 1808 T2023 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 1808 T2023 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 1811 T2024 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@106022da main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 1812 T2023 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 1812 T2023 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 1813 T2003 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d64a70fc40003 type:create cxid:0x55 zxid:0x2a txntype:-1 reqpath:n/a Error Path:/solr/overseer/queue Error:KeeperErrorCode = NoNode for /solr/overseer/queue
[junit4:junit4]   2> 2380 T2021 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2381 T2021 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"control_collection",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:61442_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:61442"}
[junit4:junit4]   2> 2381 T2021 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4]   2> 2381 T2021 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 2382 T2003 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d64a70fc40003 type:create cxid:0x5d zxid:0x2d txntype:-1 reqpath:n/a Error Path:/solr/overseer/queue-work Error:KeeperErrorCode = NoNode for /solr/overseer/queue-work
[junit4:junit4]   2> 2398 T2020 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 2829 T2023 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 2829 T2023 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:61442 collection:control_collection shard:shard1
[junit4:junit4]   2> 2829 T2023 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4]   2> 2837 T2003 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d64a70fc40003 type:delete cxid:0x77 zxid:0x37 txntype:-1 reqpath:n/a Error Path:/solr/collections/control_collection/leaders Error:KeeperErrorCode = NoNode for /solr/collections/control_collection/leaders
[junit4:junit4]   2> 2838 T2023 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 2838 T2003 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d64a70fc40003 type:create cxid:0x78 zxid:0x38 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 2850 T2023 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 2850 T2023 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 2850 T2023 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:61442/collection1/
[junit4:junit4]   2> 2850 T2023 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 2851 T2023 oasc.SyncStrategy.syncToMe http://127.0.0.1:61442/collection1/ has no replicas
[junit4:junit4]   2> 2851 T2023 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:61442/collection1/
[junit4:junit4]   2> 2851 T2023 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4]   2> 2855 T2003 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d64a70fc40003 type:create cxid:0x82 zxid:0x3c txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 3902 T2021 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 3918 T2020 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 3935 T2023 oasc.ZkController.register We are http://127.0.0.1:61442/collection1/ and leader is http://127.0.0.1:61442/collection1/
[junit4:junit4]   2> 3935 T2023 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:61442
[junit4:junit4]   2> 3935 T2023 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 3935 T2023 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 3935 T2023 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 3937 T2023 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 3938 T1998 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 3939 T1998 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 3939 T1998 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 3952 T1998 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4]   2> 3953 T1998 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:61439/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@4b9cd906
[junit4:junit4]   2> 3954 T1998 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 3955 T2026 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:61439. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 3955 T2026 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:61439, initiating session
[junit4:junit4]   2> 3955 T2000 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:61445
[junit4:junit4]   2> 3956 T2000 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:61445
[junit4:junit4]   2> 3957 T2002 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d64a70fc40004 with negotiated timeout 10000 for client /140.211.11.196:61445
[junit4:junit4]   2> 3957 T2026 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:61439, sessionid = 0x13d64a70fc40004, negotiated timeout = 10000
[junit4:junit4]   2> 3957 T2027 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4b9cd906 name:ZooKeeperConnection Watcher:127.0.0.1:61439/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 3957 T1998 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 3959 T1998 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 3961 T1998 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4]   2> 4145 T1998 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 4147 T1998 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:61446
[junit4:junit4]   2> 4148 T1998 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 4149 T1998 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 4149 T1998 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty1-1363193306930
[junit4:junit4]   2> 4149 T1998 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty1-1363193306930/solr.xml
[junit4:junit4]   2> 4150 T1998 oasc.CoreContainer.<init> New CoreContainer 1512277211
[junit4:junit4]   2> 4151 T1998 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty1-1363193306930/'
[junit4:junit4]   2> 4151 T1998 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty1-1363193306930/'
[junit4:junit4]   2> 4202 T1998 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 4202 T1998 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 4203 T1998 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 4203 T1998 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 4204 T1998 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 4204 T1998 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 4204 T1998 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 4205 T1998 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 4205 T1998 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 4206 T1998 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 4225 T1998 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 4243 T1998 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:61439/solr
[junit4:junit4]   2> 4244 T1998 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 4244 T1998 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:61439 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@3276fdaa
[junit4:junit4]   2> 4245 T1998 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 4246 T2037 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:61439. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 4246 T2037 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:61439, initiating session
[junit4:junit4]   2> 4247 T2000 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:61447
[junit4:junit4]   2> 4247 T2000 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:61447
[junit4:junit4]   2> 4249 T2002 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d64a70fc40005 with negotiated timeout 20000 for client /140.211.11.196:61447
[junit4:junit4]   2> 4249 T2037 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:61439, sessionid = 0x13d64a70fc40005, negotiated timeout = 20000
[junit4:junit4]   2> 4249 T2038 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3276fdaa name:ZooKeeperConnection Watcher:127.0.0.1:61439 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 4249 T1998 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 4251 T2003 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d64a70fc40005
[junit4:junit4]   2> 4252 T2038 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 4252 T2000 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:61447 which had sessionid 0x13d64a70fc40005
[junit4:junit4]   2> 4252 T1998 oaz.ZooKeeper.close Session: 0x13d64a70fc40005 closed
[junit4:junit4]   2> 4253 T1998 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 4262 T1998 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:61439/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@7184674c
[junit4:junit4]   2> 4263 T1998 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 4263 T2039 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:61439. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 4264 T2039 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:61439, initiating session
[junit4:junit4]   2> 4264 T2000 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:61448
[junit4:junit4]   2> 4264 T2000 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:61448
[junit4:junit4]   2> 4265 T2002 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d64a70fc40006 with negotiated timeout 20000 for client /140.211.11.196:61448
[junit4:junit4]   2> 4265 T2039 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:61439, sessionid = 0x13d64a70fc40006, negotiated timeout = 20000
[junit4:junit4]   2> 4266 T2040 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7184674c name:ZooKeeperConnection Watcher:127.0.0.1:61439/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 4266 T1998 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 4267 T2003 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d64a70fc40006 type:create cxid:0x1 zxid:0x4a txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 4268 T2003 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d64a70fc40006 type:create cxid:0x2 zxid:0x4b txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 4277 T1998 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 5279 T1998 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:61446_
[junit4:junit4]   2> 5280 T2003 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d64a70fc40006 type:delete cxid:0xb zxid:0x4c txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:61446_ Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:61446_
[junit4:junit4]   2> 5282 T1998 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:61446_
[junit4:junit4]   2> 5285 T2020 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 5285 T2027 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 5285 T2040 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 5286 T2020 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 5299 T2041 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty1-1363193306930/collection1
[junit4:junit4]   2> 5299 T2041 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 5300 T2041 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 5300 T2041 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 5301 T2041 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty1-1363193306930/collection1/'
[junit4:junit4]   2> 5302 T2041 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-jetty1-1363193306930/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 5303 T2041 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-jetty1-1363193306930/collection1/lib/README' to classloader
[junit4:junit4]   2> 5360 T2041 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 5418 T2041 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 5431 T2021 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 5432 T2021 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"control_collection",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:61442_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:61442"}
[junit4:junit4]   2> 5436 T2027 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 5436 T2020 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 5519 T2040 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 5519 T2041 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 5525 T2041 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 6146 T2041 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 6158 T2041 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 6162 T2041 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 6174 T2041 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 6179 T2041 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 6184 T2041 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 6185 T2041 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 6185 T2041 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 6185 T2041 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 6186 T2041 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 6186 T2041 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 6187 T2041 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 6187 T2041 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty1-1363193306930/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1363193302973/jetty1/
[junit4:junit4]   2> 6187 T2041 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3dbe8711
[junit4:junit4]   2> 6188 T2041 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 6188 T2041 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363193302973/jetty1 forceNew: false
[junit4:junit4]   2> 6189 T2041 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1363193302973/jetty1/index/
[junit4:junit4]   2> 6189 T2041 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1363193302973/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 6189 T2041 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363193302973/jetty1/index forceNew: false
[junit4:junit4]   2> 6192 T2041 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1a3f9f3b lockFactory=org.apache.lucene.store.NativeFSLockFactory@6ebe26e6),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 6192 T2041 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 6195 T2041 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 6195 T2041 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 6196 T2041 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 6196 T2041 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 6197 T2041 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 6197 T2041 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 6197 T2041 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 6198 T2041 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 6198 T2041 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 6205 T2041 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 6211 T2041 oass.SolrIndexSearcher.<init> Opening Searcher@7c41500f main
[junit4:junit4]   2> 6211 T2041 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.SyncSliceTest-1363193302973/jetty1/tlog
[junit4:junit4]   2> 6212 T2041 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 6213 T2041 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 6216 T2042 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7c41500f main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 6218 T2041 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 6218 T2041 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 6948 T2021 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 6949 T2021 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:61446_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:61446"}
[junit4:junit4]   2> 6949 T2021 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 1
[junit4:junit4]   2> 6949 T2021 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 6963 T2020 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 6963 T2040 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 6963 T2027 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 7247 T2041 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 7247 T2041 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:61446 collection:collection1 shard:shard1
[junit4:junit4]   2> 7248 T2041 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 7264 T2003 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d64a70fc40006 type:delete cxid:0x48 zxid:0x5c txntype:-1 reqpath:n/a Error Path:/solr/collections/collection1/leaders Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders
[junit4:junit4]   2> 7270 T2041 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 7271 T2003 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d64a70fc40006 type:create cxid:0x49 zxid:0x5d txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 7273 T2041 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 7273 T2041 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 7273 T2041 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:61446/collection1/
[junit4:junit4]   2> 7274 T2041 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 7274 T2041 oasc.SyncStrategy.syncToMe http://127.0.0.1:61446/collection1/ has no replicas
[junit4:junit4]   2> 7274 T2041 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:61446/collection1/
[junit4:junit4]   2> 7274 T2041 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4]   2> 7287 T2003 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d64a70fc40006 type:create cxid:0x53 zxid:0x61 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 8467 T2021 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 8518 T2027 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 8519 T2040 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 8518 T2020 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 8569 T2041 oasc.ZkController.register We are http://127.0.0.1:61446/collection1/ and leader is http://127.0.0.1:61446/collection1/
[junit4:junit4]   2> 8569 T2041 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:61446
[junit4:junit4]   2> 8569 T2041 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 8569 T2041 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 8570 T2041 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 8571 T2041 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 8572 T1998 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 8573 T1998 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 8573 T1998 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 8766 T1998 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 8769 T1998 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:61449
[junit4:junit4]   2> 8770 T1998 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 8770 T1998 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 8771 T1998 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty2-1363193311555
[junit4:junit4]   2> 8771 T1998 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty2-1363193311555/solr.xml
[junit4:junit4]   2> 8772 T1998 oasc.CoreContainer.<init> New CoreContainer 1371750636
[junit4:junit4]   2> 8773 T1998 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty2-1363193311555/'
[junit4:junit4]   2> 8773 T1998 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty2-1363193311555/'
[junit4:junit4]   2> 8823 T1998 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 8824 T1998 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 8824 T1998 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 8825 T1998 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 8825 T1998 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 8825 T1998 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 8826 T1998 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 8826 T1998 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 8827 T1998 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 8827 T1998 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 8848 T1998 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 8866 T1998 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:61439/solr
[junit4:junit4]   2> 8867 T1998 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 8867 T1998 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:61439 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@458ed660
[junit4:junit4]   2> 8868 T1998 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 8868 T2053 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:61439. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 8869 T2053 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:61439, initiating session
[junit4:junit4]   2> 8869 T2000 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:61450
[junit4:junit4]   2> 8870 T2000 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:61450
[junit4:junit4]   2> 8872 T2002 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d64a70fc40007 with negotiated timeout 20000 for client /140.211.11.196:61450
[junit4:junit4]   2> 8872 T2053 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:61439, sessionid = 0x13d64a70fc40007, negotiated timeout = 20000
[junit4:junit4]   2> 8872 T2054 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@458ed660 name:ZooKeeperConnection Watcher:127.0.0.1:61439 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 8873 T1998 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 8874 T2003 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d64a70fc40007
[junit4:junit4]   2> 8886 T2054 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 8887 T2000 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:61450 which had sessionid 0x13d64a70fc40007
[junit4:junit4]   2> 8886 T1998 oaz.ZooKeeper.close Session: 0x13d64a70fc40007 closed
[junit4:junit4]   2> 8887 T1998 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 8898 T1998 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:61439/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@4d4cd792
[junit4:junit4]   2> 8899 T1998 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 8900 T2055 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:61439. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 8900 T2055 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:61439, initiating session
[junit4:junit4]   2> 8900 T2000 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:61451
[junit4:junit4]   2> 8901 T2000 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:61451
[junit4:junit4]   2> 8902 T2002 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d64a70fc40008 with negotiated timeout 20000 for client /140.211.11.196:61451
[junit4:junit4]   2> 8902 T2055 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:61439, sessionid = 0x13d64a70fc40008, negotiated timeout = 20000
[junit4:junit4]   2> 8902 T2056 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4d4cd792 name:ZooKeeperConnection Watcher:127.0.0.1:61439/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 8902 T1998 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 8903 T2003 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d64a70fc40008 type:create cxid:0x1 zxid:0x6e txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 8905 T2003 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d64a70fc40008 type:create cxid:0x2 zxid:0x6f txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 8907 T1998 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 9910 T1998 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:61449_
[junit4:junit4]   2> 9911 T2003 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d64a70fc40008 type:delete cxid:0xb zxid:0x70 txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:61449_ Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:61449_
[junit4:junit4]   2> 9930 T1998 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:61449_
[junit4:junit4]   2> 9933 T2040 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 9933 T2020 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 9933 T2027 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 9934 T2056 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 9935 T2040 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 9936 T2020 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 9936 T2027 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 9941 T2057 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty2-1363193311555/collection1
[junit4:junit4]   2> 9941 T2057 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 9941 T2057 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 9942 T2057 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 9943 T2057 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty2-1363193311555/collection1/'
[junit4:junit4]   2> 9944 T2057 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-jetty2-1363193311555/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 9944 T2057 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-jetty2-1363193311555/collection1/lib/README' to classloader
[junit4:junit4]   2> 9992 T2057 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 10032 T2021 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 10033 T2021 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:61446_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:61446"}
[junit4:junit4]   2> 10038 T2056 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 10038 T2040 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 10038 T2027 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 10038 T2020 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 10050 T2057 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 10151 T2057 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 10157 T2057 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 10779 T2057 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 10791 T2057 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 10796 T2057 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 10808 T2057 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 10812 T2057 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 10817 T2057 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 10818 T2057 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 10818 T2057 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 10819 T2057 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 10820 T2057 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 10820 T2057 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 10820 T2057 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 10820 T2057 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty2-1363193311555/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1363193302973/jetty2/
[junit4:junit4]   2> 10821 T2057 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3dbe8711
[junit4:junit4]   2> 10821 T2057 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 10822 T2057 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363193302973/jetty2 forceNew: false
[junit4:junit4]   2> 10822 T2057 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1363193302973/jetty2/index/
[junit4:junit4]   2> 10822 T2057 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1363193302973/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 10823 T2057 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363193302973/jetty2/index forceNew: false
[junit4:junit4]   2> 10825 T2057 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3ce15725 lockFactory=org.apache.lucene.store.NativeFSLockFactory@ed17b80),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 10826 T2057 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 10828 T2057 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 10828 T2057 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 10829 T2057 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 10830 T2057 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 10830 T2057 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 10830 T2057 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 10831 T2057 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 10831 T2057 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 10832 T2057 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 10839 T2057 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 10844 T2057 oass.SolrIndexSearcher.<init> Opening Searcher@100a28c7 main
[junit4:junit4]   2> 10845 T2057 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.SyncSliceTest-1363193302973/jetty2/tlog
[junit4:junit4]   2> 10846 T2057 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 10846 T2057 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 10850 T2058 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@100a28c7 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 10852 T2057 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 10852 T2057 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 11542 T2021 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 11543 T2021 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:61449_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:61449"}
[junit4:junit4]   2> 11543 T2021 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4]   2> 11543 T2021 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 11583 T2056 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 11584 T2020 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 11583 T2040 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 11583 T2027 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 11882 T2057 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 11882 T2057 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:61449 collection:collection1 shard:shard1
[junit4:junit4]   2> 11885 T2057 oasc.ZkController.register We are http://127.0.0.1:61449/collection1/ and leader is http://127.0.0.1:61446/collection1/
[junit4:junit4]   2> 11886 T2057 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:61449
[junit4:junit4]   2> 11886 T2057 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 11886 T2057 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C209 name=collection1 org.apache.solr.core.SolrCore@1d0a7f35 url=http://127.0.0.1:61449/collection1 node=127.0.0.1:61449_ C209_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:61449_, base_url=http://127.0.0.1:61449}
[junit4:junit4]   2> 11888 T2059 C209 P61449 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 11888 T2059 C209 P61449 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 11888 T2057 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 11888 T2059 C209 P61449 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4]   2> 11888 T2059 C209 P61449 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 11889 T1998 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 11890 T2059 C209 P61449 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 11890 T1998 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 11890 T1998 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 12090 T1998 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 12093 T1998 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:61453
[junit4:junit4]   2> 12094 T1998 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 12094 T1998 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 12095 T1998 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty3-1363193314873
[junit4:junit4]   2> 12095 T1998 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty3-1363193314873/solr.xml
[junit4:junit4]   2> 12095 T1998 oasc.CoreContainer.<init> New CoreContainer 2059951029
[junit4:junit4]   2> 12096 T1998 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty3-1363193314873/'
[junit4:junit4]   2> 12097 T1998 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty3-1363193314873/'
[junit4:junit4]   2> 12147 T1998 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 12148 T1998 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 12149 T1998 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 12149 T1998 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 12149 T1998 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 12150 T1998 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 12150 T1998 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 12151 T1998 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 12151 T1998 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 12152 T1998 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 12171 T1998 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 12189 T1998 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:61439/solr
[junit4:junit4]   2> 12190 T1998 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 12190 T1998 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:61439 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@4d9e25f
[junit4:junit4]   2> 12191 T1998 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 12192 T2070 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:61439. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 12193 T2070 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:61439, initiating session
[junit4:junit4]   2> 12193 T2000 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:61454
[junit4:junit4]   2> 12193 T2000 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:61454
[junit4:junit4]   2> 12195 T2002 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d64a70fc40009 with negotiated timeout 20000 for client /140.211.11.196:61454
[junit4:junit4]   2> 12195 T2070 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:61439, sessionid = 0x13d64a70fc40009, negotiated timeout = 20000
[junit4:junit4]   2> 12195 T2071 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4d9e25f name:ZooKeeperConnection Watcher:127.0.0.1:61439 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 12195 T1998 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 12197 T2003 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d64a70fc40009
[junit4:junit4]   2> 12198 T2071 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 12198 T2000 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:61454 which had sessionid 0x13d64a70fc40009
[junit4:junit4]   2> 12198 T1998 oaz.ZooKeeper.close Session: 0x13d64a70fc40009 closed
[junit4:junit4]   2> 12198 T1998 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 12208 T1998 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:61439/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@53ef7ba0
[junit4:junit4]   2> 12209 T1998 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 12209 T2072 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:61439. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 12209 T2072 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:61439, initiating session
[junit4:junit4]   2> 12210 T2000 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:61455
[junit4:junit4]   2> 12210 T2000 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:61455
[junit4:junit4]   2> 12211 T2002 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d64a70fc4000a with negotiated timeout 20000 for client /140.211.11.196:61455
[junit4:junit4]   2> 12211 T2072 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:61439, sessionid = 0x13d64a70fc4000a, negotiated timeout = 20000
[junit4:junit4]   2> 12211 T2073 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@53ef7ba0 name:ZooKeeperConnection Watcher:127.0.0.1:61439/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 12211 T1998 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 12213 T2003 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d64a70fc4000a type:create cxid:0x1 zxid:0x81 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 12214 T2003 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d64a70fc4000a type:create cxid:0x2 zxid:0x82 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 12216 T1998 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 13088 T2021 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 13089 T2021 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:61449_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:61449"}
[junit4:junit4]   2> 13093 T2073 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 13094 T2020 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 13094 T2027 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 13094 T2040 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 13094 T2056 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 13218 T1998 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:61453_
[junit4:junit4]   2> 13219 T2003 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d64a70fc4000a type:delete cxid:0xd zxid:0x87 txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:61453_ Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:61453_
[junit4:junit4]   2> 13220 T1998 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:61453_
[junit4:junit4]   2> 13222 T2056 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 13222 T2020 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 13222 T2040 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 13223 T2073 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 13222 T2027 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 13223 T2073 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 13224 T2056 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 13225 T2020 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 13225 T2027 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 13225 T2040 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 13230 T2074 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty3-1363193314873/collection1
[junit4:junit4]   2> 13230 T2074 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 13231 T2074 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 13231 T2074 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 13233 T2074 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty3-1363193314873/collection1/'
[junit4:junit4]   2> 13234 T2074 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-jetty3-1363193314873/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 13234 T2074 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-jetty3-1363193314873/collection1/lib/README' to classloader
[junit4:junit4]   2> 13288 T2074 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 13350 T2074 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 13451 T2074 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 13457 T2074 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 13908 T2030 oass.SolrDispatchFilter.handleAdminRequest /admin/cores {coreNodeName=127.0.0.1:61449__collection1&state=recovering&nodeName=127.0.0.1:61449_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=2002 
[junit4:junit4]   2> 14084 T2074 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 14096 T2074 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 14100 T2074 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 14112 T2074 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 14117 T2074 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 14121 T2074 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 14122 T2074 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 14122 T2074 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 14123 T2074 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 14123 T2074 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 14124 T2074 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 14124 T2074 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 14124 T2074 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty3-1363193314873/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1363193302973/jetty3/
[junit4:junit4]   2> 14125 T2074 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3dbe8711
[junit4:junit4]   2> 14125 T2074 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 14126 T2074 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363193302973/jetty3 forceNew: false
[junit4:junit4]   2> 14126 T2074 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1363193302973/jetty3/index/
[junit4:junit4]   2> 14126 T2074 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1363193302973/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 14127 T2074 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363193302973/jetty3/index forceNew: false
[junit4:junit4]   2> 14129 T2074 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@65d5a68d lockFactory=org.apache.lucene.store.NativeFSLockFactory@5d20e1a5),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 14130 T2074 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 14132 T2074 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 14132 T2074 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 14133 T2074 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 14134 T2074 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 14134 T2074 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 14134 T2074 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 14135 T2074 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 14135 T2074 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 14136 T2074 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 14143 T2074 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 14148 T2074 oass.SolrIndexSearcher.<init> Opening Searcher@3d67fcee main
[junit4:junit4]   2> 14149 T2074 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.SyncSliceTest-1363193302973/jetty3/tlog
[junit4:junit4]   2> 14150 T2074 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 14150 T2074 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 14154 T2075 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3d67fcee main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 14155 T2074 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 14155 T2074 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 14599 T2021 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 14600 T2021 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:61453_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:61453"}
[junit4:junit4]   2> 14600 T2021 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4]   2> 14601 T2021 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 14656 T2073 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 14656 T2056 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 14656 T2020 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 14656 T2027 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 14656 T2040 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 15184 T2074 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 15184 T2074 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:61453 collection:collection1 shard:shard1
[junit4:junit4]   2> 15188 T2074 oasc.ZkController.register We are http://127.0.0.1:61453/collection1/ and leader is http://127.0.0.1:61446/collection1/
[junit4:junit4]   2> 15188 T2074 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:61453
[junit4:junit4]   2> 15188 T2074 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 15189 T2074 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C210 name=collection1 org.apache.solr.core.SolrCore@4afedaa8 url=http://127.0.0.1:61453/collection1 node=127.0.0.1:61453_ C210_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:61453_, base_url=http://127.0.0.1:61453}
[junit4:junit4]   2> 15190 T2076 C210 P61453 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 15190 T2074 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 15190 T2076 C210 P61453 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 15191 T2076 C210 P61453 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4]   2> 15191 T2076 C210 P61453 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 15191 T1998 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 15192 T1998 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 15192 T2076 C210 P61453 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 15192 T1998 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 15397 T1998 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 15400 T1998 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:27180
[junit4:junit4]   2> 15400 T1998 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 15401 T1998 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 15401 T1998 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty4-1363193318180
[junit4:junit4]   2> 15402 T1998 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty4-1363193318180/solr.xml
[junit4:junit4]   2> 15402 T1998 oasc.CoreContainer.<init> New CoreContainer 1201471002
[junit4:junit4]   2> 15403 T1998 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty4-1363193318180/'
[junit4:junit4]   2> 15404 T1998 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty4-1363193318180/'
[junit4:junit4]   2> 15454 T1998 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 15454 T1998 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 15455 T1998 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 15455 T1998 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 15456 T1998 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 15456 T1998 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 15457 T1998 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 15457 T1998 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 15457 T1998 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 15458 T1998 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 15478 T1998 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 15497 T1998 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:61439/solr
[junit4:junit4]   2> 15497 T1998 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 15498 T1998 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:61439 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@f8495b0
[junit4:junit4]   2> 15499 T1998 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 15499 T2087 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:61439. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 15500 T2087 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:61439, initiating session
[junit4:junit4]   2> 15500 T2000 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:61571
[junit4:junit4]   2> 15501 T2000 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:61571
[junit4:junit4]   2> 15502 T2002 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d64a70fc4000b with negotiated timeout 20000 for client /140.211.11.196:61571
[junit4:junit4]   2> 15502 T2087 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:61439, sessionid = 0x13d64a70fc4000b, negotiated timeout = 20000
[junit4:junit4]   2> 15503 T2088 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f8495b0 name:ZooKeeperConnection Watcher:127.0.0.1:61439 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 15503 T1998 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 15504 T2003 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d64a70fc4000b
[junit4:junit4]   2> 15514 T2088 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 15514 T2000 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:61571 which had sessionid 0x13d64a70fc4000b
[junit4:junit4]   2> 15514 T1998 oaz.ZooKeeper.close Session: 0x13d64a70fc4000b closed
[junit4:junit4]   2> 15515 T1998 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 15526 T1998 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:61439/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@236a6060
[junit4:junit4]   2> 15526 T1998 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 15527 T2089 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:61439. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 15527 T2089 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:61439, initiating session
[junit4:junit4]   2> 15527 T2000 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:16981
[junit4:junit4]   2> 15528 T2000 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:16981
[junit4:junit4]   2> 15529 T2002 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d64a70fc4000c with negotiated timeout 20000 for client /140.211.11.196:16981
[junit4:junit4]   2> 15529 T2089 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:61439, sessionid = 0x13d64a70fc4000c, negotiated timeout = 20000
[junit4:junit4]   2> 15529 T2090 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@236a6060 name:ZooKeeperConnection Watcher:127.0.0.1:61439/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 15529 T1998 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 15531 T2003 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d64a70fc4000c type:create cxid:0x1 zxid:0x94 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 15532 T2003 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d64a70fc4000c type:create cxid:0x2 zxid:0x95 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 15534 T1998 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2>  C209_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:61449_, base_url=http://127.0.0.1:61449}
[junit4:junit4]   2> 15910 T2059 C209 P61449 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:61446/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4]   2> 15910 T2059 C209 P61449 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:61449 START replicas=[http://127.0.0.1:61446/collection1/] nUpdates=100
[junit4:junit4]   2> 15910 T2059 C209 P61449 oasu.PeerSync.sync WARNING no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 15911 T2059 C209 P61449 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4]   2> 15911 T2059 C209 P61449 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4]   2> 15912 T2059 C209 P61449 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4]   2> 15912 T2059 C209 P61449 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4]   2> 15912 T2059 C209 P61449 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:61446/collection1/. core=collection1
[junit4:junit4]   2> 15912 T2059 C209 P61449 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C211 name=collection1 org.apache.solr.core.SolrCore@144aed05 url=http://127.0.0.1:61446/collection1 node=127.0.0.1:61446_ C211_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:61446_, base_url=http://127.0.0.1:61446, leader=true}
[junit4:junit4]   2> 15918 T2032 C211 P61446 REQ /get {getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0 
[junit4:junit4]   2> 15924 T2033 C211 P61446 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 15926 T2033 C211 P61446 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1a3f9f3b lockFactory=org.apache.lucene.store.NativeFSLockFactory@6ebe26e6),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 15926 T2033 C211 P61446 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 15927 T2033 C211 P61446 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1a3f9f3b lockFactory=org.apache.lucene.store.NativeFSLockFactory@6ebe26e6),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1a3f9f3b lockFactory=org.apache.lucene.store.NativeFSLockFactory@6ebe26e6),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 15927 T2033 C211 P61446 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[segments_2]
[junit4:junit4]   2> 15928 T2033 C211 P61446 oass.SolrIndexSearcher.<init> Opening Searcher@69ab6e55 realtime
[junit4:junit4]   2> 15928 T2033 C211 P61446 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 15928 T2033 C211 P61446 /update {waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 4
[junit4:junit4]   2> 15929 T2059 C209 P61449 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 15929 T2059 C209 P61449 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4]   2> 15931 T2034 C211 P61446 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 15932 T2034 C211 P61446 REQ /replication {command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 15932 T2059 C209 P61449 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4]   2> 15933 T2059 C209 P61449 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4]   2> 15933 T2059 C209 P61449 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4]   2> 15935 T2034 C211 P61446 REQ /replication {command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0 
[junit4:junit4]   2> 15936 T2059 C209 P61449 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4]   2> 15937 T2059 C209 P61449 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363193302973/jetty2/index.20130313164838904 forceNew: false
[junit4:junit4]   2> 15937 T2059 C209 P61449 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@303bd31 lockFactory=org.apache.lucene.store.NativeFSLockFactory@66bdb86a)) fullCopy=false
[junit4:junit4]   2> 15939 T2034 C211 P61446 REQ /replication {file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> 15940 T2059 C209 P61449 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4]   2> 15941 T2059 C209 P61449 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4]   2> 15941 T2059 C209 P61449 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4]   2> 15943 T2059 C209 P61449 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3ce15725 lockFactory=org.apache.lucene.store.NativeFSLockFactory@ed17b80),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3ce15725 lockFactory=org.apache.lucene.store.NativeFSLockFactory@ed17b80),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 15943 T2059 C209 P61449 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[segments_2]
[junit4:junit4]   2> 15943 T2059 C209 P61449 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4]   2> 15944 T2059 C209 P61449 oass.SolrIndexSearcher.<init> Opening Searcher@3e9a1f6b main
[junit4:junit4]   2> 15944 T2058 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3e9a1f6b main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 15945 T2059 C209 P61449 oash.SnapPuller.fetchLatestIndex removing temporary index download directory files MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@303bd31 lockFactory=org.apache.lucene.store.NativeFSLockFactory@66bdb86a))
[junit4:junit4]   2> 15945 T2059 C209 P61449 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4]   2> 15945 T2059 C209 P61449 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4]   2> 15945 T2059 C209 P61449 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 15946 T2059 C209 P61449 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 15947 T2059 C209 P61449 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4]   2> 16161 T2021 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 16162 T2021 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:61453_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:61453"}
[junit4:junit4]   2> 16165 T2021 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:61449_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:61449"}
[junit4:junit4]   2> 16178 T2056 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 16178 T2090 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 16178 T2027 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 16178 T2020 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 16178 T2073 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 16178 T2040 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 16215 T2031 oass.SolrDispatchFilter.handleAdminRequest /admin/cores {coreNodeName=127.0.0.1:61453__collection1&state=recovering&nodeName=127.0.0.1:61453_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001 
[junit4:junit4]   2> 16537 T1998 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:27180_
[junit4:junit4]   2> 16538 T2003 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d64a70fc4000c type:delete cxid:0xd zxid:0x9e txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:27180_ Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:27180_
[junit4:junit4]   2> 16539 T1998 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:27180_
[junit4:junit4]   2> 16541 T2040 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 16541 T2056 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 16541 T2020 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 16541 T2027 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 16542 T2073 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 16542 T2090 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 16542 T2090 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 16542 T2073 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 16543 T2040 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 16543 T2020 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 16552 T2027 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 16553 T2056 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 16557 T2093 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty4-1363193318180/collection1
[junit4:junit4]   2> 16558 T2093 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 16558 T2093 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 16558 T2093 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 16560 T2093 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty4-1363193318180/collection1/'
[junit4:junit4]   2> 16561 T2093 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-jetty4-1363193318180/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 16561 T2093 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-jetty4-1363193318180/collection1/lib/README' to classloader
[junit4:junit4]   2> 16609 T2093 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 16667 T2093 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 16767 T2093 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 16773 T2093 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 17401 T2093 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 17409 T2093 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 17412 T2093 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 17421 T2093 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 17424 T2093 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 17427 T2093 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 17428 T2093 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 17429 T2093 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 17429 T2093 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 17430 T2093 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 17430 T2093 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 17430 T2093 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 17431 T2093 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty4-1363193318180/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1363193302973/jetty4/
[junit4:junit4]   2> 17431 T2093 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3dbe8711
[junit4:junit4]   2> 17431 T2093 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 17432 T2093 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363193302973/jetty4 forceNew: false
[junit4:junit4]   2> 17432 T2093 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1363193302973/jetty4/index/
[junit4:junit4]   2> 17432 T2093 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1363193302973/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 17432 T2093 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363193302973/jetty4/index forceNew: false
[junit4:junit4]   2> 17434 T2093 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@34059959 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5414b45),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 17434 T2093 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 17436 T2093 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 17436 T2093 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 17437 T2093 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 17437 T2093 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 17437 T2093 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 17438 T2093 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 17438 T2093 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 17438 T2093 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 17438 T2093 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 17443 T2093 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 17448 T2093 oass.SolrIndexSearcher.<init> Opening Searcher@1437b12e main
[junit4:junit4]   2> 17448 T2093 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.SyncSliceTest-1363193302973/jetty4/tlog
[junit4:junit4]   2> 17448 T2093 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 17449 T2093 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 17452 T2094 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1437b12e main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 17453 T2093 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 17453 T2093 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 17691 T2021 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 17692 T2021 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:27180_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:27180"}
[junit4:junit4]   2> 17692 T2021 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4]   2> 17692 T2021 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 17705 T2090 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 17705 T2020 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 17705 T2040 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 17705 T2027 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 17705 T2073 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 17705 T2056 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2>  C210_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:61453_, base_url=http://127.0.0.1:61453}
[junit4:junit4]   2> 18216 T2076 C210 P61453 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:61446/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4]   2> 18216 T2076 C210 P61453 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:61453 START replicas=[http://127.0.0.1:61446/collection1/] nUpdates=100
[junit4:junit4]   2> 18217 T2076 C210 P61453 oasu.PeerSync.sync WARNING no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 18218 T2076 C210 P61453 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4]   2> 18219 T2076 C210 P61453 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4]   2> 18219 T2076 C210 P61453 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4]   2> 18219 T2076 C210 P61453 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4]   2> 18219 T2076 C210 P61453 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:61446/collection1/. core=collection1
[junit4:junit4]   2> 18219 T2076 C210 P61453 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 18221 T2032 C211 P61446 REQ /get {getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0 
[junit4:junit4]   2> 18246 T2035 C211 P61446 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 18249 T2035 C211 P61446 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1a3f9f3b lockFactory=org.apache.lucene.store.NativeFSLockFactory@6ebe26e6),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1a3f9f3b lockFactory=org.apache.lucene.store.NativeFSLockFactory@6ebe26e6),segFN=segments_3,generation=3,filenames=[segments_3]
[junit4:junit4]   2> 18249 T2035 C211 P61446 oasc.SolrDeletionPolicy.updateCommits newest commit = 3[segments_3]
[junit4:junit4]   2> 18250 T2035 C211 P61446 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 18250 T2035 C211 P61446 /update {waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 4
[junit4:junit4]   2> 18251 T2076 C210 P61453 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 18251 T2076 C210 P61453 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4]   2> 18255 T2034 C211 P61446 REQ /replication {command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 18256 T2076 C210 P61453 oash.SnapPuller.fetchLatestIndex Master's generation: 3
[junit4:junit4]   2> 18256 T2076 C210 P61453 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4]   2> 18257 T2076 C210 P61453 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4]   2> 18259 T2034 C211 P61446 REQ /replication {command=filelist&qt=/replication&wt=javabin&generation=3&version=2} status=0 QTime=0 
[junit4:junit4]   2> 18261 T2076 C210 P61453 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4]   2> 18261 T2076 C210 P61453 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363193302973/jetty3/index.20130313164841229 forceNew: false
[junit4:junit4]   2> 18262 T2076 C210 P61453 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@3caa3588 lockFactory=org.apache.lucene.store.NativeFSLockFactory@74080439)) fullCopy=false
[junit4:junit4]   2> 18264 T2034 C211 P61446 REQ /replication {file=segments_3&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=3} status=0 QTime=0 
[junit4:junit4]   2> 18266 T2076 C210 P61453 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4]   2> 18266 T2076 C210 P61453 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4]   2> 18267 T2076 C210 P61453 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4]   2> 18269 T2076 C210 P61453 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@65d5a68d lockFactory=org.apache.lucene.store.NativeFSLockFactory@5d20e1a5),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@65d5a68d lockFactory=org.apache.lucene.store.NativeFSLockFactory@5d20e1a5),segFN=segments_3,generation=3,filenames=[segments_3]
[junit4:junit4]   2> 18270 T2076 C210 P61453 oasc.SolrDeletionPolicy.updateCommits newest commit = 3[segments_3]
[junit4:junit4]   2> 18270 T2076 C210 P61453 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4]   2> 18270 T2076 C210 P61453 oass.SolrIndexSearcher.<init> Opening Searcher@7ce534f main
[junit4:junit4]   2> 18272 T2075 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7ce534f main{StandardDirectoryReader(segments_3:1:nrt)}
[junit4:junit4]   2> 18273 T2076 C210 P61453 oash.SnapPuller.fetchLatestIndex removing temporary index download directory files MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@3caa3588 lockFactory=org.apache.lucene.store.NativeFSLockFactory@74080439))
[junit4:junit4]   2> 18273 T2076 C210 P61453 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4]   2> 18274 T2076 C210 P61453 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4]   2> 18274 T2076 C210 P61453 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 18274 T2076 C210 P61453 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 18277 T2076 C210 P61453 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4]   2> 18481 T2093 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 18481 T2093 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:27180 collection:collection1 shard:shard1
[junit4:junit4]   2> 18504 T2093 oasc.ZkController.register We are http://127.0.0.1:27180/collection1/ and leader is http://127.0.0.1:61446/collection1/
[junit4:junit4]   2> 18504 T2093 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:27180
[junit4:junit4]   2> 18504 T2093 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 18504 T2093 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C212 name=collection1 org.apache.solr.core.SolrCore@a076827 url=http://127.0.0.1:27180/collection1 node=127.0.0.1:27180_ C212_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:27180_, base_url=http://127.0.0.1:27180}
[junit4:junit4]   2> 18509 T2097 C212 P27180 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 18509 T2097 C212 P27180 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 18510 T2097 C212 P27180 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4]   2> 18510 T2097 C212 P27180 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 18514 T2093 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 18523 T2097 C212 P27180 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 18527 T1998 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 18528 T1998 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 18531 T1998 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 18576 T1998 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 18583 T1998 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 15 for each attempt
[junit4:junit4]   2> 18585 T1998 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):15
[junit4:junit4]   2> 18592 T1998 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 19212 T2021 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 19214 T2021 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:61453_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:61453"}
[junit4:junit4]   2> 19224 T2021 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:27180_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:27180"}
[junit4:junit4]   2> 19238 T2040 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 19238 T2056 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 19238 T2073 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 19238 T2027 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 19238 T2020 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 19238 T2090 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 19571 T2036 oass.SolrDispatchFilter.handleAdminRequest /admin/cores {coreNodeName=127.0.0.1:27180__collection1&state=recovering&nodeName=127.0.0.1:27180_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001 
[junit4:junit4]   2> 19595 T1998 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 20600 T1998 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2>  C212_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:27180_, base_url=http://127.0.0.1:27180}
[junit4:junit4]   2> 21573 T2097 C212 P27180 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:61446/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4]   2> 21573 T2097 C212 P27180 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:27180 START replicas=[http://127.0.0.1:61446/collection1/] nUpdates=100
[junit4:junit4]   2> 21574 T2097 C212 P27180 oasu.PeerSync.sync WARNING no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 21574 T2097 C212 P27180 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4]   2> 21574 T2097 C212 P27180 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4]   2> 21574 T2097 C212 P27180 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4]   2> 21574 T2097 C212 P27180 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4]   2> 21574 T2097 C212 P27180 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:61446/collection1/. core=collection1
[junit4:junit4]   2> 21575 T2097 C212 P27180 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 21576 T2032 C211 P61446 REQ /get {getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0 
[junit4:junit4]   2> 21584 T2030 C211 P61446 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 21585 T2030 C211 P61446 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=3
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1a3f9f3b lockFactory=org.apache.lucene.store.NativeFSLockFactory@6ebe26e6),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1a3f9f3b lockFactory=org.apache.lucene.store.NativeFSLockFactory@6ebe26e6),segFN=segments_3,generation=3,filenames=[segments_3]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1a3f9f3b lockFactory=org.apache.lucene.store.NativeFSLockFactory@6ebe26e6),segFN=segments_4,generation=4,filenames=[segments_4]
[junit4:junit4]   2> 21585 T2030 C211 P61446 oasc.SolrDeletionPolicy.updateCommits newest commit = 4[segments_4]
[junit4:junit4]   2> 21585 T2030 C211 P61446 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 21586 T2030 C211 P61446 /update {waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 2
[junit4:junit4]   2> 21587 T2097 C212 P27180 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 21587 T2097 C212 P27180 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4]   2> 21589 T2034 C211 P61446 REQ /replication {command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 21591 T2097 C212 P27180 oash.SnapPuller.fetchLatestIndex Master's generation: 4
[junit4:junit4]   2> 21591 T2097 C212 P27180 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4]   2> 21591 T2097 C212 P27180 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4]   2> 21593 T2034 C211 P61446 REQ /replication {command=filelist&qt=/replication&wt=javabin&generation=4&version=2} status=0 QTime=0 
[junit4:junit4]   2> 21594 T2097 C212 P27180 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4]   2> 21595 T2097 C212 P27180 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363193302973/jetty4/index.20130313164844563 forceNew: false
[junit4:junit4]   2> 21595 T2097 C212 P27180 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@240581ad lockFactory=org.apache.lucene.store.NativeFSLockFactory@63eb3fdd)) fullCopy=false
[junit4:junit4]   2> 21598 T2034 C211 P61446 REQ /replication {file=segments_4&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=4} status=0 QTime=0 
[junit4:junit4]   2> 21600 T2097 C212 P27180 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4]   2> 21600 T2097 C212 P27180 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4]   2> 21600 T2097 C212 P27180 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4]   2> 21602 T2097 C212 P27180 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@34059959 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5414b45),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@34059959 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5414b45),segFN=segments_4,generation=4,filenames=[segments_4]
[junit4:junit4]   2> 21602 T2097 C212 P27180 oasc.SolrDeletionPolicy.updateCommits newest commit = 4[segments_4]
[junit4:junit4]   2> 21602 T2097 C212 P27180 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4]   2> 21602 T1998 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 21603 T2097 C212 P27180 oass.SolrIndexSearcher.<init> Opening Searcher@3774ef26 main
[junit4:junit4]   2> 21604 T2094 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3774ef26 main{StandardDirectoryReader(segments_4:1:nrt)}
[junit4:junit4]   2> 21604 T2097 C212 P27180 oash.SnapPuller.fetchLatestIndex removing temporary index download directory files MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@240581ad lockFactory=org.apache.lucene.store.NativeFSLockFactory@63eb3fdd))
[junit4:junit4]   2> 21604 T2097 C212 P27180 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4]   2> 21604 T2097 C212 P27180 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4]   2> 21604 T2097 C212 P27180 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 21605 T2097 C212 P27180 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 21607 T2097 C212 P27180 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4]   2> 22253 T2021 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 22255 T2021 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:27180_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:27180"}
[junit4:junit4]   2> 22262 T2073 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 22262 T2056 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 22263 T2020 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 22262 T2090 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 22262 T2027 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 22262 T2040 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 22605 T1998 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 22607 T1998 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4]   2> ASYNC  NEW_CORE C213 name=collection1 org.apache.solr.core.SolrCore@543fded7 url=http://127.0.0.1:61442/collection1 node=127.0.0.1:61442_ C213_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:61442_, base_url=http://127.0.0.1:61442, leader=true}
[junit4:junit4]   2> 22615 T2010 C213 P61442 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 22618 T2010 C213 P61442 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@60c481cc lockFactory=org.apache.lucene.store.NativeFSLockFactory@6ba8ee5d),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 22618 T2010 C213 P61442 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 22619 T2010 C213 P61442 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@60c481cc lockFactory=org.apache.lucene.store.NativeFSLockFactory@6ba8ee5d),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@60c481cc lockFactory=org.apache.lucene.store.NativeFSLockFactory@6ba8ee5d),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 22619 T2010 C213 P61442 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[segments_2]
[junit4:junit4]   2> 22620 T2010 C213 P61442 oass.SolrIndexSearcher.<init> Opening Searcher@11166e59 main
[junit4:junit4]   2> 22620 T2010 C213 P61442 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 22621 T2024 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@11166e59 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 22622 T2010 C213 P61442 /update {waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 7
[junit4:junit4]   2> 22635 T2033 C211 P61446 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 22636 T2033 C211 P61446 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=4
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1a3f9f3b lockFactory=org.apache.lucene.store.NativeFSLockFactory@6ebe26e6),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1a3f9f3b lockFactory=org.apache.lucene.store.NativeFSLockFactory@6ebe26e6),segFN=segments_3,generation=3,filenames=[segments_3]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1a3f9f3b lockFactory=org.apache.lucene.store.NativeFSLockFactory@6ebe26e6),segFN=segments_4,generation=4,filenames=[segments_4]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1a3f9f3b lockFactory=org.apache.lucene.store.NativeFSLockFactory@6ebe26e6),segFN=segments_5,generation=5,filenames=[segments_5]
[junit4:junit4]   2> 22637 T2033 C211 P61446 oasc.SolrDeletionPolicy.updateCommits newest commit = 5[segments_5]
[junit4:junit4]   2> 22638 T2033 C211 P61446 oass.SolrIndexSearcher.<init> Opening Searcher@4951d1ed main
[junit4:junit4]   2> 22639 T2033 C211 P61446 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 22640 T2042 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4951d1ed main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 22641 T2033 C211 P61446 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:61449/collection1/, StdNode: http://127.0.0.1:61453/collection1/, StdNode: http://127.0.0.1:27180/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4]   2>  C209_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:61449_, base_url=http://127.0.0.1:61449}
[junit4:junit4]   2> 22654 T2046 C209 P61449 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2>  C212_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:27180_, base_url=http://127.0.0.1:27180}
[junit4:junit4]   2> 22654 T2080 C212 P27180 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 22656 T2080 C212 P27180 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@34059959 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5414b45),segFN=segments_4,generation=4,filenames=[segments_4]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@34059959 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5414b45),segFN=segments_5,generation=5,filenames=[segments_5]
[junit4:junit4]   2> 22657 T2080 C212 P27180 oasc.SolrDeletionPolicy.updateCommits newest commit = 5[segments_5]
[junit4:junit4]   2> 22656 T2046 C209 P61449 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3ce15725 lockFactory=org.apache.lucene.store.NativeFSLockFactory@ed17b80),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3ce15725 lockFactory=org.apache.lucene.store.NativeFSLockFactory@ed17b80),segFN=segments_3,generation=3,filenames=[segments_3]
[junit4:junit4]   2>  C210_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:61453_, base_url=http://127.0.0.1:61453}
[junit4:junit4]   2> 22658 T2063 C210 P61453 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 22658 T2046 C209 P61449 oasc.SolrDeletionPolicy.updateCommits newest commit = 3[segments_3]
[junit4:junit4]   2> 22658 T2080 C212 P27180 oass.SolrIndexSearcher.<init> Opening Searcher@f6d4253 main
[junit4:junit4]   2> 22659 T2046 C209 P61449 oass.SolrIndexSearcher.<init> Opening Searcher@36000c53 main
[junit4:junit4]   2> 22659 T2080 C212 P27180 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 22659 T2063 C210 P61453 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@65d5a68d lockFactory=org.apache.lucene.store.NativeFSLockFactory@5d20e1a5),segFN=segments_3,generation=3,filenames=[segments_3]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@65d5a68d lockFactory=org.apache.lucene.store.NativeFSLockFactory@5d20e1a5),segFN=segments_4,generation=4,filenames=[segments_4]
[junit4:junit4]   2> 22660 T2058 oasc.JmxMonitoredMap.put WARNING Failed to register info bean: searcher javax.management.InstanceAlreadyExistsException: solr/collection1:type=searcher,id=org.apache.solr.search.SolrIndexSearcher
[junit4:junit4]   2> 	at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:467)
[junit4:junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.internal_addObject(DefaultMBeanServerInterceptor.java:1520)
[junit4:junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:986)
[junit4:junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:938)
[junit4:junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:330)
[junit4:junit4]   2> 	at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:517)
[junit4:junit4]   2> 	at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:140)
[junit4:junit4]   2> 	at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:51)
[junit4:junit4]   2> 	at org.apache.solr.search.SolrIndexSearcher.register(SolrIndexSearcher.java:239)
[junit4:junit4]   2> 	at org.apache.solr.core.SolrCore.registerSearcher(SolrCore.java:1763)
[junit4:junit4]   2> 	at org.apache.solr.core.SolrCore.access$000(SolrCore.java:129)
[junit4:junit4]   2> 	at org.apache.solr.core.SolrCore$7.call(SolrCore.java:1640)
[junit4:junit4]   2> 	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
[junit4:junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
[junit4:junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
[junit4:junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4:junit4]   2> 	at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 
[junit4:junit4]   2> 22660 T2063 C210 P61453 oasc.SolrDeletionPolicy.updateCommits newest commit = 4[segments_4]
[junit4:junit4]   2> 22660 T2046 C209 P61449 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 22662 T2058 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@36000c53 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 22661 T2094 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@f6d4253 main{StandardDirectoryReader(segments_4:1:nrt)}
[junit4:junit4]   2> 22663 T2046 C209 P61449 /update {waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 9
[junit4:junit4]   2> 22663 T2063 C210 P61453 oass.SolrIndexSearcher.<init> Opening Searcher@7a095066 main
[junit4:junit4]   2> 22663 T2080 C212 P27180 /update {waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 9
[junit4:junit4]   2> 22664 T2063 C210 P61453 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 22665 T2075 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7a095066 main{StandardDirectoryReader(segments_3:1:nrt)}
[junit4:junit4]   2> 22666 T2063 C210 P61453 /update {waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 9
[junit4:junit4]   2> 22667 T2033 C211 P61446 /update {waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 32
[junit4:junit4]   2> 22669 T1998 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 22672 T2031 C211 P61446 REQ /select {tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0 
[junit4:junit4]   2> 22675 T2047 C209 P61449 REQ /select {tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0 
[junit4:junit4]   2> 22679 T2064 C210 P61453 REQ /select {tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 22681 T2081 C212 P27180 REQ /select {tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0 
[junit4:junit4]   2> 24687 T2010 C213 P61442 oass.SolrIndexSearcher.<init> Opening Searcher@3b3596d0 realtime
[junit4:junit4]   2> 24688 T2010 C213 P61442 /update {wt=javabin&version=2} {deleteByQuery=*:* (-1429411806738120704)} 0 3
[junit4:junit4]   2> 24697 T2063 C210 P61453 /update {update.distrib=FROMLEADER&_version_=-1429411806743363584&update.from=http://127.0.0.1:61446/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1429411806743363584)} 0 2
[junit4:junit4]   2> 24697 T2080 C212 P27180 /update {update.distrib=FROMLEADER&_version_=-1429411806743363584&update.from=http://127.0.0.1:61446/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1429411806743363584)} 0 2
[junit4:junit4]   2> 24697 T2046 C209 P61449 /update {update.distrib=FROMLEADER&_version_=-1429411806743363584&update.from=http://127.0.0.1:61446/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1429411806743363584)} 0 2
[junit4:junit4]   2> 24698 T2033 C211 P61446 /update {wt=javabin&version=2} {deleteByQuery=*:* (-1429411806743363584)} 0 7
[junit4:junit4]   2> 24704 T2010 C213 P61442 /update {wt=javabin&version=2} {add=[0 (1429411806755946496)]} 0 2
[junit4:junit4]   2> 24713 T2046 C209 P61449 /update {distrib.from=http://127.0.0.1:61446/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1429411806760140800)]} 0 2
[junit4:junit4]   2> 24713 T2080 C212 P27180 /update {distrib.from=http://127.0.0.1:61446/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1429411806760140800)]} 0 2
[junit4:junit4]   2> 24713 T2063 C210 P61453 /update {distrib.from=http://127.0.0.1:61446/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1429411806760140800)]} 0 2
[junit4:junit4]   2> 24714 T2033 C211 P61446 /update {wt=javabin&version=2} {add=[0 (1429411806760140800)]} 0 7
[junit4:junit4]   2> 24718 T2010 C213 P61442 /update {wt=javabin&version=2} {add=[1 (1429411806770626560)]} 0 1
[junit4:junit4]   2> 24725 T2080 C212 P27180 /update {distrib.from=http://127.0.0.1:61446/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1429411806774820864)]} 0 1
[junit4:junit4]   2> 24725 T2046 C209 P61449 /update {distrib.from=http://127.0.0.1:61446/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1429411806774820864)]} 0 1
[junit4:junit4]   2> 24726 T2063 C210 P61453 /update {distrib.from=http://127.0.0.1:61446/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1429411806774820864)]} 0 2
[junit4:junit4]   2> 24727 T2033 C211 P61446 /update {wt=javabin&version=2} {add=[1 (1429411806774820864)]} 0 6
[junit4:junit4]   2> 24731 T2010 C213 P61442 /update {wt=javabin&version=2} {add=[2 (1429411806784258048)]} 0 1
[junit4:junit4]   2> 24739 T2080 C212 P27180 /update {distrib.from=http://127.0.0.1:61446/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[2 (1429411806789500928)]} 0 1
[junit4:junit4]   2> 24739 T2063 C210 P61453 /update {distrib.from=http://127.0.0.1:61446/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[2 (1429411806789500928)]} 0 1
[junit4:junit4]   2> 24740 T2033 C211 P61446 /update {test.distrib.skip.servers=http://127.0.0.1:61449/collection1/&wt=javabin&version=2} {add=[2 (1429411806789500928)]} 0 6
[junit4:junit4]   2> 24744 T2010 C213 P61442 /update {wt=javabin&version=2} {add=[3 (1429411806797889536)]} 0 1
[junit4:junit4]   2> 24753 T2080 C212 P27180 /update {distrib.from=http://127.0.0.1:61446/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[3 (1429411806803132416)]} 0 2
[junit4:junit4]   2> 24754 T2033 C211 P61446 /update {test.distrib.skip.servers=http://127.0.0.1:61449/collection1/&test.distrib.skip.servers=http://127.0.0.1:61453/collection1/&wt=javabin&version=2} {add=[3 (1429411806803132416)]} 0 7
[junit4:junit4]   2> 24757 T2010 C213 P61442 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 24762 T2010 C213 P61442 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@60c481cc lockFactory=org.apache.lucene.store.NativeFSLockFactory@6ba8ee5d),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@60c481cc lockFactory=org.apache.lucene.store.NativeFSLockFactory@6ba8ee5d),segFN=segments_3,generation=3,filenames=[_0_Lucene40_0.tim, _0.fnm, _0_dv.cfs, _0_nrm.cfs, _0_Lucene40_0.prx, _0_Lucene40_0.tip, _0_Lucene40_0.frq, _0_nrm.cfe, _0.fdx, _0.si, segments_3, _0.fdt, _0_dv.cfe]
[junit4:junit4]   2> 24763 T2010 C213 P61442 oasc.SolrDeletionPolicy.updateCommits newest commit = 3[_0_Lucene40_0.tim, _0.fnm, _0_dv.cfs, _0_nrm.cfs, _0_Lucene40_0.prx, _0_Lucene40_0.tip, _0_Lucene40_0.frq, _0_nrm.cfe, _0.fdx, _0.si, segments_3, _0.fdt, _0_dv.cfe]
[junit4:junit4]   2> 24765 T2010 C213 P61442 oass.SolrIndexSearcher.<init> Opening Searcher@52a9fac1 main
[junit4:junit4]   2> 24766 T2010 C213 P61442 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 24767 T2024 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@52a9fac1 main{StandardDirectoryReader(segments_3:3:nrt _0(5.0):C4)}
[junit4:junit4]   2> 24767 T2010 C213 P61442 /update {waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 10
[junit4:junit4]   2> 24770 T2033 C211 P61446 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 24776 T2033 C211 P61446 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=5
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1a3f9f3b lockFactory=org.apache.lucene.store.NativeFSLockFactory@6ebe26e6),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1a3f9f3b lockFactory=org.apache.lucene.store.NativeFSLockFactory@6ebe26e6),segFN=segments_3,generation=3,filenames=[segments_3]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1a3f9f3b lockFactory=org.apache.lucene.store.NativeFSLockFactory@6ebe26e6),segFN=segments_4,generation=4,filenames=[segments_4]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1a3f9f3b lockFactory=org.apache.lucene.store.NativeFSLockFactory@6ebe26e6),segFN=segments_5,generation=5,filenames=[segments_5]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1a3f9f3b lockFactory=org.apache.lucene.store.NativeFSLockFactory@6ebe26e6),segFN=segments_6,generation=6,filenames=[_0_Lucene40_0.tim, _0.fnm, _0_dv.cfs, _0_nrm.cfs, _0_Lucene40_0.prx, _0_Lucene40_0.tip, _0_Lucene40_0.frq, _0_nrm.cfe, _0.fdx, _0.si, segments_6, _0.fdt, _0_dv.cfe]
[junit4:junit4]   2> 24777 T2033 C211 P61446 oasc.SolrDeletionPolicy.updateCommits newest commit = 6[_0_Lucene40_0.tim, _0.fnm, _0_dv.cfs, _0_nrm.cfs, _0_Lucene40_0.prx, _0_Lucene40_0.tip, _0_Lucene40_0.frq, _0_nrm.cfe, _0.fdx, _0.si, segments_6, _0.fdt, _0_dv.cfe]
[junit4:junit4]   2> 24779 T2033 C211 P61446 oass.SolrIndexSearcher.<init> Opening Searcher@e88da92 main
[junit4:junit4]   2> 24780 T2033 C211 P61446 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 24780 T2042 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@e88da92 main{StandardDirectoryReader(segments_6:3:nrt _0(5.0):C4)}
[junit4:junit4]   2> 24781 T2033 C211 P61446 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:61449/collection1/, StdNode: http://127.0.0.1:61453/collection1/, StdNode: http://127.0.0.1:27180/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4]   2> 24784 T2046 C209 P61449 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 24784 T2080 C212 P27180 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 24784 T2063 C210 P61453 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 24789 T2046 C209 P61449 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3ce15725 lockFactory=org.apache.lucene.store.NativeFSLockFactory@ed17b80),segFN=segments_3,generation=3,filenames=[segments_3]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3ce15725 lockFactory=org.apache.lucene.store.NativeFSLockFactory@ed17b80),segFN=segments_4,generation=4,filenames=[_0_Lucene40_0.tim, _0.fnm, _0_dv.cfs, _0_nrm.cfs, _0_Lucene40_0.prx, _0_Lucene40_0.tip, _0_Lucene40_0.frq, _0_nrm.cfe, _0.fdx, _0.si, _0.fdt, segments_4, _0_dv.cfe]
[junit4:junit4]   2> 24789 T2063 C210 P61453 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@65d5a68d lockFactory=org.apache.lucene.store.NativeFSLockFactory@5d20e1a5),segFN=segments_4,generation=4,filenames=[segments_4]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@65d5a68d lockFactory=org.apache.lucene.store.NativeFSLockFactory@5d20e1a5),segFN=segments_5,generation=5,filenames=[_0_Lucene40_0.tim, _0.fnm, _0_dv.cfs, _0_nrm.cfs, _0_Lucene40_0.prx, _0_Lucene40_0.tip, _0_Lucene40_0.frq, _0_nrm.cfe, _0.fdx, segments_5, _0.si, _0.fdt, _0_dv.cfe]
[junit4:junit4]   2> 24790 T2080 C212 P27180 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@34059959 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5414b45),segFN=segments_5,generation=5,filenames=[segments_5]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@34059959 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5414b45),segFN=segments_6,generation=6,filenames=[_0_Lucene40_0.tim, _0.fnm, _0_dv.cfs, _0_nrm.cfs, _0_Lucene40_0.prx, _0_Lucene40_0.tip, _0_Lucene40_0.frq, _0_nrm.cfe, _0.fdx, _0.si, segments_6, _0.fdt, _0_dv.cfe]
[junit4:junit4]   2> 24790 T2063 C210 P61453 oasc.SolrDeletionPolicy.updateCommits newest commit = 5[_0_Lucene40_0.tim, _0.fnm, _0_dv.cfs, _0_nrm.cfs, _0_Lucene40_0.prx, _0_Lucene40_0.tip, _0_Lucene40_0.frq, _0_nrm.cfe, _0.fdx, segments_5, _0.si, _0.fdt, _0_dv.cfe]
[junit4:junit4]   2> 24790 T2046 C209 P61449 oasc.SolrDeletionPolicy.updateCommits newest commit = 4[_0_Lucene40_0.tim, _0.fnm, _0_dv.cfs, _0_nrm.cfs, _0_Lucene40_0.prx, _0_Lucene40_0.tip, _0_Lucene40_0.frq, _0_nrm.cfe, _0.fdx, _0.si, _0.fdt, segments_4, _0_dv.cfe]
[junit4:junit4]   2> 24791 T2080 C212 P27180 oasc.SolrDeletionPolicy.updateCommits newest commit = 6[_0_Lucene40_0.tim, _0.fnm, _0_dv.cfs, _0_nrm.cfs, _0_Lucene40_0.prx, _0_Lucene40_0.tip, _0_Lucene40_0.frq, _0_nrm.cfe, _0.fdx, _0.si, segments_6, _0.fdt, _0_dv.cfe]
[junit4:junit4]   2> 24792 T2063 C210 P61453 oass.SolrIndexSearcher.<init> Opening Searcher@36130ae2 main
[junit4:junit4]   2> 24792 T2046 C209 P61449 oass.SolrIndexSearcher.<init> Opening Searcher@1f264bd6 main
[junit4:junit4]   2> 24793 T2063 C210 P61453 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 24793 T2080 C212 P27180 oass.SolrIndexSearcher.<init> Opening Searcher@243d0c2a main
[junit4:junit4]   2> 24793 T2046 C209 P61449 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 24794 T2075 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@36130ae2 main{StandardDirectoryReader(segments_5:3:nrt _0(5.0):C3)}
[junit4:junit4]   2> 24793 T2080 C212 P27180 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 24794 T2094 oasc.JmxMonitoredMap.put WARNING Failed to register info bean: searcher javax.management.InstanceAlreadyExistsException: solr/collection1:type=searcher,id=org.apache.solr.search.SolrIndexSearcher
[junit4:junit4]   2> 	at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:467)
[junit4:junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.internal_addObject(DefaultMBeanServerInterceptor.java:1520)
[junit4:junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:986)
[junit4:junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:938)
[junit4:junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:330)
[junit4:junit4]   2> 	at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:517)
[junit4:junit4]   2> 	at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:140)
[junit4:junit4]   2> 	at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:51)
[junit4:junit4]   2> 	at org.apache.solr.search.SolrIndexSearcher.register(SolrIndexSearcher.java:239)
[junit4:junit4]   2> 	at org.apache.solr.core.SolrCore.registerSearcher(SolrCore.java:1763)
[junit4:junit4]   2> 	at org.apache.solr.core.SolrCore.access$000(SolrCore.java:129)
[junit4:junit4]   2> 	at org.apache.solr.core.SolrCore$7.call(SolrCore.java:1640)
[junit4:junit4]   2> 	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
[junit4:junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
[junit4:junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
[junit4:junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4:junit4]   2> 	at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 
[junit4:junit4]   2> 24794 T2058 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1f264bd6 main{StandardDirectoryReader(segments_4:3:nrt _0(5.0):C2)}
[junit4:junit4]   2> 24794 T2063 C210 P61453 /update {waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 10
[junit4:junit4]   2> 24795 T2094 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@243d0c2a main{StandardDirectoryReader(segments_6:3:nrt _0(5.0):C4)}
[junit4:junit4]   2> 24795 T2046 C209 P61449 /update {waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 11
[junit4:junit4]   2> 24796 T2080 C212 P27180 /update {waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 12
[junit4:junit4]   2> 24797 T2033 C211 P61446 /update {waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 27
[junit4:junit4]   2> 24797 T1998 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):230
[junit4:junit4]   2> 24798 T1998 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 24800 T1998 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4]   2> 24803 T2031 C211 P61446 REQ /select {tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=1 
[junit4:junit4]   2> 24807 T2047 C209 P61449 REQ /select {tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=2 status=0 QTime=1 
[junit4:junit4]   2> 24811 T2064 C210 P61453 REQ /select {tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=3 status=0 QTime=1 
[junit4:junit4]   2> 24815 T2081 C212 P27180 REQ /select {tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=1 
[junit4:junit4]   2> 24816 T1998 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 24829 T2065 oasha.CollectionsHandler.handleSyncShardAction Syncing shard : shard=shard1&action=SYNCSHARD&collection=collection1&wt=javabin&version=2
[junit4:junit4]   2> 24830 T2065 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 24846 T2035 oasha.CoreAdminHandler.handleRequestSyncAction I have been requested to sync up my shard
[junit4:junit4]   2> 24847 T2035 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:61446/collection1/
[junit4:junit4]   2> 24848 T2035 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:61446 START replicas=[http://127.0.0.1:61449/collection1/, http://127.0.0.1:61453/collection1/, http://127.0.0.1:27180/collection1/] nUpdates=100
[junit4:junit4]   2> 24850 T2048 C209 P61449 REQ /get {getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0 
[junit4:junit4]   2> 24850 T2082 C212 P27180 REQ /get {getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0 
[junit4:junit4]   2> 24850 T2066 C210 P61453 REQ /get {getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0 
[junit4:junit4]   2> 24852 T2035 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:61446  Received 3 versions from 127.0.0.1:61449/collection1/
[junit4:junit4]   2> 24852 T2035 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:61446  Our versions are newer. ourLowThreshold=1429411806743363584 otherHigh=1429411806774820864
[junit4:junit4]   2> 24853 T2035 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:61446  Received 5 versions from 127.0.0.1:27180/collection1/
[junit4:junit4]   2> 24853 T2035 oasu.PeerSync.

[...truncated too long message...]

e$1.evaluate(TestRuleStoreClassName.java:42)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
[junit4:junit4]    > 	at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 138255 T1998 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4]   2> 139072 T2090 oascc.ZkStateReader$3.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 139072 T2090 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 139072 T2090 oascc.ZkStateReader$2.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 139072 T2090 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 165648 T2073 oasu.PeerSync.handleResponse WARNING PeerSync: core=collection1 url=http://127.0.0.1:61453  couldn't connect to http://127.0.0.1:61449/collection1/, counting as success
[junit4:junit4]   2> 165648 T2073 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:61453 DONE. sync succeeded
[junit4:junit4]   2> 165649 T2073 oasc.SyncStrategy.syncReplicas We have been closed, won't attempt to sync replicas back to leader
[junit4:junit4]   2> 165650 T2073 oasc.ShardLeaderElectionContext.rejoinLeaderElection Not rejoining election because CoreContainer is shutdown
[junit4:junit4]   2> 165650 T2073 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@4afedaa8
[junit4:junit4]   2> 165651 T2073 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=10,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=305,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0}
[junit4:junit4]   2> 165652 T2073 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4]   2> 165652 T2073 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4]   2> 165652 T2073 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4]   2> 165652 T2073 C210 P61453 oasu.TransactionLog.forceClose SEVERE Error: Forcing close of tlog{file=./org.apache.solr.cloud.SyncSliceTest-1363193302973/jetty3/tlog/tlog.0000000000000000005 refcount=1}
[junit4:junit4]   2> 165655 T2073 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4]   2> 165655 T2073 oasc.CachingDirectoryFactory.close Closing directory when closing factory: ./org.apache.solr.cloud.SyncSliceTest-1363193302973/jetty3/index
[junit4:junit4]   2> 165655 T2073 oasc.CachingDirectoryFactory.closeDirectory Closing directory: ./org.apache.solr.cloud.SyncSliceTest-1363193302973/jetty3/index
[junit4:junit4]   2> 165656 T2073 oasc.CachingDirectoryFactory.close Closing directory when closing factory: ./org.apache.solr.cloud.SyncSliceTest-1363193302973/jetty3/index.20130313164841229
[junit4:junit4]   2> 165656 T2073 oasc.CachingDirectoryFactory.closeDirectory Closing directory: ./org.apache.solr.cloud.SyncSliceTest-1363193302973/jetty3/index.20130313164841229
[junit4:junit4]   2> 165656 T2073 oasc.CachingDirectoryFactory.close Closing directory when closing factory: ./org.apache.solr.cloud.SyncSliceTest-1363193302973/jetty3
[junit4:junit4]   2> 165656 T2073 oasc.CachingDirectoryFactory.closeDirectory Closing directory: ./org.apache.solr.cloud.SyncSliceTest-1363193302973/jetty3
[junit4:junit4]   2> 165656 T2073 oasc.LeaderElector$1.process WARNING  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4:junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:236)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:233)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:233)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:84)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:129)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
[junit4:junit4]   2> 
[junit4:junit4]   2> 165657 T2073 oascc.ZkStateReader$3.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 165657 T2073 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 165657 T2073 oascc.ZkStateReader$2.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 165657 T2073 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> NOTE: test params are: codec=Lucene40, sim=RandomSimilarityProvider(queryNorm=true,coord=no): {}, locale=fr_CA, timezone=Africa/Lome
[junit4:junit4]   2> NOTE: FreeBSD 9.0-RELEASE amd64/Sun Microsystems Inc. 1.6.0_32 (64-bit)/cpus=16,threads=3,free=47918512,total=437846016
[junit4:junit4]   2> NOTE: All tests run in this JVM: [NotRequiredUniqueKeyTest, PingRequestHandlerTest, TestDistributedSearch, CachingDirectoryFactoryTest, XmlUpdateRequestHandlerTest, BadIndexSchemaTest, DateMathParserTest, TermVectorComponentDistributedTest, TestCSVResponseWriter, TestLazyCores, TestFastOutputStream, TestLuceneMatchVersion, UpdateParamsTest, IndexBasedSpellCheckerTest, OpenExchangeRatesOrgProviderTest, TestNumberUtils, JSONWriterTest, SpellCheckComponentTest, TestUpdate, TestFoldingMultitermQuery, TestOmitPositions, BasicDistributedZk2Test, TestSuggestSpellingConverter, NoCacheHeaderTest, TestReload, SpellCheckCollatorTest, TestRecovery, HighlighterTest, TestStressReorder, CollectionsAPIDistributedZkTest, URLClassifyProcessorTest, TestCollationField, CopyFieldTest, TestDynamicFieldResource, TestWordDelimiterFilterFactory, SimplePostToolTest, SignatureUpdateProcessorFactoryTest, TestMultiCoreConfBootstrap, TestComponentsName, EchoParamsTest, TestDefaultSimilarityFactory, SortByFunctionTest, ConvertedLegacyTest, TestPostingsSolrHighlighter, DirectSolrSpellCheckerTest, SpellPossibilityIteratorTest, TestSolrXMLSerializer, ShardRoutingCustomTest, TestIndexingPerformance, TestJmxIntegration, ZkNodePropsTest, FieldAnalysisRequestHandlerTest, SolrRequestParserTest, TestPropInject, TestPartialUpdateDeduplication, PathHierarchyTokenizerFactoryTest, TestSolrDiscoveryProperties, XsltUpdateRequestHandlerTest, OutputWriterTest, DistanceFunctionTest, SampleTest, TestStressLucene, BasicZkTest, ChaosMonkeySafeLeaderTest, SyncSliceTest]
[junit4:junit4] Completed on J1 in 166.31s, 1 test, 1 failure <<< FAILURES!

[...truncated 406 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/build.xml:381: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/build.xml:361: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build.xml:183: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/common-build.xml:449: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/lucene/common-build.xml:1213: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/lucene/common-build.xml:877: There were test failures: 271 suites, 1141 tests, 1 failure, 27 ignored (8 assumptions)

Total time: 50 minutes 36 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-Tests-trunk-Java6 - Build # 16007 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-trunk-Java6/16007/

All tests passed

Build Log:
[...truncated 7920 lines...]
    [javac] Compiling 538 source files to /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/classes/java
    [javac] /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/core/src/java/org/apache/solr/servlet/LoadAdminUiServlet.java:57: not a statement
    [javac]         Package package = SolrCore.class.getPackage();
    [javac]         ^
    [javac] /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/core/src/java/org/apache/solr/servlet/LoadAdminUiServlet.java:57: ';' expected
    [javac]         Package package = SolrCore.class.getPackage();
    [javac]                ^
    [javac] /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/core/src/java/org/apache/solr/servlet/LoadAdminUiServlet.java:67: illegal start of expression
    [javac]             StringEscapeUtils.escapeJavaScript(package.getSpecificationVersion())
    [javac]                                                ^
    [javac] /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/core/src/java/org/apache/solr/servlet/LoadAdminUiServlet.java:67: '}' expected
    [javac]             StringEscapeUtils.escapeJavaScript(package.getSpecificationVersion())
    [javac]                                                                                 ^
    [javac] /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/core/src/java/org/apache/solr/servlet/LoadAdminUiServlet.java:51: 'try' without 'catch' or 'finally'
    [javac]       try {
    [javac]       ^
    [javac] /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/core/src/java/org/apache/solr/servlet/LoadAdminUiServlet.java:72: 'finally' without 'try'
    [javac]       } finally {
    [javac]         ^
    [javac] /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/core/src/java/org/apache/solr/servlet/LoadAdminUiServlet.java:75: illegal start of type
    [javac]     } else {
    [javac]       ^
    [javac] /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/core/src/java/org/apache/solr/servlet/LoadAdminUiServlet.java:75: ';' expected
    [javac]     } else {
    [javac]           ^
    [javac] /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/core/src/java/org/apache/solr/servlet/LoadAdminUiServlet.java:76: <identifier> expected
    [javac]       response.sendError(404);
    [javac]                         ^
    [javac] /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/core/src/java/org/apache/solr/servlet/LoadAdminUiServlet.java:76: illegal start of type
    [javac]       response.sendError(404);
    [javac]                          ^
    [javac] /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/core/src/java/org/apache/solr/servlet/LoadAdminUiServlet.java:78: class, interface, or enum expected
    [javac]   }
    [javac]   ^
    [javac] 11 errors

BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/build.xml:381: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/build.xml:361: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build.xml:183: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/common-build.xml:449: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/common-build.xml:389: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/common-build.xml:335: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/lucene/common-build.xml:479: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/lucene/common-build.xml:1578: Compile failed; see the compiler error output for details.

Total time: 27 minutes 2 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-Tests-trunk-Java6 - Build # 16006 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-trunk-Java6/16006/

1 tests failed.
REGRESSION:  org.apache.solr.cloud.SyncSliceTest.testDistribSearch

Error Message:
Test Setup Failure: shard1 should have just been set up to be inconsistent - but it's still consistent. Leader:http://127.0.0.1:22357/collection1skip list:[CloudJettyRunner [url=http://127.0.0.1:38389/collection1], CloudJettyRunner [url=http://127.0.0.1:38389/collection1]]

Stack Trace:
java.lang.AssertionError: Test Setup Failure: shard1 should have just been set up to be inconsistent - but it's still consistent. Leader:http://127.0.0.1:22357/collection1skip list:[CloudJettyRunner [url=http://127.0.0.1:38389/collection1], CloudJettyRunner [url=http://127.0.0.1:38389/collection1]]
	at __randomizedtesting.SeedInfo.seed([A1E6A0DE354761F0:20002EC6421801CC]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNotNull(Assert.java:526)
	at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:212)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:806)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:679)




Build Log:
[...truncated 8519 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.SyncSliceTest
[junit4:junit4]   2> 0 T999 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4:junit4]   2> 5 T999 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4]   2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./solrtest-SyncSliceTest-1363354612464
[junit4:junit4]   2> 6 T999 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 6 T1000 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4]   2> 9 T1000 oazs.ZooKeeperServer.setTickTime tickTime set to 1000
[junit4:junit4]   2> 10 T1000 oazs.ZooKeeperServer.setMinSessionTimeout minSessionTimeout set to -1
[junit4:junit4]   2> 10 T1000 oazs.ZooKeeperServer.setMaxSessionTimeout maxSessionTimeout set to -1
[junit4:junit4]   2> 10 T1000 oazs.NIOServerCnxnFactory.configure binding to port 0.0.0.0/0.0.0.0:0
[junit4:junit4]   2> 11 T1000 oazsp.FileTxnSnapLog.save Snapshotting: 0x0 to /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-1363354612463/zookeeper/server1/data/version-2/snapshot.0
[junit4:junit4]   2> 107 T999 oasc.ZkTestServer.run start zk server on port:22341
[junit4:junit4]   2> 107 T999 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:22341 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@46a3b95a
[junit4:junit4]   2> 108 T999 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 109 T1005 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:22341. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 109 T1005 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:22341, initiating session
[junit4:junit4]   2> 110 T1001 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:22342
[junit4:junit4]   2> 111 T1001 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:22342
[junit4:junit4]   2> 111 T1003 oazsp.FileTxnLog.append Creating new log file: log.1
[junit4:junit4]   2> 113 T1003 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6e4472f60000 with negotiated timeout 10000 for client /140.211.11.196:22342
[junit4:junit4]   2> 113 T1005 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:22341, sessionid = 0x13d6e4472f60000, negotiated timeout = 10000
[junit4:junit4]   2> 114 T1006 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@46a3b95a name:ZooKeeperConnection Watcher:127.0.0.1:22341 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 114 T999 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 115 T999 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4]   2> 118 T1004 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d6e4472f60000
[junit4:junit4]   2> 126 T1006 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 127 T1001 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:22342 which had sessionid 0x13d6e4472f60000
[junit4:junit4]   2> 126 T999 oaz.ZooKeeper.close Session: 0x13d6e4472f60000 closed
[junit4:junit4]   2> 128 T999 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:22341/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@3f003e7f
[junit4:junit4]   2> 129 T999 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 129 T1007 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:22341. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 129 T1007 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:22341, initiating session
[junit4:junit4]   2> 130 T1001 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:22343
[junit4:junit4]   2> 130 T1001 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:22343
[junit4:junit4]   2> 131 T1003 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6e4472f60001 with negotiated timeout 10000 for client /140.211.11.196:22343
[junit4:junit4]   2> 131 T1007 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:22341, sessionid = 0x13d6e4472f60001, negotiated timeout = 10000
[junit4:junit4]   2> 132 T1008 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3f003e7f name:ZooKeeperConnection Watcher:127.0.0.1:22341/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 132 T999 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 132 T999 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4]   2> 137 T999 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4]   2> 141 T999 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4]   2> 143 T999 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4]   2> 153 T999 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 154 T999 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 167 T999 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4]   2> 168 T999 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4]   2> 273 T999 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4]   2> 273 T999 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4]   2> 276 T999 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4]   2> 277 T999 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4]   2> 285 T999 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4]   2> 286 T999 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4]   2> 289 T999 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 290 T999 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 293 T999 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 293 T999 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 296 T999 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 297 T999 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 300 T999 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4]   2> 301 T999 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4]   2> 304 T1004 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d6e4472f60001
[junit4:junit4]   2> 311 T1008 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 311 T1001 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:22343 which had sessionid 0x13d6e4472f60001
[junit4:junit4]   2> 311 T999 oaz.ZooKeeper.close Session: 0x13d6e4472f60001 closed
[junit4:junit4]   2> 505 T999 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 509 T999 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:22346
[junit4:junit4]   2> 510 T999 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 510 T999 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 511 T999 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1363354612770
[junit4:junit4]   2> 511 T999 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1363354612770/solr.xml
[junit4:junit4]   2> 512 T999 oasc.CoreContainer.<init> New CoreContainer 1230774714
[junit4:junit4]   2> 512 T999 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1363354612770/'
[junit4:junit4]   2> 513 T999 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1363354612770/'
[junit4:junit4]   2> 562 T999 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 563 T999 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 563 T999 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 564 T999 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 564 T999 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 565 T999 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 565 T999 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 566 T999 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 566 T999 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 566 T999 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 589 T999 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 607 T999 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:22341/solr
[junit4:junit4]   2> 608 T999 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 608 T999 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:22341 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@4a823021
[junit4:junit4]   2> 609 T999 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 610 T1018 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:22341. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 611 T1018 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:22341, initiating session
[junit4:junit4]   2> 611 T1001 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:22347
[junit4:junit4]   2> 611 T1001 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:22347
[junit4:junit4]   2> 613 T1003 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6e4472f60002 with negotiated timeout 20000 for client /140.211.11.196:22347
[junit4:junit4]   2> 613 T1018 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:22341, sessionid = 0x13d6e4472f60002, negotiated timeout = 20000
[junit4:junit4]   2> 613 T1019 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4a823021 name:ZooKeeperConnection Watcher:127.0.0.1:22341 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 614 T999 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 615 T1004 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d6e4472f60002
[junit4:junit4]   2> 623 T1019 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 623 T1001 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:22347 which had sessionid 0x13d6e4472f60002
[junit4:junit4]   2> 623 T999 oaz.ZooKeeper.close Session: 0x13d6e4472f60002 closed
[junit4:junit4]   2> 624 T999 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 637 T999 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:22341/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@39312903
[junit4:junit4]   2> 638 T999 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 638 T1020 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:22341. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 639 T1020 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:22341, initiating session
[junit4:junit4]   2> 639 T1001 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:22348
[junit4:junit4]   2> 639 T1001 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:22348
[junit4:junit4]   2> 641 T1003 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6e4472f60003 with negotiated timeout 20000 for client /140.211.11.196:22348
[junit4:junit4]   2> 641 T1020 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:22341, sessionid = 0x13d6e4472f60003, negotiated timeout = 20000
[junit4:junit4]   2> 641 T1021 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@39312903 name:ZooKeeperConnection Watcher:127.0.0.1:22341/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 641 T999 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 648 T1004 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6e4472f60003 type:create cxid:0x2 zxid:0x1a txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 649 T999 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4]   2> 652 T999 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:22346_
[junit4:junit4]   2> 653 T1004 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6e4472f60003 type:delete cxid:0x8 zxid:0x1c txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:22346_ Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:22346_
[junit4:junit4]   2> 659 T999 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:22346_
[junit4:junit4]   2> 662 T999 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4]   2> 675 T1004 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6e4472f60003 type:delete cxid:0x16 zxid:0x21 txntype:-1 reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4]   2> 676 T999 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 685 T999 oasc.Overseer.start Overseer (id=89348807882833923-127.0.0.1:22346_-n_0000000000) starting
[junit4:junit4]   2> 686 T1004 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6e4472f60003 type:create cxid:0x1b zxid:0x23 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 687 T1004 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6e4472f60003 type:create cxid:0x1c zxid:0x24 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 688 T1004 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6e4472f60003 type:create cxid:0x1d zxid:0x25 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 698 T1004 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6e4472f60003 type:create cxid:0x1e zxid:0x26 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 700 T1023 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4]   2> 700 T999 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4]   2> 712 T999 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4]   2> 714 T999 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 720 T1022 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 720 T1024 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1363354612770/collection1
[junit4:junit4]   2> 721 T1024 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4]   2> 721 T1024 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 721 T1024 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4]   2> 723 T1024 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1363354612770/collection1/'
[junit4:junit4]   2> 724 T1024 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-controljetty-1363354612770/collection1/lib/README' to classloader
[junit4:junit4]   2> 724 T1024 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-controljetty-1363354612770/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 772 T1024 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 827 T1024 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 928 T1024 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 935 T1024 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 1549 T1024 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 1560 T1024 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 1564 T1024 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 1576 T1024 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 1580 T1024 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 1583 T1024 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 1584 T1024 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 1585 T1024 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 1585 T1024 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 1586 T1024 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 1586 T1024 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 1586 T1024 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 1586 T1024 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1363354612770/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1363354612463/control/data/
[junit4:junit4]   2> 1586 T1024 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6477eb97
[junit4:junit4]   2> 1587 T1024 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 1587 T1024 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363354612463/control/data forceNew: false
[junit4:junit4]   2> 1587 T1024 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1363354612463/control/data/index/
[junit4:junit4]   2> 1587 T1024 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1363354612463/control/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 1588 T1024 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363354612463/control/data/index forceNew: false
[junit4:junit4]   2> 1590 T1024 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@13f40fb4 lockFactory=org.apache.lucene.store.NativeFSLockFactory@776977a4),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 1591 T1024 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 1592 T1024 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 1593 T1024 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 1593 T1024 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 1593 T1024 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 1594 T1024 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 1594 T1024 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 1594 T1024 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 1594 T1024 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 1594 T1024 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 1600 T1024 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 1605 T1024 oass.SolrIndexSearcher.<init> Opening Searcher@2ef0c2dc main
[junit4:junit4]   2> 1606 T1024 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.SyncSliceTest-1363354612463/control/data/tlog
[junit4:junit4]   2> 1606 T1024 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 1606 T1024 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 1609 T1025 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2ef0c2dc main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 1610 T1024 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 1610 T1024 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 1610 T1004 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6e4472f60003 type:create cxid:0x55 zxid:0x2a txntype:-1 reqpath:n/a Error Path:/solr/overseer/queue Error:KeeperErrorCode = NoNode for /solr/overseer/queue
[junit4:junit4]   2> 2223 T1022 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2224 T1022 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"control_collection",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:22346_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:22346"}
[junit4:junit4]   2> 2224 T1022 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4]   2> 2224 T1022 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 2224 T1004 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6e4472f60003 type:create cxid:0x5d zxid:0x2d txntype:-1 reqpath:n/a Error Path:/solr/overseer/queue-work Error:KeeperErrorCode = NoNode for /solr/overseer/queue-work
[junit4:junit4]   2> 2239 T1021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 2624 T1024 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 2624 T1024 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:22346 collection:control_collection shard:shard1
[junit4:junit4]   2> 2624 T1024 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4]   2> 2640 T1004 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6e4472f60003 type:delete cxid:0x77 zxid:0x37 txntype:-1 reqpath:n/a Error Path:/solr/collections/control_collection/leaders Error:KeeperErrorCode = NoNode for /solr/collections/control_collection/leaders
[junit4:junit4]   2> 2641 T1024 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 2641 T1004 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6e4472f60003 type:create cxid:0x78 zxid:0x38 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 2643 T1024 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 2643 T1024 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 2643 T1024 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:22346/collection1/
[junit4:junit4]   2> 2644 T1024 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 2644 T1024 oasc.SyncStrategy.syncToMe http://127.0.0.1:22346/collection1/ has no replicas
[junit4:junit4]   2> 2644 T1024 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:22346/collection1/
[junit4:junit4]   2> 2644 T1024 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4]   2> 2647 T1004 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6e4472f60003 type:create cxid:0x82 zxid:0x3c txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 3743 T1022 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 3764 T1021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 3822 T1024 oasc.ZkController.register We are http://127.0.0.1:22346/collection1/ and leader is http://127.0.0.1:22346/collection1/
[junit4:junit4]   2> 3822 T1024 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:22346
[junit4:junit4]   2> 3822 T1024 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 3822 T1024 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 3822 T1024 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 3824 T1024 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 3825 T999 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0
[junit4:junit4]   2> 3826 T999 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 3826 T999 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 3842 T999 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4]   2> 3843 T999 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:22341/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@26460762
[junit4:junit4]   2> 3844 T999 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 3844 T1027 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:22341. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 3845 T1027 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:22341, initiating session
[junit4:junit4]   2> 3845 T1001 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:22351
[junit4:junit4]   2> 3845 T1001 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:22351
[junit4:junit4]   2> 3847 T1003 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6e4472f60004 with negotiated timeout 10000 for client /140.211.11.196:22351
[junit4:junit4]   2> 3847 T1027 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:22341, sessionid = 0x13d6e4472f60004, negotiated timeout = 10000
[junit4:junit4]   2> 3847 T1028 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@26460762 name:ZooKeeperConnection Watcher:127.0.0.1:22341/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 3847 T999 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 3849 T999 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 3851 T999 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4]   2> 4035 T999 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 4038 T999 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:22352
[junit4:junit4]   2> 4038 T999 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 4039 T999 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 4039 T999 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1363354616310
[junit4:junit4]   2> 4040 T999 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1363354616310/solr.xml
[junit4:junit4]   2> 4040 T999 oasc.CoreContainer.<init> New CoreContainer 885495503
[junit4:junit4]   2> 4041 T999 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1363354616310/'
[junit4:junit4]   2> 4041 T999 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1363354616310/'
[junit4:junit4]   2> 4090 T999 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 4090 T999 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 4091 T999 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 4091 T999 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 4092 T999 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 4092 T999 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 4093 T999 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 4093 T999 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 4094 T999 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 4094 T999 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 4115 T999 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 4133 T999 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:22341/solr
[junit4:junit4]   2> 4134 T999 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 4134 T999 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:22341 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@700b27b6
[junit4:junit4]   2> 4135 T999 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 4136 T1038 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:22341. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 4136 T1038 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:22341, initiating session
[junit4:junit4]   2> 4137 T1001 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:22353
[junit4:junit4]   2> 4137 T1001 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:22353
[junit4:junit4]   2> 4139 T1003 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6e4472f60005 with negotiated timeout 20000 for client /140.211.11.196:22353
[junit4:junit4]   2> 4139 T1038 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:22341, sessionid = 0x13d6e4472f60005, negotiated timeout = 20000
[junit4:junit4]   2> 4139 T1039 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@700b27b6 name:ZooKeeperConnection Watcher:127.0.0.1:22341 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 4139 T999 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 4141 T1004 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d6e4472f60005
[junit4:junit4]   2> 4148 T1039 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 4148 T1001 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:22353 which had sessionid 0x13d6e4472f60005
[junit4:junit4]   2> 4148 T999 oaz.ZooKeeper.close Session: 0x13d6e4472f60005 closed
[junit4:junit4]   2> 4149 T999 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 4160 T999 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:22341/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@335d5786
[junit4:junit4]   2> 4161 T999 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 4161 T1040 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:22341. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 4162 T1040 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:22341, initiating session
[junit4:junit4]   2> 4162 T1001 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:22354
[junit4:junit4]   2> 4162 T1001 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:22354
[junit4:junit4]   2> 4163 T1003 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6e4472f60006 with negotiated timeout 20000 for client /140.211.11.196:22354
[junit4:junit4]   2> 4163 T1040 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:22341, sessionid = 0x13d6e4472f60006, negotiated timeout = 20000
[junit4:junit4]   2> 4164 T1041 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@335d5786 name:ZooKeeperConnection Watcher:127.0.0.1:22341/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 4164 T999 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 4165 T1004 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6e4472f60006 type:create cxid:0x1 zxid:0x4a txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 4166 T1004 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6e4472f60006 type:create cxid:0x2 zxid:0x4b txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 4169 T999 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 5172 T999 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:22352_
[junit4:junit4]   2> 5173 T1004 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6e4472f60006 type:delete cxid:0xb zxid:0x4c txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:22352_ Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:22352_
[junit4:junit4]   2> 5200 T999 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:22352_
[junit4:junit4]   2> 5204 T1021 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 5204 T1028 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 5204 T1021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 5204 T1041 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 5211 T1042 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1363354616310/collection1
[junit4:junit4]   2> 5212 T1042 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 5212 T1042 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 5213 T1042 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 5214 T1042 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1363354616310/collection1/'
[junit4:junit4]   2> 5216 T1042 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty1-1363354616310/collection1/lib/README' to classloader
[junit4:junit4]   2> 5216 T1042 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty1-1363354616310/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 5276 T1022 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 5276 T1022 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"127.0.0.1:22346__collection1",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"control_collection",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:22346_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:22346"}
[junit4:junit4]   2> 5277 T1042 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 5290 T1028 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 5290 T1021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 5290 T1041 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 5336 T1042 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 5437 T1042 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 5443 T1042 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 6045 T1042 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 6058 T1042 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 6062 T1042 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 6077 T1042 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 6083 T1042 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 6088 T1042 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 6089 T1042 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 6090 T1042 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 6090 T1042 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 6091 T1042 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 6091 T1042 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 6091 T1042 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 6092 T1042 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1363354616310/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1363354612463/jetty1/
[junit4:junit4]   2> 6092 T1042 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6477eb97
[junit4:junit4]   2> 6093 T1042 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 6093 T1042 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363354612463/jetty1 forceNew: false
[junit4:junit4]   2> 6093 T1042 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1363354612463/jetty1/index/
[junit4:junit4]   2> 6094 T1042 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1363354612463/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 6094 T1042 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363354612463/jetty1/index forceNew: false
[junit4:junit4]   2> 6097 T1042 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@f22d615 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6bd7fbbd),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 6097 T1042 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 6100 T1042 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 6100 T1042 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 6100 T1042 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 6101 T1042 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 6102 T1042 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 6102 T1042 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 6102 T1042 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 6103 T1042 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 6103 T1042 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 6110 T1042 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 6116 T1042 oass.SolrIndexSearcher.<init> Opening Searcher@23fc80f1 main
[junit4:junit4]   2> 6116 T1042 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.SyncSliceTest-1363354612463/jetty1/tlog
[junit4:junit4]   2> 6117 T1042 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 6117 T1042 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 6122 T1043 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@23fc80f1 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 6123 T1042 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 6124 T1042 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 6795 T1022 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 6795 T1022 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:22352_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:22352"}
[junit4:junit4]   2> 6796 T1022 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 1
[junit4:junit4]   2> 6796 T1022 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 6811 T1028 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 6811 T1021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 6811 T1041 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 7126 T1042 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 7126 T1042 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:22352 collection:collection1 shard:shard1
[junit4:junit4]   2> 7127 T1042 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 7145 T1004 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6e4472f60006 type:delete cxid:0x48 zxid:0x5c txntype:-1 reqpath:n/a Error Path:/solr/collections/collection1/leaders Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders
[junit4:junit4]   2> 7146 T1042 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 7146 T1004 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6e4472f60006 type:create cxid:0x49 zxid:0x5d txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 7148 T1042 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 7149 T1042 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 7149 T1042 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:22352/collection1/
[junit4:junit4]   2> 7149 T1042 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 7149 T1042 oasc.SyncStrategy.syncToMe http://127.0.0.1:22352/collection1/ has no replicas
[junit4:junit4]   2> 7149 T1042 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:22352/collection1/
[junit4:junit4]   2> 7150 T1042 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4]   2> 7153 T1004 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6e4472f60006 type:create cxid:0x53 zxid:0x61 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 8316 T1022 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 8332 T1028 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 8332 T1041 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 8332 T1021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 8379 T1042 oasc.ZkController.register We are http://127.0.0.1:22352/collection1/ and leader is http://127.0.0.1:22352/collection1/
[junit4:junit4]   2> 8379 T1042 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:22352
[junit4:junit4]   2> 8379 T1042 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 8379 T1042 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 8380 T1042 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 8381 T1042 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 8382 T999 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0
[junit4:junit4]   2> 8383 T999 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 8383 T999 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 8589 T999 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 8592 T999 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:22357
[junit4:junit4]   2> 8593 T999 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 8593 T999 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 8594 T999 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1363354620856
[junit4:junit4]   2> 8594 T999 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1363354620856/solr.xml
[junit4:junit4]   2> 8595 T999 oasc.CoreContainer.<init> New CoreContainer 218433027
[junit4:junit4]   2> 8595 T999 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1363354620856/'
[junit4:junit4]   2> 8596 T999 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1363354620856/'
[junit4:junit4]   2> 8644 T999 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 8645 T999 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 8645 T999 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 8646 T999 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 8646 T999 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 8647 T999 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 8647 T999 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 8647 T999 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 8648 T999 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 8648 T999 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 8668 T999 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 8686 T999 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:22341/solr
[junit4:junit4]   2> 8687 T999 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 8687 T999 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:22341 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@1d4fc1f8
[junit4:junit4]   2> 8688 T999 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 8689 T1054 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:22341. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 8689 T1054 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:22341, initiating session
[junit4:junit4]   2> 8690 T1001 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:22358
[junit4:junit4]   2> 8690 T1001 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:22358
[junit4:junit4]   2> 8691 T1003 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6e4472f60007 with negotiated timeout 20000 for client /140.211.11.196:22358
[junit4:junit4]   2> 8691 T1054 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:22341, sessionid = 0x13d6e4472f60007, negotiated timeout = 20000
[junit4:junit4]   2> 8692 T1055 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d4fc1f8 name:ZooKeeperConnection Watcher:127.0.0.1:22341 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 8692 T999 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 8694 T1004 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d6e4472f60007
[junit4:junit4]   2> 8695 T1055 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 8695 T1001 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:22358 which had sessionid 0x13d6e4472f60007
[junit4:junit4]   2> 8695 T999 oaz.ZooKeeper.close Session: 0x13d6e4472f60007 closed
[junit4:junit4]   2> 8695 T999 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 8706 T999 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:22341/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@604b1a70
[junit4:junit4]   2> 8708 T999 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 8708 T1056 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:22341. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 8709 T1056 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:22341, initiating session
[junit4:junit4]   2> 8709 T1001 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:22359
[junit4:junit4]   2> 8709 T1001 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:22359
[junit4:junit4]   2> 8710 T1003 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6e4472f60008 with negotiated timeout 20000 for client /140.211.11.196:22359
[junit4:junit4]   2> 8710 T1056 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:22341, sessionid = 0x13d6e4472f60008, negotiated timeout = 20000
[junit4:junit4]   2> 8711 T1057 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@604b1a70 name:ZooKeeperConnection Watcher:127.0.0.1:22341/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 8711 T999 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 8712 T1004 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6e4472f60008 type:create cxid:0x1 zxid:0x6e txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 8713 T1004 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6e4472f60008 type:create cxid:0x2 zxid:0x6f txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 8715 T999 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 9718 T999 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:22357_
[junit4:junit4]   2> 9719 T1004 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6e4472f60008 type:delete cxid:0xb zxid:0x70 txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:22357_ Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:22357_
[junit4:junit4]   2> 9720 T999 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:22357_
[junit4:junit4]   2> 9727 T1041 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 9728 T1021 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 9728 T1028 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 9729 T1021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 9729 T1028 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 9728 T1057 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 9730 T1041 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 9736 T1058 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1363354620856/collection1
[junit4:junit4]   2> 9736 T1058 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 9737 T1058 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 9737 T1058 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 9738 T1058 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1363354620856/collection1/'
[junit4:junit4]   2> 9739 T1058 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty2-1363354620856/collection1/lib/README' to classloader
[junit4:junit4]   2> 9740 T1058 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty2-1363354620856/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 9786 T1058 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 9838 T1022 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 9839 T1022 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"127.0.0.1:22352__collection1",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:22352_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:22352"}
[junit4:junit4]   2> 9846 T1058 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 9852 T1028 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 9852 T1021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 9852 T1041 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 9852 T1057 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 9853 T1058 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 9859 T1058 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 10464 T1058 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 10476 T1058 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 10480 T1058 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 10495 T1058 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 10500 T1058 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 10504 T1058 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 10505 T1058 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 10506 T1058 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 10506 T1058 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 10507 T1058 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 10507 T1058 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 10508 T1058 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 10508 T1058 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1363354620856/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1363354612463/jetty2/
[junit4:junit4]   2> 10508 T1058 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6477eb97
[junit4:junit4]   2> 10509 T1058 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 10509 T1058 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363354612463/jetty2 forceNew: false
[junit4:junit4]   2> 10509 T1058 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1363354612463/jetty2/index/
[junit4:junit4]   2> 10510 T1058 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1363354612463/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 10510 T1058 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363354612463/jetty2/index forceNew: false
[junit4:junit4]   2> 10513 T1058 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@16340bd8 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1b29c110),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 10513 T1058 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 10516 T1058 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 10516 T1058 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 10516 T1058 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 10517 T1058 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 10518 T1058 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 10518 T1058 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 10518 T1058 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 10519 T1058 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 10519 T1058 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 10526 T1058 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 10532 T1058 oass.SolrIndexSearcher.<init> Opening Searcher@17f70082 main
[junit4:junit4]   2> 10532 T1058 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.SyncSliceTest-1363354612463/jetty2/tlog
[junit4:junit4]   2> 10533 T1058 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 10533 T1058 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 10537 T1059 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@17f70082 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 10538 T1058 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 10539 T1058 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 11578 T1022 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 11579 T1022 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:22357_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:22357"}
[junit4:junit4]   2> 11579 T1022 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4]   2> 11580 T1022 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 11601 T1021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 11601 T1041 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 11601 T1057 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 11601 T1028 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 12542 T1058 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 12542 T1058 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:22357 collection:collection1 shard:shard1
[junit4:junit4]   2> 12545 T1058 oasc.ZkController.register We are http://127.0.0.1:22357/collection1/ and leader is http://127.0.0.1:22352/collection1/
[junit4:junit4]   2> 12546 T1058 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:22357
[junit4:junit4]   2> 12546 T1058 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 12546 T1058 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C68 name=collection1 org.apache.solr.core.SolrCore@47426a2d url=http://127.0.0.1:22357/collection1 node=127.0.0.1:22357_ C68_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:22357_, base_url=http://127.0.0.1:22357}
[junit4:junit4]   2> 12547 T1060 C68 P22357 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 12547 T1060 C68 P22357 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 12547 T1058 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 12548 T1060 C68 P22357 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4]   2> 12548 T1060 C68 P22357 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 12549 T999 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0
[junit4:junit4]   2> 12549 T1060 C68 P22357 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 12550 T999 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 12550 T999 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 12756 T999 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 12759 T999 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:22364
[junit4:junit4]   2> 12760 T999 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 12760 T999 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 12761 T999 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1363354625024
[junit4:junit4]   2> 12761 T999 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1363354625024/solr.xml
[junit4:junit4]   2> 12762 T999 oasc.CoreContainer.<init> New CoreContainer 949545703
[junit4:junit4]   2> 12762 T999 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1363354625024/'
[junit4:junit4]   2> 12763 T999 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1363354625024/'
[junit4:junit4]   2> 12811 T999 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 12812 T999 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 12813 T999 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 12813 T999 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 12813 T999 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 12814 T999 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 12814 T999 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 12815 T999 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 12815 T999 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 12816 T999 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 12835 T999 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 12853 T999 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:22341/solr
[junit4:junit4]   2> 12854 T999 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 12854 T999 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:22341 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@41b9508c
[junit4:junit4]   2> 12855 T999 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 12856 T1071 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:22341. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 12856 T1071 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:22341, initiating session
[junit4:junit4]   2> 12857 T1001 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:22366
[junit4:junit4]   2> 12857 T1001 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:22366
[junit4:junit4]   2> 12858 T1003 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6e4472f60009 with negotiated timeout 20000 for client /140.211.11.196:22366
[junit4:junit4]   2> 12859 T1071 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:22341, sessionid = 0x13d6e4472f60009, negotiated timeout = 20000
[junit4:junit4]   2> 12859 T1072 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@41b9508c name:ZooKeeperConnection Watcher:127.0.0.1:22341 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 12859 T999 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 12860 T1004 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d6e4472f60009
[junit4:junit4]   2> 12862 T1072 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 12862 T1001 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:22366 which had sessionid 0x13d6e4472f60009
[junit4:junit4]   2> 12862 T999 oaz.ZooKeeper.close Session: 0x13d6e4472f60009 closed
[junit4:junit4]   2> 12863 T999 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 12873 T999 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:22341/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@7f5be684
[junit4:junit4]   2> 12874 T999 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 12874 T1073 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:22341. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 12875 T1073 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:22341, initiating session
[junit4:junit4]   2> 12875 T1001 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:22367
[junit4:junit4]   2> 12875 T1001 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:22367
[junit4:junit4]   2> 12876 T1003 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6e4472f6000a with negotiated timeout 20000 for client /140.211.11.196:22367
[junit4:junit4]   2> 12876 T1073 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:22341, sessionid = 0x13d6e4472f6000a, negotiated timeout = 20000
[junit4:junit4]   2> 12877 T1074 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7f5be684 name:ZooKeeperConnection Watcher:127.0.0.1:22341/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 12877 T999 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 12878 T1004 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6e4472f6000a type:create cxid:0x1 zxid:0x81 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 12879 T1004 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6e4472f6000a type:create cxid:0x2 zxid:0x82 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 12888 T999 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 13111 T1022 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 13112 T1022 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"127.0.0.1:22357__collection1",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:22357_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:22357"}
[junit4:junit4]   2> 13117 T1028 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 13117 T1074 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 13117 T1041 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 13117 T1057 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 13117 T1021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 13568 T1031 oass.SolrDispatchFilter.handleAdminRequest /admin/cores {coreNodeName=127.0.0.1:22357__collection1&state=recovering&nodeName=127.0.0.1:22357_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001 
[junit4:junit4]   2> 13891 T999 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:22364_
[junit4:junit4]   2> 13893 T1004 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6e4472f6000a type:delete cxid:0xd zxid:0x87 txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:22364_ Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:22364_
[junit4:junit4]   2> 13894 T999 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:22364_
[junit4:junit4]   2> 13896 T1041 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 13897 T1028 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 13897 T1021 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 13897 T1074 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 13897 T1028 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 13897 T1074 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 13897 T1057 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 13897 T1021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 13898 T1057 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 13898 T1041 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 13906 T1075 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1363354625024/collection1
[junit4:junit4]   2> 13906 T1075 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 13907 T1075 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 13907 T1075 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 13908 T1075 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1363354625024/collection1/'
[junit4:junit4]   2> 13909 T1075 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty3-1363354625024/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 13910 T1075 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty3-1363354625024/collection1/lib/README' to classloader
[junit4:junit4]   2> 13958 T1075 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 14019 T1075 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 14120 T1075 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 14126 T1075 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 14799 T1075 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 14812 T1075 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 14816 T1075 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 14831 T1075 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 14836 T1075 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 14840 T1075 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 14841 T1075 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 14842 T1075 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 14842 T1075 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 14843 T1075 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 14843 T1075 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 14843 T1075 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 14844 T1075 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1363354625024/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1363354612463/jetty3/
[junit4:junit4]   2> 14844 T1075 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6477eb97
[junit4:junit4]   2> 14844 T1075 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 14845 T1075 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363354612463/jetty3 forceNew: false
[junit4:junit4]   2> 14845 T1075 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1363354612463/jetty3/index/
[junit4:junit4]   2> 14845 T1075 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1363354612463/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 14846 T1075 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363354612463/jetty3/index forceNew: false
[junit4:junit4]   2> 14848 T1075 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1a459854 lockFactory=org.apache.lucene.store.NativeFSLockFactory@19b41a92),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 14849 T1075 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 14851 T1075 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 14851 T1075 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 14852 T1075 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 14853 T1075 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 14853 T1075 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 14853 T1075 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 14854 T1075 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 14854 T1075 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 14855 T1075 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 14862 T1075 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 14867 T1075 oass.SolrIndexSearcher.<init> Opening Searcher@5ced448f main
[junit4:junit4]   2> 14868 T1075 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.SyncSliceTest-1363354612463/jetty3/tlog
[junit4:junit4]   2> 14868 T1075 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 14869 T1075 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 14873 T1076 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5ced448f main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 14874 T1075 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 14874 T1075 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2>  C68_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:22357_, base_url=http://127.0.0.1:22357}
[junit4:junit4]   2> 15569 T1060 C68 P22357 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:22352/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4]   2> 15569 T1060 C68 P22357 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:22357 START replicas=[http://127.0.0.1:22352/collection1/] nUpdates=100
[junit4:junit4]   2> 15569 T1060 C68 P22357 oasu.PeerSync.sync WARNING no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 15570 T1060 C68 P22357 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4]   2> 15570 T1060 C68 P22357 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4]   2> 15570 T1060 C68 P22357 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4]   2> 15571 T1060 C68 P22357 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4]   2> 15571 T1060 C68 P22357 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:22352/collection1/. core=collection1
[junit4:junit4]   2> 15571 T1060 C68 P22357 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C69 name=collection1 org.apache.solr.core.SolrCore@31fc1e62 url=http://127.0.0.1:22352/collection1 node=127.0.0.1:22352_ C69_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:22352_, base_url=http://127.0.0.1:22352, leader=true}
[junit4:junit4]   2> 15577 T1032 C69 P22352 REQ /get {getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1 
[junit4:junit4]   2> 15587 T1033 C69 P22352 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 15589 T1033 C69 P22352 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@f22d615 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6bd7fbbd),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 15590 T1033 C69 P22352 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 15590 T1033 C69 P22352 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@f22d615 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6bd7fbbd),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@f22d615 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6bd7fbbd),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 15591 T1033 C69 P22352 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[segments_2]
[junit4:junit4]   2> 15591 T1033 C69 P22352 oass.SolrIndexSearcher.<init> Opening Searcher@153ac7ad realtime
[junit4:junit4]   2> 15591 T1033 C69 P22352 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 15592 T1033 C69 P22352 /update {waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 5
[junit4:junit4]   2> 15592 T1060 C68 P22357 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 15593 T1060 C68 P22357 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4]   2> 15594 T1034 C69 P22352 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 15594 T1034 C69 P22352 REQ /replication {command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 15595 T1060 C68 P22357 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4]   2> 15595 T1060 C68 P22357 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4]   2> 15595 T1060 C68 P22357 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4]   2> 15597 T1034 C69 P22352 REQ /replication {command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0 
[junit4:junit4]   2> 15598 T1060 C68 P22357 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4]   2> 15599 T1060 C68 P22357 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363354612463/jetty2/index.20130315063708056 forceNew: false
[junit4:junit4]   2> 15599 T1060 C68 P22357 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@42a94fe0 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6b81ad9f) fullCopy=false
[junit4:junit4]   2> 15602 T1034 C69 P22352 REQ /replication {file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> 15603 T1060 C68 P22357 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4]   2> 15604 T1060 C68 P22357 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4]   2> 15604 T1060 C68 P22357 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4]   2> 15605 T1060 C68 P22357 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@16340bd8 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1b29c110),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@16340bd8 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1b29c110),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 15606 T1060 C68 P22357 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[segments_2]
[junit4:junit4]   2> 15606 T1060 C68 P22357 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4]   2> 15606 T1060 C68 P22357 oass.SolrIndexSearcher.<init> Opening Searcher@16cdc6e3 main
[junit4:junit4]   2> 15607 T1059 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@16cdc6e3 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 15608 T1060 C68 P22357 oash.SnapPuller.fetchLatestIndex removing temporary index download directory files MockDirWrapper(org.apache.lucene.store.RAMDirectory@42a94fe0 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6b81ad9f)
[junit4:junit4]   2> 15608 T1060 C68 P22357 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4]   2> 15608 T1060 C68 P22357 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4]   2> 15608 T1060 C68 P22357 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 15609 T1060 C68 P22357 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 15610 T1060 C68 P22357 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4]   2> 16125 T1022 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 16125 T1022 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:22364_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:22364"}
[junit4:junit4]   2> 16126 T1022 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4]   2> 16126 T1022 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 16133 T1022 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"127.0.0.1:22357__collection1",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:22357_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:22357"}
[junit4:junit4]   2> 16203 T1021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 16203 T1074 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 16203 T1057 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 16203 T1041 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 16203 T1028 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 16878 T1075 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 16878 T1075 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:22364 collection:collection1 shard:shard1
[junit4:junit4]   2> 16881 T1075 oasc.ZkController.register We are http://127.0.0.1:22364/collection1/ and leader is http://127.0.0.1:22352/collection1/
[junit4:junit4]   2> 16882 T1075 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:22364
[junit4:junit4]   2> 16882 T1075 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 16882 T1075 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C70 name=collection1 org.apache.solr.core.SolrCore@1c21fd6d url=http://127.0.0.1:22364/collection1 node=127.0.0.1:22364_ C70_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:22364_, base_url=http://127.0.0.1:22364}
[junit4:junit4]   2> 16882 T1079 C70 P22364 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 16883 T1075 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 16884 T1079 C70 P22364 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 16884 T1079 C70 P22364 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4]   2> 16884 T1079 C70 P22364 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 16885 T999 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0
[junit4:junit4]   2> 16885 T999 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 16886 T999 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 16927 T1079 C70 P22364 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 17082 T999 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 17085 T999 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:38389
[junit4:junit4]   2> 17086 T999 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 17087 T999 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 17087 T999 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1363354629356
[junit4:junit4]   2> 17088 T999 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1363354629356/solr.xml
[junit4:junit4]   2> 17088 T999 oasc.CoreContainer.<init> New CoreContainer 2074077083
[junit4:junit4]   2> 17089 T999 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1363354629356/'
[junit4:junit4]   2> 17089 T999 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1363354629356/'
[junit4:junit4]   2> 17138 T999 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 17138 T999 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 17139 T999 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 17139 T999 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 17140 T999 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 17140 T999 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 17140 T999 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 17141 T999 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 17141 T999 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 17142 T999 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 17163 T999 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 17181 T999 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:22341/solr
[junit4:junit4]   2> 17182 T999 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 17182 T999 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:22341 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@77d70c22
[junit4:junit4]   2> 17183 T999 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 17184 T1090 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:22341. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 17184 T1090 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:22341, initiating session
[junit4:junit4]   2> 17185 T1001 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:20200
[junit4:junit4]   2> 17185 T1001 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:20200
[junit4:junit4]   2> 17187 T1003 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6e4472f6000b with negotiated timeout 20000 for client /140.211.11.196:20200
[junit4:junit4]   2> 17187 T1090 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:22341, sessionid = 0x13d6e4472f6000b, negotiated timeout = 20000
[junit4:junit4]   2> 17187 T1091 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@77d70c22 name:ZooKeeperConnection Watcher:127.0.0.1:22341 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 17187 T999 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 17189 T1004 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d6e4472f6000b
[junit4:junit4]   2> 17198 T1091 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 17199 T1001 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:20200 which had sessionid 0x13d6e4472f6000b
[junit4:junit4]   2> 17198 T999 oaz.ZooKeeper.close Session: 0x13d6e4472f6000b closed
[junit4:junit4]   2> 17199 T999 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 17212 T999 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:22341/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@13f6325c
[junit4:junit4]   2> 17213 T999 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 17213 T1092 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:22341. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 17214 T1092 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:22341, initiating session
[junit4:junit4]   2> 17214 T1001 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:10640
[junit4:junit4]   2> 17214 T1001 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:10640
[junit4:junit4]   2> 17215 T1003 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6e4472f6000c with negotiated timeout 20000 for client /140.211.11.196:10640
[junit4:junit4]   2> 17215 T1092 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:22341, sessionid = 0x13d6e4472f6000c, negotiated timeout = 20000
[junit4:junit4]   2> 17216 T1093 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@13f6325c name:ZooKeeperConnection Watcher:127.0.0.1:22341/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 17216 T999 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 17217 T1004 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6e4472f6000c type:create cxid:0x1 zxid:0x98 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 17219 T1004 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6e4472f6000c type:create cxid:0x2 zxid:0x99 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 17227 T999 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 17731 T1022 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 17731 T1022 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"127.0.0.1:22364__collection1",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:22364_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:22364"}
[junit4:junit4]   2> 17745 T1021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 17745 T1028 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 17745 T1074 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 17745 T1041 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 17745 T1093 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 17745 T1057 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 17941 T1035 oass.SolrDispatchFilter.handleAdminRequest /admin/cores {coreNodeName=127.0.0.1:22364__collection1&state=recovering&nodeName=127.0.0.1:22364_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001 
[junit4:junit4]   2> 18230 T999 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38389_
[junit4:junit4]   2> 18231 T1004 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6e4472f6000c type:delete cxid:0xd zxid:0x9e txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:38389_ Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:38389_
[junit4:junit4]   2> 18232 T999 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:38389_
[junit4:junit4]   2> 18234 T1041 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 18234 T1093 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 18235 T1028 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 18235 T1057 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 18235 T1028 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 18235 T1021 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 18235 T1074 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 18235 T1057 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 18236 T1074 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 18236 T1021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 18237 T1041 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 18237 T1093 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 18250 T1094 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1363354629356/collection1
[junit4:junit4]   2> 18250 T1094 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 18251 T1094 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 18251 T1094 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 18253 T1094 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1363354629356/collection1/'
[junit4:junit4]   2> 18254 T1094 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty4-1363354629356/collection1/lib/README' to classloader
[junit4:junit4]   2> 18254 T1094 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty4-1363354629356/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 18301 T1094 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 18364 T1094 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 18465 T1094 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 18471 T1094 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 19067 T1094 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 19075 T1094 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 19078 T1094 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 19090 T1094 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 19093 T1094 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 19096 T1094 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 19097 T1094 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 19097 T1094 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 19098 T1094 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 19098 T1094 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 19099 T1094 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 19099 T1094 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 19099 T1094 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1363354629356/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1363354612463/jetty4/
[junit4:junit4]   2> 19099 T1094 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6477eb97
[junit4:junit4]   2> 19100 T1094 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 19100 T1094 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363354612463/jetty4 forceNew: false
[junit4:junit4]   2> 19100 T1094 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1363354612463/jetty4/index/
[junit4:junit4]   2> 19100 T1094 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1363354612463/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 19101 T1094 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363354612463/jetty4/index forceNew: false
[junit4:junit4]   2> 19103 T1094 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@53d07a4 lockFactory=org.apache.lucene.store.NativeFSLockFactory@65f7beca),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 19103 T1094 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 19105 T1094 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 19105 T1094 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 19105 T1094 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 19106 T1094 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 19106 T1094 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 19106 T1094 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 19106 T1094 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 19107 T1094 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 19107 T1094 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 19111 T1094 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 19115 T1094 oass.SolrIndexSearcher.<init> Opening Searcher@73533bc2 main
[junit4:junit4]   2> 19116 T1094 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.SyncSliceTest-1363354612463/jetty4/tlog
[junit4:junit4]   2> 19116 T1094 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 19116 T1094 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 19119 T1095 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@73533bc2 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 19120 T1094 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 19120 T1094 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 19251 T1022 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 19252 T1022 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:38389_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:38389"}
[junit4:junit4]   2> 19252 T1022 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4]   2> 19252 T1022 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 19264 T1021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 19264 T1028 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 19264 T1074 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 19264 T1041 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 19264 T1057 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 19264 T1093 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2>  C70_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:22364_, base_url=http://127.0.0.1:22364}
[junit4:junit4]   2> 19942 T1079 C70 P22364 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:22352/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4]   2> 19942 T1079 C70 P22364 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:22364 START replicas=[http://127.0.0.1:22352/collection1/] nUpdates=100
[junit4:junit4]   2> 19942 T1079 C70 P22364 oasu.PeerSync.sync WARNING no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 19943 T1079 C70 P22364 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4]   2> 19943 T1079 C70 P22364 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4]   2> 19943 T1079 C70 P22364 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4]   2> 19943 T1079 C70 P22364 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4]   2> 19944 T1079 C70 P22364 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:22352/collection1/. core=collection1
[junit4:junit4]   2> 19944 T1079 C70 P22364 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 19945 T1032 C69 P22352 REQ /get {getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0 
[junit4:junit4]   2> 19959 T1036 C69 P22352 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 19960 T1036 C69 P22352 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@f22d615 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6bd7fbbd),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@f22d615 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6bd7fbbd),segFN=segments_3,generation=3,filenames=[segments_3]
[junit4:junit4]   2> 19960 T1036 C69 P22352 oasc.SolrDeletionPolicy.updateCommits newest commit = 3[segments_3]
[junit4:junit4]   2> 19960 T1036 C69 P22352 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 19961 T1036 C69 P22352 /update {waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 2
[junit4:junit4]   2> 19961 T1079 C70 P22364 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 19961 T1079 C70 P22364 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4]   2> 19963 T1034 C69 P22352 REQ /replication {command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 19963 T1079 C70 P22364 oash.SnapPuller.fetchLatestIndex Master's generation: 3
[junit4:junit4]   2> 19964 T1079 C70 P22364 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4]   2> 19964 T1079 C70 P22364 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4]   2> 19966 T1034 C69 P22352 REQ /replication {command=filelist&qt=/replication&wt=javabin&generation=3&version=2} status=0 QTime=0 
[junit4:junit4]   2> 19966 T1079 C70 P22364 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4]   2> 19967 T1079 C70 P22364 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363354612463/jetty3/index.20130315063712425 forceNew: false
[junit4:junit4]   2> 19967 T1079 C70 P22364 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@258fff35 lockFactory=org.apache.lucene.store.NativeFSLockFactory@27a020d6) fullCopy=false
[junit4:junit4]   2> 19970 T1034 C69 P22352 REQ /replication {file=segments_3&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=3} status=0 QTime=0 
[junit4:junit4]   2> 19971 T1079 C70 P22364 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4]   2> 19971 T1079 C70 P22364 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4]   2> 19972 T1079 C70 P22364 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4]   2> 19973 T1079 C70 P22364 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1a459854 lockFactory=org.apache.lucene.store.NativeFSLockFactory@19b41a92),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1a459854 lockFactory=org.apache.lucene.store.NativeFSLockFactory@19b41a92),segFN=segments_3,generation=3,filenames=[segments_3]
[junit4:junit4]   2> 19974 T1079 C70 P22364 oasc.SolrDeletionPolicy.updateCommits newest commit = 3[segments_3]
[junit4:junit4]   2> 19974 T1079 C70 P22364 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4]   2> 19974 T1079 C70 P22364 oass.SolrIndexSearcher.<init> Opening Searcher@2b1ac320 main
[junit4:junit4]   2> 19975 T1076 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2b1ac320 main{StandardDirectoryReader(segments_3:1:nrt)}
[junit4:junit4]   2> 19975 T1079 C70 P22364 oash.SnapPuller.fetchLatestIndex removing temporary index download directory files MockDirWrapper(org.apache.lucene.store.RAMDirectory@258fff35 lockFactory=org.apache.lucene.store.NativeFSLockFactory@27a020d6)
[junit4:junit4]   2> 19976 T1079 C70 P22364 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4]   2> 19976 T1079 C70 P22364 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4]   2> 19976 T1079 C70 P22364 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 19976 T1079 C70 P22364 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 20003 T1079 C70 P22364 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4]   2> 20150 T1094 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 20150 T1094 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:38389 collection:collection1 shard:shard1
[junit4:junit4]   2> 20153 T1094 oasc.ZkController.register We are http://127.0.0.1:38389/collection1/ and leader is http://127.0.0.1:22352/collection1/
[junit4:junit4]   2> 20153 T1094 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:38389
[junit4:junit4]   2> 20153 T1094 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 20154 T1094 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C71 name=collection1 org.apache.solr.core.SolrCore@652e74a9 url=http://127.0.0.1:38389/collection1 node=127.0.0.1:38389_ C71_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:38389_, base_url=http://127.0.0.1:38389}
[junit4:junit4]   2> 20154 T1098 C71 P38389 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 20155 T1098 C71 P38389 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 20155 T1094 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 20155 T1098 C71 P38389 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4]   2> 20155 T1098 C71 P38389 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 20156 T999 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0
[junit4:junit4]   2> 20156 T1098 C71 P38389 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 20156 T999 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 20157 T999 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 20168 T999 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 20169 T999 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 15 for each attempt
[junit4:junit4]   2> 20170 T999 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):15
[junit4:junit4]   2> 20171 T999 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 20768 T1022 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 20769 T1022 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"127.0.0.1:22364__collection1",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:22364_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:22364"}
[junit4:junit4]   2> 20800 T1022 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"127.0.0.1:38389__collection1",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:38389_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:38389"}
[junit4:junit4]   2> 20820 T1028 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 20820 T1021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 20820 T1074 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 20820 T1093 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 20820 T1057 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 20820 T1041 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 21169 T1037 oass.SolrDispatchFilter.handleAdminRequest /admin/cores {coreNodeName=127.0.0.1:38389__collection1&state=recovering&nodeName=127.0.0.1:38389_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001 
[junit4:junit4]   2> 21172 T999 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 22174 T999 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2>  C71_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:38389_, base_url=http://127.0.0.1:38389}
[junit4:junit4]   2> 23171 T1098 C71 P38389 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:22352/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4]   2> 23171 T1098 C71 P38389 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:38389 START replicas=[http://127.0.0.1:22352/collection1/] nUpdates=100
[junit4:junit4]   2> 23171 T1098 C71 P38389 oasu.PeerSync.sync WARNING no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 23172 T1098 C71 P38389 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4]   2> 23172 T1098 C71 P38389 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4]   2> 23172 T1098 C71 P38389 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4]   2> 23172 T1098 C71 P38389 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4]   2> 23172 T1098 C71 P38389 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:22352/collection1/. core=collection1
[junit4:junit4]   2> 23172 T1098 C71 P38389 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 23174 T1032 C69 P22352 REQ /get {getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0 
[junit4:junit4]   2> 23176 T999 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 23184 T1031 C69 P22352 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 23184 T1031 C69 P22352 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=3
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@f22d615 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6bd7fbbd),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@f22d615 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6bd7fbbd),segFN=segments_3,generation=3,filenames=[segments_3]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@f22d615 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6bd7fbbd),segFN=segments_4,generation=4,filenames=[segments_4]
[junit4:junit4]   2> 23185 T1031 C69 P22352 oasc.SolrDeletionPolicy.updateCommits newest commit = 4[segments_4]
[junit4:junit4]   2> 23185 T1031 C69 P22352 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 23186 T1031 C69 P22352 /update {waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 2
[junit4:junit4]   2> 23187 T1098 C71 P38389 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 23187 T1098 C71 P38389 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4]   2> 23189 T1034 C69 P22352 REQ /replication {command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 23190 T1098 C71 P38389 oash.SnapPuller.fetchLatestIndex Master's generation: 4
[junit4:junit4]   2> 23190 T1098 C71 P38389 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4]   2> 23190 T1098 C71 P38389 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4]   2> 23192 T1034 C69 P22352 REQ /replication {command=filelist&qt=/replication&wt=javabin&generation=4&version=2} status=0 QTime=0 
[junit4:junit4]   2> 23193 T1098 C71 P38389 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4]   2> 23194 T1098 C71 P38389 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363354612463/jetty4/index.20130315063715651 forceNew: false
[junit4:junit4]   2> 23194 T1098 C71 P38389 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@50dbd14 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2e27b4c2) fullCopy=false
[junit4:junit4]   2> 23197 T1034 C69 P22352 REQ /replication {file=segments_4&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=4} status=0 QTime=0 
[junit4:junit4]   2> 23198 T1098 C71 P38389 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4]   2> 23199 T1098 C71 P38389 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4]   2> 23199 T1098 C71 P38389 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4]   2> 23200 T1098 C71 P38389 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@53d07a4 lockFactory=org.apache.lucene.store.NativeFSLockFactory@65f7beca),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@53d07a4 lockFactory=org.apache.lucene.store.NativeFSLockFactory@65f7beca),segFN=segments_4,generation=4,filenames=[segments_4]
[junit4:junit4]   2> 23201 T1098 C71 P38389 oasc.SolrDeletionPolicy.updateCommits newest commit = 4[segments_4]
[junit4:junit4]   2> 23202 T1098 C71 P38389 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4]   2> 23202 T1098 C71 P38389 oass.SolrIndexSearcher.<init> Opening Searcher@22007b4 main
[junit4:junit4]   2> 23203 T1095 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@22007b4 main{StandardDirectoryReader(segments_4:1:nrt)}
[junit4:junit4]   2> 23203 T1098 C71 P38389 oash.SnapPuller.fetchLatestIndex removing temporary index download directory files MockDirWrapper(org.apache.lucene.store.RAMDirectory@50dbd14 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2e27b4c2)
[junit4:junit4]   2> 23203 T1098 C71 P38389 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4]   2> 23203 T1098 C71 P38389 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4]   2> 23204 T1098 C71 P38389 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 23204 T1098 C71 P38389 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 23205 T1098 C71 P38389 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4]   2> 23828 T1022 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 23829 T1022 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"127.0.0.1:38389__collection1",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:38389_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:38389"}
[junit4:junit4]   2> 23843 T1028 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 23843 T1021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 23843 T1041 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 23843 T1093 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 23843 T1057 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 23843 T1074 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 24178 T999 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 24179 T999 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4]   2> ASYNC  NEW_CORE C72 name=collection1 org.apache.solr.core.SolrCore@7cffe09c url=http://127.0.0.1:22346/collection1 node=127.0.0.1:22346_ C72_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:22346_, base_url=http://127.0.0.1:22346, leader=true}
[junit4:junit4]   2> 24186 T1011 C72 P22346 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 24189 T1011 C72 P22346 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@13f40fb4 lockFactory=org.apache.lucene.store.NativeFSLockFactory@776977a4),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 24189 T1011 C72 P22346 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 24190 T1011 C72 P22346 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@13f40fb4 lockFactory=org.apache.lucene.store.NativeFSLockFactory@776977a4),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@13f40fb4 lockFactory=org.apache.lucene.store.NativeFSLockFactory@776977a4),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 24191 T1011 C72 P22346 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[segments_2]
[junit4:junit4]   2> 24191 T1011 C72 P22346 oass.SolrIndexSearcher.<init> Opening Searcher@4036f065 main
[junit4:junit4]   2> 24191 T1011 C72 P22346 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 24192 T1025 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4036f065 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 24193 T1011 C72 P22346 /update {waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 7
[junit4:junit4]   2> 24204 T1033 C69 P22352 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 24204 T1033 C69 P22352 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=4
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@f22d615 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6bd7fbbd),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@f22d615 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6bd7fbbd),segFN=segments_3,generation=3,filenames=[segments_3]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@f22d615 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6bd7fbbd),segFN=segments_4,generation=4,filenames=[segments_4]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@f22d615 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6bd7fbbd),segFN=segments_5,generation=5,filenames=[segments_5]
[junit4:junit4]   2> 24205 T1033 C69 P22352 oasc.SolrDeletionPolicy.updateCommits newest commit = 5[segments_5]
[junit4:junit4]   2> 24205 T1033 C69 P22352 oass.SolrIndexSearcher.<init> Opening Searcher@3ab1384b main
[junit4:junit4]   2> 24206 T1033 C69 P22352 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 24206 T1043 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3ab1384b main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 24207 T1033 C69 P22352 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:22357/collection1/, StdNode: http://127.0.0.1:22364/collection1/, StdNode: http://127.0.0.1:38389/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4]   2>  C70_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:22364_, base_url=http://127.0.0.1:22364}
[junit4:junit4]   2> 24220 T1064 C70 P22364 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 24221 T1064 C70 P22364 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1a459854 lockFactory=org.apache.lucene.store.NativeFSLockFactory@19b41a92),segFN=segments_3,generation=3,filenames=[segments_3]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1a459854 lockFactory=org.apache.lucene.store.NativeFSLockFactory@19b41a92),segFN=segments_4,generation=4,filenames=[segments_4]
[junit4:junit4]   2> 24221 T1064 C70 P22364 oasc.SolrDeletionPolicy.updateCommits newest commit = 4[segments_4]
[junit4:junit4]   2> 24222 T1064 C70 P22364 oass.SolrIndexSearcher.<init> Opening Searcher@67c9166e main
[junit4:junit4]   2> 24222 T1064 C70 P22364 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2>  C68_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:22357_, base_url=http://127.0.0.1:22357}
[junit4:junit4]   2> 24222 T1047 C68 P22357 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2>  C71_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:38389_, base_url=http://127.0.0.1:38389}
[junit4:junit4]   2> 24222 T1083 C71 P38389 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 24223 T1076 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@67c9166e main{StandardDirectoryReader(segments_3:1:nrt)}
[junit4:junit4]   2> 24224 T1083 C71 P38389 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@53d07a4 lockFactory=org.apache.lucene.store.NativeFSLockFactory@65f7beca),segFN=segments_4,generation=4,filenames=[segments_4]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@53d07a4 lockFactory=org.apache.lucene.store.NativeFSLockFactory@65f7beca),segFN=segments_5,generation=5,filenames=[segments_5]
[junit4:junit4]   2> 24224 T1047 C68 P22357 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@16340bd8 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1b29c110),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@16340bd8 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1b29c110),segFN=segments_3,generation=3,filenames=[segments_3]
[junit4:junit4]   2> 24224 T1064 C70 P22364 /update {waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 4
[junit4:junit4]   2> 24225 T1083 C71 P38389 oasc.SolrDeletionPolicy.updateCommits newest commit = 5[segments_5]
[junit4:junit4]   2> 24225 T1047 C68 P22357 oasc.SolrDeletionPolicy.updateCommits newest commit = 3[segments_3]
[junit4:junit4]   2> 24226 T1083 C71 P38389 oass.SolrIndexSearcher.<init> Opening Searcher@6dbffddf main
[junit4:junit4]   2> 24226 T1047 C68 P22357 oass.SolrIndexSearcher.<init> Opening Searcher@6dbffddf main
[junit4:junit4]   2> 24226 T1083 C71 P38389 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 24226 T1047 C68 P22357 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 24226 T1095 oasc.JmxMonitoredMap.put WARNING Failed to register info bean: searcher javax.management.InstanceNotFoundException: solr/collection1:type=searcher,id=org.apache.solr.search.SolrIndexSearcher
[junit4:junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1118)
[junit4:junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:433)
[junit4:junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:421)
[junit4:junit4]   2> 	at com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:541)
[junit4:junit4]   2> 	at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:138)
[junit4:junit4]   2> 	at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:51)
[junit4:junit4]   2> 	at org.apache.solr.search.SolrIndexSearcher.register(SolrIndexSearcher.java:239)
[junit4:junit4]   2> 	at org.apache.solr.core.SolrCore.registerSearcher(SolrCore.java:1763)
[junit4:junit4]   2> 	at org.apache.solr.core.SolrCore.access$000(SolrCore.java:129)
[junit4:junit4]   2> 	at org.apache.solr.core.SolrCore$7.call(SolrCore.java:1640)
[junit4:junit4]   2> 	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
[junit4:junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
[junit4:junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
[junit4:junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4:junit4]   2> 	at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 
[junit4:junit4]   2> 24227 T1059 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6dbffddf main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 24228 T1095 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6dbffddf main{StandardDirectoryReader(segments_4:1:nrt)}
[junit4:junit4]   2> 24228 T1047 C68 P22357 /update {waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 6
[junit4:junit4]   2> 24228 T1083 C71 P38389 /update {waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 6
[junit4:junit4]   2> 24229 T1033 C69 P22352 /update {waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 25
[junit4:junit4]   2> 24230 T999 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 24233 T1035 C69 P22352 REQ /select {tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0 
[junit4:junit4]   2> 24235 T1048 C68 P22357 REQ /select {tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 24236 T1065 C70 P22364 REQ /select {tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0 
[junit4:junit4]   2> 24238 T1084 C71 P38389 REQ /select {tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0 
[junit4:junit4]   2> 26244 T1011 C72 P22346 oass.SolrIndexSearcher.<init> Opening Searcher@700e4f96 realtime
[junit4:junit4]   2> 26245 T1011 C72 P22346 /update {wt=javabin&version=2} {deleteByQuery=*:* (-1429580953630539776)} 0 3
[junit4:junit4]   2> 26252 T1047 C68 P22357 /update {update.distrib=FROMLEADER&_version_=-1429580953634734080&update.from=http://127.0.0.1:22352/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1429580953634734080)} 0 1
[junit4:junit4]   2> 26252 T1064 C70 P22364 /update {update.distrib=FROMLEADER&_version_=-1429580953634734080&update.from=http://127.0.0.1:22352/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1429580953634734080)} 0 1
[junit4:junit4]   2> 26252 T1083 C71 P38389 /update {update.distrib=FROMLEADER&_version_=-1429580953634734080&update.from=http://127.0.0.1:22352/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1429580953634734080)} 0 1
[junit4:junit4]   2> 26253 T1033 C69 P22352 /update {wt=javabin&version=2} {deleteByQuery=*:* (-1429580953634734080)} 0 6
[junit4:junit4]   2> 26258 T1011 C72 P22346 /update {wt=javabin&version=2} {add=[0 (1429580953644171264)]} 0 2
[junit4:junit4]   2> 26264 T1064 C70 P22364 /update {distrib.from=http://127.0.0.1:22352/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1429580953648365568)]} 0 1
[junit4:junit4]   2> 26264 T1047 C68 P22357 /update {distrib.from=http://127.0.0.1:22352/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1429580953648365568)]} 0 1
[junit4:junit4]   2> 26265 T1083 C71 P38389 /update {distrib.from=http://127.0.0.1:22352/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1429580953648365568)]} 0 2
[junit4:junit4]   2> 26265 T1033 C69 P22352 /update {wt=javabin&version=2} {add=[0 (1429580953648365568)]} 0 5
[junit4:junit4]   2> 26269 T1011 C72 P22346 /update {wt=javabin&version=2} {add=[1 (1429580953656754176)]} 0 1
[junit4:junit4]   2> 26275 T1064 C70 P22364 /update {distrib.from=http://127.0.0.1:22352/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1429580953659899904)]} 0 1
[junit4:junit4]   2> 26275 T1047 C68 P22357 /update {distrib.from=http://127.0.0.1:22352/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1429580953659899904)]} 0 1
[junit4:junit4]   2> 26275 T1083 C71 P38389 /update {distrib.from=http://127.0.0.1:22352/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1429580953659899904)]} 0 1
[junit4:junit4]   2> 26276 T1033 C69 P22352 /update {wt=javabin&version=2} {add=[1 (1429580953659899904)]} 0 5
[junit4:junit4]   2> 26279 T1011 C72 P22346 /update {wt=javabin&version=2} {add=[2 (1429580953667239936)]} 0 1
[junit4:junit4]   2> 26285 T1064 C70 P22364 /update {distrib.from=http://127.0.0.1:22352/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[2 (1429580953670385664)]} 0 1
[junit4:junit4]   2> 26285 T1083 C71 P38389 /update {distrib.from=http://127.0.0.1:22352/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[2 (1429580953670385664)]} 0 1
[junit4:junit4]   2> 26285 T1033 C69 P22352 /update {test.distrib.skip.servers=http://127.0.0.1:22357/collection1/&wt=javabin&version=2} {add=[2 (1429580953670385664)]} 0 4
[junit4:junit4]   2> 26289 T1011 C72 P22346 /update {wt=javabin&version=2} {add=[3 (1429580953677725696)]} 0 1
[junit4:junit4]   2> 26295 T1083 C71 P38389 /update {distrib.from=http://127.0.0.1:22352/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[3 (1429580953680871424)]} 0 1
[junit4:junit4]   2> 26295 T1033 C69 P22352 /update {test.distrib.skip.servers=http://127.0.0.1:22357/collection1/&test.distrib.skip.servers=http://127.0.0.1:22364/collection1/&wt=javabin&version=2} {add=[3 (1429580953680871424)]} 0 4
[junit4:junit4]   2> 26298 T1011 C72 P22346 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 26301 T1011 C72 P22346 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@13f40fb4 lockFactory=org.apache.lucene.store.NativeFSLockFactory@776977a4),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@13f40fb4 lockFactory=org.apache.lucene.store.NativeFSLockFactory@776977a4),segFN=segments_3,generation=3,filenames=[_0_Lucene40_0.tim, _0.fnm, _0_dv.cfs, _0_nrm.cfs, _0_Lucene40_0.prx, _0_Lucene40_0.tip, _0_Lucene40_0.frq, _0_nrm.cfe, _0.fdx, _0.si, segments_3, _0.fdt, _0_dv.cfe]
[junit4:junit4]   2> 26302 T1011 C72 P22346 oasc.SolrDeletionPolicy.updateCommits newest commit = 3[_0_Lucene40_0.tim, _0.fnm, _0_dv.cfs, _0_nrm.cfs, _0_Lucene40_0.prx, _0_Lucene40_0.tip, _0_Lucene40_0.frq, _0_nrm.cfe, _0.fdx, _0.si, segments_3, _0.fdt, _0_dv.cfe]
[junit4:junit4]   2> 26303 T1011 C72 P22346 oass.SolrIndexSearcher.<init> Opening Searcher@48c52efb main
[junit4:junit4]   2> 26303 T1011 C72 P22346 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 26304 T1025 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@48c52efb main{StandardDirectoryReader(segments_3:3:nrt _0(5.0):C4)}
[junit4:junit4]   2> 26304 T1011 C72 P22346 /update {waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 6
[junit4:junit4]   2> 26307 T1033 C69 P22352 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 26310 T1033 C69 P22352 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=5
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@f22d615 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6bd7fbbd),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@f22d615 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6bd7fbbd),segFN=segments_3,generation=3,filenames=[segments_3]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@f22d615 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6bd7fbbd),segFN=segments_4,generation=4,filenames=[segments_4]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@f22d615 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6bd7fbbd),segFN=segments_5,generation=5,filenames=[segments_5]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@f22d615 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6bd7fbbd),segFN=segments_6,generation=6,filenames=[_0_Lucene40_0.tim, _0.fnm, _0_dv.cfs, _0_nrm.cfs, _0_Lucene40_0.prx, _0_Lucene40_0.tip, _0_Lucene40_0.frq, _0_nrm.cfe, _0.fdx, _0.si, segments_6, _0.fdt, _0_dv.cfe]
[junit4:junit4]   2> 26310 T1033 C69 P22352 oasc.SolrDeletionPolicy.updateCommits newest commit = 6[_0_Lucene40_0.tim, _0.fnm, _0_dv.cfs, _0_nrm.cfs, _0_Lucene40_0.prx, _0_Lucene40_0.tip, _0_Lucene40_0.frq, _0_nrm.cfe, _0.fdx, _0.si, segments_6, _0.fdt, _0_dv.cfe]
[junit4:junit4]   2> 26312 T1033 C69 P22352 oass.SolrIndexSearcher.<init> Opening Searcher@7c93b91f main
[junit4:junit4]   2> 26312 T1033 C69 P22352 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 26312 T1043 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7c93b91f main{StandardDirectoryReader(segments_6:3:nrt _0(5.0):C4)}
[junit4:junit4]   2> 26313 T1033 C69 P22352 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:22357/collection1/, StdNode: http://127.0.0.1:22364/collection1/, StdNode: http://127.0.0.1:38389/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4]   2> 26315 T1083 C71 P38389 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 26315 T1047 C68 P22357 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 26316 T1064 C70 P22364 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 26318 T1083 C71 P38389 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@53d07a4 lockFactory=org.apache.lucene.store.NativeFSLockFactory@65f7beca),segFN=segments_5,generation=5,filenames=[segments_5]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@53d07a4 lockFactory=org.apache.lucene.store.NativeFSLockFactory@65f7beca),segFN=segments_6,generation=6,filenames=[_0_Lucene40_0.tim, _0.fnm, _0_dv.cfs, _0_nrm.cfs, _0_Lucene40_0.prx, _0_Lucene40_0.tip, _0_Lucene40_0.frq, _0_nrm.cfe, _0.fdx, _0.si, segments_6, _0.fdt, _0_dv.cfe]
[junit4:junit4]   2> 26318 T1047 C68 P22357 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@16340bd8 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1b29c110),segFN=segments_3,generation=3,filenames=[segments_3]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@16340bd8 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1b29c110),segFN=segments_4,generation=4,filenames=[_0_Lucene40_0.tim, _0.fnm, _0_dv.cfs, _0_nrm.cfs, _0_Lucene40_0.prx, _0_Lucene40_0.tip, _0_Lucene40_0.frq, _0_nrm.cfe, _0.fdx, _0.si, _0.fdt, segments_4, _0_dv.cfe]
[junit4:junit4]   2> 26318 T1083 C71 P38389 oasc.SolrDeletionPolicy.updateCommits newest commit = 6[_0_Lucene40_0.tim, _0.fnm, _0_dv.cfs, _0_nrm.cfs, _0_Lucene40_0.prx, _0_Lucene40_0.tip, _0_Lucene40_0.frq, _0_nrm.cfe, _0.fdx, _0.si, segments_6, _0.fdt, _0_dv.cfe]
[junit4:junit4]   2> 26319 T1047 C68 P22357 oasc.SolrDeletionPolicy.updateCommits newest commit = 4[_0_Lucene40_0.tim, _0.fnm, _0_dv.cfs, _0_nrm.cfs, _0_Lucene40_0.prx, _0_Lucene40_0.tip, _0_Lucene40_0.frq, _0_nrm.cfe, _0.fdx, _0.si, _0.fdt, segments_4, _0_dv.cfe]
[junit4:junit4]   2> 26319 T1064 C70 P22364 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1a459854 lockFactory=org.apache.lucene.store.NativeFSLockFactory@19b41a92),segFN=segments_4,generation=4,filenames=[segments_4]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1a459854 lockFactory=org.apache.lucene.store.NativeFSLockFactory@19b41a92),segFN=segments_5,generation=5,filenames=[_0_Lucene40_0.tim, _0.fnm, _0_dv.cfs, _0_nrm.cfs, _0_Lucene40_0.prx, _0_Lucene40_0.tip, _0_Lucene40_0.frq, _0_nrm.cfe, _0.fdx, segments_5, _0.si, _0.fdt, _0_dv.cfe]
[junit4:junit4]   2> 26320 T1064 C70 P22364 oasc.SolrDeletionPolicy.updateCommits newest commit = 5[_0_Lucene40_0.tim, _0.fnm, _0_dv.cfs, _0_nrm.cfs, _0_Lucene40_0.prx, _0_Lucene40_0.tip, _0_Lucene40_0.frq, _0_nrm.cfe, _0.fdx, segments_5, _0.si, _0.fdt, _0_dv.cfe]
[junit4:junit4]   2> 26320 T1083 C71 P38389 oass.SolrIndexSearcher.<init> Opening Searcher@6b3c996b main
[junit4:junit4]   2> 26320 T1047 C68 P22357 oass.SolrIndexSearcher.<init> Opening Searcher@c703fb9 main
[junit4:junit4]   2> 26320 T1083 C71 P38389 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 26321 T1047 C68 P22357 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 26321 T1095 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6b3c996b main{StandardDirectoryReader(segments_6:3:nrt _0(5.0):C4)}
[junit4:junit4]   2> 26321 T1064 C70 P22364 oass.SolrIndexSearcher.<init> Opening Searcher@48159a63 main
[junit4:junit4]   2> 26321 T1059 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@c703fb9 main{StandardDirectoryReader(segments_4:3:nrt _0(5.0):C2)}
[junit4:junit4]   2> 26321 T1083 C71 P38389 /update {waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 6
[junit4:junit4]   2> 26322 T1064 C70 P22364 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 26322 T1047 C68 P22357 /update {waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 7
[junit4:junit4]   2> 26322 T1076 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@48159a63 main{StandardDirectoryReader(segments_5:3:nrt _0(5.0):C3)}
[junit4:junit4]   2> 26323 T1064 C70 P22364 /update {waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 7
[junit4:junit4]   2> 26323 T1033 C69 P22352 /update {waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 16
[junit4:junit4]   2> 26324 T999 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):230
[junit4:junit4]   2> 26325 T999 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 26326 T999 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4]   2> 26328 T1035 C69 P22352 REQ /select {tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0 
[junit4:junit4]   2> 26331 T1048 C68 P22357 REQ /select {tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=2 status=0 QTime=0 
[junit4:junit4]   2> 26334 T1065 C70 P22364 REQ /select {tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=3 status=0 QTime=0 
[junit4:junit4]   2> 26337 T1084 C71 P38389 REQ /select {tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0 
[junit4:junit4]   2> 26338 T999 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 26350 T1066 oasha.CollectionsHandler.handleSyncShardAction Syncing shard : shard=shard1&action=SYNCSHARD&collection=collection1&wt=javabin&version=2
[junit4:junit4]   2> 26351 T1066 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 26362 T1036 oasha.CoreAdminHandler.handleRequestSyncAction I have been requested to sync up my shard
[junit4:junit4]   2> 26363 T1036 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:22352/collection1/
[junit4:junit4]   2> 26363 T1036 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:22352 START replicas=[http://127.0.0.1:22357/collection1/, http://127.0.0.1:22364/collection1/, http://127.0.0.1:38389/collection1/] nUpdates=100
[junit4:junit4]   2> 26365 T1049 C68 P22357 REQ /get {getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0 
[junit4:junit4]   2> 26365 T1067 C70 P22364 REQ /get {getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0 
[junit4:junit4]   2> 26366 T1085 C71 P38389 REQ /get {getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1 
[junit4:junit4]   2> 26366 T1036 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:22352  Received 3 versions from 127.0.0.1:22357/collection1/
[junit4:junit4]   2> 26366 T1036 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:22352  Our versions are newer. ourLowThreshold=1429580953634734080 otherHigh=1429580953659899904
[junit4:junit4]   2> 26367 T1036 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:22352  Received 4 versions from 127.0.0.1:22364/collection1/
[junit4:junit4]   2> 26367 T1036 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:22352  Our versions are newer. ourLowThreshold=1429580953634734080 otherHigh=1429580953670385664
[junit4:junit4]   2> 26367 T1036 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:22352  Received 5 versions from 127.0.0.1:38389/collection1/
[junit4:junit4]   2> 26368 T1036 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:22352  Our versions are newer. ourLowThreshold=1429580953634734080 otherHigh=1429580953670385664
[junit4:junit4]   2> 26368 T1036 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:22352 DONE. sync succeeded
[junit4:junit4]   2> 26368 T1036 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 26369 T1036 oasc.SyncStrategy.syncToMe http://127.0.0.1:22352/collection1/: try and ask http://127.0.0.1:22357/collection1/ to sync
[junit4:junit4]   2> 26369 T1036 oasc.SyncStrategy.syncToMe http://127.0.0.1:22352/collection1/: try and ask http://127.0.0.1:22364/collection1/ to sync
[junit4:junit4]   2> 26370 T1036 oasc.SyncStrategy.syncToMe http://127.0.0.1:22352/collection1/: try and ask http://127.0.0.1:38389/collection1/ to sync
[junit4:junit4]   2> 26370 T1050 C68 P22357 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:22357 START replicas=[http://127.0.0.1:22352/collection1/] nUpdates=100
[junit4:junit4]   2> 26371 T1068 C70 P22364 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:22364 START replicas=[http://127.0.0.1:22352/collection1/] nUpdates=100
[junit4:junit4]   2> 26371 T1086 C71 P38389 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:38389 START replicas=[http://127.0.0.1:22352/collection1/] nUpdates=100
[junit4:junit4]   2> 26372 T1037 C69 P22352 REQ /get {getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0 
[junit4:junit4]   2> 26373 T1031 C69 P22352 REQ /get {getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1 
[junit4:junit4]   2> 26373 T1068 C70 P22364 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:22364  Received 5 versions from 127.0.0.1:22352/collection1/
[junit4:junit4]   2> 26373 T1032 C69 P22352 REQ /get {getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0 
[junit4:junit4]   2> 26373 T1086 C71 P38389 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:38389  Received 5 versions from 127.0.0.1:22352/collection1/
[junit4:junit4]   2> 26373 T1068 C70 P22364 oasu.PeerSync.requestUpdates PeerSync: core=collection1 url=http://127.0.0.1:22364 Requesting updates from 

[...truncated too long message...]

9 oazs.PrepRequestProcessor.shutdown Shutting down
[junit4:junit4]   2> 139753 T1004 oazs.PrepRequestProcessor.run PrepRequestProcessor exited loop!
[junit4:junit4]   2> 139753 T999 oazs.SyncRequestProcessor.shutdown Shutting down
[junit4:junit4]   2> 139754 T1003 oazs.SyncRequestProcessor.run SyncRequestProcessor exited!
[junit4:junit4]   2> 139755 T999 oazs.FinalRequestProcessor.shutdown shutdown of request processor complete
[junit4:junit4]   2> 139757 T999 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:22341 22341
[junit4:junit4]   2> 139890 T1001 oazs.NIOServerCnxnFactory.run NIOServerCnxn factory exited run method
[junit4:junit4]   2> 139891 T999 oazs.ZooKeeperServer.shutdown shutting down
[junit4:junit4]   2> 139892 T999 oazs.SessionTrackerImpl.shutdown Shutting down
[junit4:junit4]   2> 139892 T999 oazs.PrepRequestProcessor.shutdown Shutting down
[junit4:junit4]   2> 139893 T999 oazs.SyncRequestProcessor.shutdown Shutting down
[junit4:junit4]   2> 139893 T999 oazs.FinalRequestProcessor.shutdown shutdown of request processor complete
[junit4:junit4]   1> client port:0.0.0.0/0.0.0.0:0
[junit4:junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=SyncSliceTest -Dtests.method=testDistribSearch -Dtests.seed=A1E6A0DE354761F0 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=es_PR -Dtests.timezone=America/Whitehorse -Dtests.file.encoding=US-ASCII
[junit4:junit4] FAILURE  140s J0 | SyncSliceTest.testDistribSearch <<<
[junit4:junit4]    > Throwable #1: java.lang.AssertionError: Test Setup Failure: shard1 should have just been set up to be inconsistent - but it's still consistent. Leader:http://127.0.0.1:22357/collection1skip list:[CloudJettyRunner [url=http://127.0.0.1:38389/collection1], CloudJettyRunner [url=http://127.0.0.1:38389/collection1]]
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([A1E6A0DE354761F0:20002EC6421801CC]:0)
[junit4:junit4]    > 	at org.junit.Assert.fail(Assert.java:93)
[junit4:junit4]    > 	at org.junit.Assert.assertTrue(Assert.java:43)
[junit4:junit4]    > 	at org.junit.Assert.assertNotNull(Assert.java:526)
[junit4:junit4]    > 	at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:212)
[junit4:junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:806)
[junit4:junit4]    > 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit4:junit4]    > 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
[junit4:junit4]    > 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit4:junit4]    > 	at java.lang.reflect.Method.invoke(Method.java:616)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
[junit4:junit4]    > 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
[junit4:junit4]    > 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
[junit4:junit4]    > 	at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 139942 T999 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4]   2> 139949 T998 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
[junit4:junit4]   2> 140543 T1002 oazs.SessionTrackerImpl.run SessionTrackerImpl exited loop!
[junit4:junit4]   2> NOTE: test params are: codec=Lucene40, sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=es_PR, timezone=America/Whitehorse
[junit4:junit4]   2> NOTE: FreeBSD 9.0-RELEASE amd64/Sun Microsystems Inc. 1.6.0_32 (64-bit)/cpus=16,threads=5,free=61495216,total=452198400
[junit4:junit4]   2> NOTE: All tests run in this JVM: [TestLFUCache, AutoCommitTest, TestStressVersions, SolrCoreTest, SliceStateTest, SolrInfoMBeanTest, ZkCLITest, TestReversedWildcardFilterFactory, PluginInfoTest, TermVectorComponentTest, TestPseudoReturnFields, TestSurroundQueryParser, SystemInfoHandlerTest, FieldMutatingUpdateProcessorTest, MBeansHandlerTest, DistributedQueryElevationComponentTest, SliceStateUpdateTest, ClusterStateUpdateTest, BasicDistributedZkTest, QueryElevationComponentTest, DateFieldTest, SchemaVersionSpecificBehaviorTest, CacheHeaderTest, UpdateRequestProcessorFactoryTest, BadComponentTest, TestSolrQueryParser, SolrIndexSplitterTest, TestLRUCache, TestDocSet, TestFastLRUCache, TestFaceting, TestJoin, CoreAdminHandlerTest, TestSort, StatelessScriptUpdateProcessorFactoryTest, DirectSolrConnectionTest, TestSolrIndexConfig, AlternateDirectoryTest, QueryEqualityTest, LukeRequestHandlerTest, SyncSliceTest]
[junit4:junit4] Completed on J0 in 140.73s, 1 test, 1 failure <<< FAILURES!

[...truncated 515 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/build.xml:381: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/build.xml:361: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build.xml:183: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/common-build.xml:449: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/lucene/common-build.xml:1213: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/lucene/common-build.xml:877: There were test failures: 272 suites, 1144 tests, 1 failure, 27 ignored (8 assumptions)

Total time: 57 minutes 13 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-Tests-trunk-Java6 - Build # 16005 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-trunk-Java6/16005/

1 tests failed.
FAILED:  org.apache.solr.cloud.BasicDistributedZk2Test.testDistribSearch

Error Message:
No registered leader was found, collection:collection1 slice:shard1

Stack Trace:
org.apache.solr.common.SolrException: No registered leader was found, collection:collection1 slice:shard1
	at __randomizedtesting.SeedInfo.seed([2A7F3D9CF10D9610:AB99B3848652F62C]:0)
	at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:484)
	at org.apache.solr.cloud.BasicDistributedZk2Test.brindDownShardIndexSomeDocsAndRecover(BasicDistributedZk2Test.java:291)
	at org.apache.solr.cloud.BasicDistributedZk2Test.doTest(BasicDistributedZk2Test.java:116)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:806)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:679)




Build Log:
[...truncated 8576 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.BasicDistributedZk2Test
[junit4:junit4]   2> 0 T1648 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /idf/
[junit4:junit4]   2> 3 T1648 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4]   2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1363332425525
[junit4:junit4]   2> 4 T1648 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 5 T1649 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4]   2> 8 T1649 oazs.ZooKeeperServer.setTickTime tickTime set to 1000
[junit4:junit4]   2> 8 T1649 oazs.ZooKeeperServer.setMinSessionTimeout minSessionTimeout set to -1
[junit4:junit4]   2> 8 T1649 oazs.ZooKeeperServer.setMaxSessionTimeout maxSessionTimeout set to -1
[junit4:junit4]   2> 8 T1649 oazs.NIOServerCnxnFactory.configure binding to port 0.0.0.0/0.0.0.0:0
[junit4:junit4]   2> 9 T1649 oazsp.FileTxnSnapLog.save Snapshotting: 0x0 to /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-1363332425525/zookeeper/server1/data/version-2/snapshot.0
[junit4:junit4]   2> 105 T1648 oasc.ZkTestServer.run start zk server on port:47880
[junit4:junit4]   2> 105 T1648 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:47880 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@67b58dfa
[junit4:junit4]   2> 106 T1648 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 107 T1654 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:47880. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 107 T1654 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:47880, initiating session
[junit4:junit4]   2> 107 T1650 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:15814
[junit4:junit4]   2> 108 T1650 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:15814
[junit4:junit4]   2> 108 T1652 oazsp.FileTxnLog.append Creating new log file: log.1
[junit4:junit4]   2> 110 T1652 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6cf1e73b0000 with negotiated timeout 10000 for client /140.211.11.196:15814
[junit4:junit4]   2> 110 T1654 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:47880, sessionid = 0x13d6cf1e73b0000, negotiated timeout = 10000
[junit4:junit4]   2> 110 T1655 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@67b58dfa name:ZooKeeperConnection Watcher:127.0.0.1:47880 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 111 T1648 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 111 T1648 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4]   2> 114 T1653 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d6cf1e73b0000
[junit4:junit4]   2> 121 T1655 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 121 T1650 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:15814 which had sessionid 0x13d6cf1e73b0000
[junit4:junit4]   2> 121 T1648 oaz.ZooKeeper.close Session: 0x13d6cf1e73b0000 closed
[junit4:junit4]   2> 122 T1648 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:47880/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@30172311
[junit4:junit4]   2> 122 T1648 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 123 T1656 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:47880. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 123 T1656 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:47880, initiating session
[junit4:junit4]   2> 123 T1650 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:63221
[junit4:junit4]   2> 124 T1650 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:63221
[junit4:junit4]   2> 124 T1652 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6cf1e73b0001 with negotiated timeout 10000 for client /140.211.11.196:63221
[junit4:junit4]   2> 125 T1656 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:47880, sessionid = 0x13d6cf1e73b0001, negotiated timeout = 10000
[junit4:junit4]   2> 125 T1657 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@30172311 name:ZooKeeperConnection Watcher:127.0.0.1:47880/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 125 T1648 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 125 T1648 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4]   2> 136 T1648 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4]   2> 139 T1648 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4]   2> 141 T1648 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4]   2> 144 T1648 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 145 T1648 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 158 T1648 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4]   2> 159 T1648 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4]   2> 275 T1648 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4]   2> 276 T1648 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4]   2> 279 T1648 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4]   2> 279 T1648 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4]   2> 296 T1648 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4]   2> 296 T1648 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4]   2> 299 T1648 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 300 T1648 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 302 T1648 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 303 T1648 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 305 T1648 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 306 T1648 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 313 T1648 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4]   2> 314 T1648 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4]   2> 316 T1653 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d6cf1e73b0001
[junit4:junit4]   2> 317 T1657 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 318 T1650 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:63221 which had sessionid 0x13d6cf1e73b0001
[junit4:junit4]   2> 317 T1648 oaz.ZooKeeper.close Session: 0x13d6cf1e73b0001 closed
[junit4:junit4]   2> 513 T1648 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 516 T1648 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:65311
[junit4:junit4]   2> 517 T1648 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 517 T1648 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 518 T1648 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1363332425839
[junit4:junit4]   2> 518 T1648 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1363332425839/solr.xml
[junit4:junit4]   2> 519 T1648 oasc.CoreContainer.<init> New CoreContainer 1486608012
[junit4:junit4]   2> 519 T1648 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1363332425839/'
[junit4:junit4]   2> 520 T1648 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1363332425839/'
[junit4:junit4]   2> 575 T1648 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 575 T1648 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 576 T1648 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 576 T1648 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 577 T1648 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 577 T1648 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 578 T1648 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 578 T1648 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 578 T1648 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 579 T1648 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 601 T1648 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 619 T1648 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:47880/solr
[junit4:junit4]   2> 619 T1648 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 620 T1648 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:47880 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@5e0079f5
[junit4:junit4]   2> 621 T1648 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 621 T1667 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:47880. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 622 T1667 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:47880, initiating session
[junit4:junit4]   2> 622 T1650 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:32043
[junit4:junit4]   2> 623 T1650 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:32043
[junit4:junit4]   2> 624 T1652 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6cf1e73b0002 with negotiated timeout 20000 for client /140.211.11.196:32043
[junit4:junit4]   2> 624 T1667 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:47880, sessionid = 0x13d6cf1e73b0002, negotiated timeout = 20000
[junit4:junit4]   2> 624 T1668 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5e0079f5 name:ZooKeeperConnection Watcher:127.0.0.1:47880 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 625 T1648 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 626 T1653 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d6cf1e73b0002
[junit4:junit4]   2> 634 T1668 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 634 T1650 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:32043 which had sessionid 0x13d6cf1e73b0002
[junit4:junit4]   2> 634 T1648 oaz.ZooKeeper.close Session: 0x13d6cf1e73b0002 closed
[junit4:junit4]   2> 635 T1648 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 643 T1648 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:47880/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@2f5c36fb
[junit4:junit4]   2> 644 T1648 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 644 T1669 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:47880. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 644 T1669 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:47880, initiating session
[junit4:junit4]   2> 644 T1650 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:50845
[junit4:junit4]   2> 645 T1650 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:50845
[junit4:junit4]   2> 646 T1652 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6cf1e73b0003 with negotiated timeout 20000 for client /140.211.11.196:50845
[junit4:junit4]   2> 646 T1669 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:47880, sessionid = 0x13d6cf1e73b0003, negotiated timeout = 20000
[junit4:junit4]   2> 646 T1670 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2f5c36fb name:ZooKeeperConnection Watcher:127.0.0.1:47880/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 646 T1648 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 653 T1653 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6cf1e73b0003 type:create cxid:0x2 zxid:0x1a txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 654 T1648 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4]   2> 657 T1648 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:65311_idf
[junit4:junit4]   2> 657 T1653 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6cf1e73b0003 type:delete cxid:0x8 zxid:0x1c txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:65311_idf Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:65311_idf
[junit4:junit4]   2> 658 T1648 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:65311_idf
[junit4:junit4]   2> 667 T1648 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4]   2> 676 T1653 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6cf1e73b0003 type:delete cxid:0x16 zxid:0x21 txntype:-1 reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4]   2> 678 T1648 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 687 T1648 oasc.Overseer.start Overseer (id=89347353839599619-127.0.0.1:65311_idf-n_0000000000) starting
[junit4:junit4]   2> 688 T1653 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6cf1e73b0003 type:create cxid:0x1b zxid:0x23 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 689 T1653 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6cf1e73b0003 type:create cxid:0x1c zxid:0x24 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 690 T1653 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6cf1e73b0003 type:create cxid:0x1d zxid:0x25 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 699 T1653 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6cf1e73b0003 type:create cxid:0x1e zxid:0x26 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 700 T1672 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4]   2> 701 T1648 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4]   2> 711 T1648 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4]   2> 712 T1648 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 714 T1671 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 719 T1673 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1363332425839/collection1
[junit4:junit4]   2> 719 T1673 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4]   2> 719 T1673 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 719 T1673 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4]   2> 720 T1673 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1363332425839/collection1/'
[junit4:junit4]   2> 721 T1673 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1363332425839/collection1/lib/README' to classloader
[junit4:junit4]   2> 722 T1673 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1363332425839/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 754 T1673 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 792 T1673 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 893 T1673 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 897 T1673 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 1316 T1673 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 1324 T1673 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 1327 T1673 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 1339 T1673 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 1343 T1673 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 1346 T1673 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 1347 T1673 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 1347 T1673 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 1348 T1673 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 1349 T1673 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 1349 T1673 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 1349 T1673 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 1349 T1673 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1363332425839/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363332425525/control/data/
[junit4:junit4]   2> 1349 T1673 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@962522b
[junit4:junit4]   2> 1350 T1673 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 1350 T1673 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363332425525/control/data forceNew: false
[junit4:junit4]   2> 1350 T1673 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1363332425525/control/data/index/
[junit4:junit4]   2> 1350 T1673 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1363332425525/control/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 1351 T1673 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363332425525/control/data/index forceNew: false
[junit4:junit4]   2> 1353 T1673 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@496cff2a lockFactory=org.apache.lucene.store.NativeFSLockFactory@48303e79),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 1353 T1673 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 1355 T1673 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 1355 T1673 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 1355 T1673 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 1356 T1673 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 1356 T1673 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 1356 T1673 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 1356 T1673 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 1357 T1673 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 1357 T1673 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 1362 T1673 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 1366 T1673 oass.SolrIndexSearcher.<init> Opening Searcher@3ef2de86 main
[junit4:junit4]   2> 1366 T1673 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363332425525/control/data/tlog
[junit4:junit4]   2> 1367 T1673 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 1367 T1673 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 1369 T1674 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3ef2de86 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 1370 T1673 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 1370 T1673 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 1370 T1653 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6cf1e73b0003 type:create cxid:0x55 zxid:0x2a txntype:-1 reqpath:n/a Error Path:/solr/overseer/queue Error:KeeperErrorCode = NoNode for /solr/overseer/queue
[junit4:junit4]   2> 2218 T1671 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2218 T1671 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"1",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"control_collection",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:65311_idf",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:65311/idf"}
[junit4:junit4]   2> 2219 T1671 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4]   2> 2219 T1671 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 2219 T1653 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6cf1e73b0003 type:create cxid:0x5d zxid:0x2d txntype:-1 reqpath:n/a Error Path:/solr/overseer/queue-work Error:KeeperErrorCode = NoNode for /solr/overseer/queue-work
[junit4:junit4]   2> 2240 T1670 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 2388 T1673 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 2388 T1673 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:65311/idf collection:control_collection shard:shard1
[junit4:junit4]   2> 2389 T1673 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4]   2> 2406 T1653 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6cf1e73b0003 type:delete cxid:0x77 zxid:0x37 txntype:-1 reqpath:n/a Error Path:/solr/collections/control_collection/leaders Error:KeeperErrorCode = NoNode for /solr/collections/control_collection/leaders
[junit4:junit4]   2> 2407 T1673 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 2407 T1653 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6cf1e73b0003 type:create cxid:0x78 zxid:0x38 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 2409 T1673 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 2409 T1673 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 2409 T1673 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:65311/idf/collection1/
[junit4:junit4]   2> 2409 T1673 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 2409 T1673 oasc.SyncStrategy.syncToMe http://127.0.0.1:65311/idf/collection1/ has no replicas
[junit4:junit4]   2> 2409 T1673 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:65311/idf/collection1/
[junit4:junit4]   2> 2410 T1673 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4]   2> 2413 T1653 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6cf1e73b0003 type:create cxid:0x82 zxid:0x3c txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 3743 T1671 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 3791 T1670 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 3843 T1673 oasc.ZkController.register We are http://127.0.0.1:65311/idf/collection1/ and leader is http://127.0.0.1:65311/idf/collection1/
[junit4:junit4]   2> 3843 T1673 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:65311/idf
[junit4:junit4]   2> 3843 T1673 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 3843 T1673 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 3844 T1673 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 3846 T1673 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 3847 T1648 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 3847 T1648 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 3848 T1648 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 3863 T1648 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4]   2> 3864 T1648 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:47880/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@48659e2e
[junit4:junit4]   2> 3865 T1648 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 3865 T1676 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:47880. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 3866 T1676 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:47880, initiating session
[junit4:junit4]   2> 3866 T1650 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:47926
[junit4:junit4]   2> 3866 T1650 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:47926
[junit4:junit4]   2> 3867 T1652 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6cf1e73b0004 with negotiated timeout 10000 for client /140.211.11.196:47926
[junit4:junit4]   2> 3867 T1676 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:47880, sessionid = 0x13d6cf1e73b0004, negotiated timeout = 10000
[junit4:junit4]   2> 3868 T1677 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@48659e2e name:ZooKeeperConnection Watcher:127.0.0.1:47880/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 3868 T1648 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 3869 T1648 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 3871 T1648 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4]   2> 4069 T1648 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 4071 T1648 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:57179
[junit4:junit4]   2> 4072 T1648 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 4072 T1648 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 4073 T1648 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363332429392
[junit4:junit4]   2> 4073 T1648 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363332429392/solr.xml
[junit4:junit4]   2> 4074 T1648 oasc.CoreContainer.<init> New CoreContainer 826063978
[junit4:junit4]   2> 4074 T1648 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363332429392/'
[junit4:junit4]   2> 4075 T1648 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363332429392/'
[junit4:junit4]   2> 4123 T1648 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 4124 T1648 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 4124 T1648 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 4124 T1648 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 4125 T1648 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 4125 T1648 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 4126 T1648 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 4126 T1648 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 4127 T1648 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 4127 T1648 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 4148 T1648 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 4168 T1648 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:47880/solr
[junit4:junit4]   2> 4168 T1648 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 4169 T1648 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:47880 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@43f3638e
[junit4:junit4]   2> 4170 T1648 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 4170 T1687 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:47880. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 4171 T1687 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:47880, initiating session
[junit4:junit4]   2> 4171 T1650 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:64328
[junit4:junit4]   2> 4171 T1650 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:64328
[junit4:junit4]   2> 4173 T1652 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6cf1e73b0005 with negotiated timeout 20000 for client /140.211.11.196:64328
[junit4:junit4]   2> 4173 T1687 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:47880, sessionid = 0x13d6cf1e73b0005, negotiated timeout = 20000
[junit4:junit4]   2> 4174 T1688 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@43f3638e name:ZooKeeperConnection Watcher:127.0.0.1:47880 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 4174 T1648 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 4175 T1653 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d6cf1e73b0005
[junit4:junit4]   2> 4188 T1688 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 4188 T1650 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:64328 which had sessionid 0x13d6cf1e73b0005
[junit4:junit4]   2> 4187 T1648 oaz.ZooKeeper.close Session: 0x13d6cf1e73b0005 closed
[junit4:junit4]   2> 4188 T1648 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 4196 T1648 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:47880/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@6d49c28e
[junit4:junit4]   2> 4197 T1648 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 4197 T1689 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:47880. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 4197 T1689 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:47880, initiating session
[junit4:junit4]   2> 4198 T1650 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:48097
[junit4:junit4]   2> 4198 T1650 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:48097
[junit4:junit4]   2> 4199 T1652 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6cf1e73b0006 with negotiated timeout 20000 for client /140.211.11.196:48097
[junit4:junit4]   2> 4199 T1689 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:47880, sessionid = 0x13d6cf1e73b0006, negotiated timeout = 20000
[junit4:junit4]   2> 4199 T1690 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6d49c28e name:ZooKeeperConnection Watcher:127.0.0.1:47880/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 4200 T1648 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 4200 T1653 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6cf1e73b0006 type:create cxid:0x1 zxid:0x4a txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 4202 T1653 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6cf1e73b0006 type:create cxid:0x2 zxid:0x4b txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 4210 T1648 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 5220 T1648 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57179_idf
[junit4:junit4]   2> 5221 T1653 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6cf1e73b0006 type:delete cxid:0xb zxid:0x4c txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:57179_idf Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:57179_idf
[junit4:junit4]   2> 5223 T1648 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:57179_idf
[junit4:junit4]   2> 5226 T1690 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 5226 T1677 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 5226 T1670 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 5227 T1670 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 5241 T1691 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363332429392/collection1
[junit4:junit4]   2> 5241 T1691 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 5242 T1691 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 5242 T1691 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 5243 T1691 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363332429392/collection1/'
[junit4:junit4]   2> 5244 T1691 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363332429392/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 5244 T1691 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363332429392/collection1/lib/README' to classloader
[junit4:junit4]   2> 5280 T1691 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 5304 T1671 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 5305 T1671 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"1",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"control_collection",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:65311_idf",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:65311/idf"}
[junit4:junit4]   2> 5318 T1690 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 5318 T1677 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 5318 T1670 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 5326 T1691 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 5428 T1691 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 5434 T1691 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 6045 T1691 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 6060 T1691 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 6065 T1691 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 6077 T1691 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 6082 T1691 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 6085 T1691 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 6086 T1691 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 6087 T1691 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 6087 T1691 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 6088 T1691 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 6088 T1691 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 6088 T1691 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 6089 T1691 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363332429392/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363332425525/jetty1/
[junit4:junit4]   2> 6089 T1691 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@962522b
[junit4:junit4]   2> 6089 T1691 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 6090 T1691 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363332425525/jetty1 forceNew: false
[junit4:junit4]   2> 6090 T1691 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1363332425525/jetty1/index/
[junit4:junit4]   2> 6090 T1691 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1363332425525/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 6091 T1691 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363332425525/jetty1/index forceNew: false
[junit4:junit4]   2> 6093 T1691 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@45c8fad1 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7eeb6440),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 6093 T1691 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 6096 T1691 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 6096 T1691 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 6096 T1691 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 6097 T1691 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 6097 T1691 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 6098 T1691 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 6098 T1691 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 6098 T1691 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 6099 T1691 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 6105 T1691 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 6111 T1691 oass.SolrIndexSearcher.<init> Opening Searcher@662523bc main
[junit4:junit4]   2> 6111 T1691 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363332425525/jetty1/tlog
[junit4:junit4]   2> 6112 T1691 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 6112 T1691 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 6116 T1692 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@662523bc main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 6117 T1691 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 6118 T1691 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 6830 T1671 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 6831 T1671 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"2",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:57179_idf",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:57179/idf"}
[junit4:junit4]   2> 6831 T1671 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 2
[junit4:junit4]   2> 6831 T1671 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 6847 T1690 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 6847 T1677 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 6848 T1670 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 7122 T1691 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 7122 T1691 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:57179/idf collection:collection1 shard:shard1
[junit4:junit4]   2> 7123 T1691 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 7174 T1653 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6cf1e73b0006 type:delete cxid:0x48 zxid:0x5c txntype:-1 reqpath:n/a Error Path:/solr/collections/collection1/leaders Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders
[junit4:junit4]   2> 7175 T1691 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 7175 T1653 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6cf1e73b0006 type:create cxid:0x49 zxid:0x5d txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 7185 T1691 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 7185 T1691 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 7186 T1691 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:57179/idf/collection1/
[junit4:junit4]   2> 7186 T1691 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 7186 T1691 oasc.SyncStrategy.syncToMe http://127.0.0.1:57179/idf/collection1/ has no replicas
[junit4:junit4]   2> 7186 T1691 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:57179/idf/collection1/
[junit4:junit4]   2> 7186 T1691 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4]   2> 7189 T1653 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6cf1e73b0006 type:create cxid:0x53 zxid:0x61 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 8352 T1671 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 8383 T1670 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 8383 T1690 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 8383 T1677 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 8422 T1691 oasc.ZkController.register We are http://127.0.0.1:57179/idf/collection1/ and leader is http://127.0.0.1:57179/idf/collection1/
[junit4:junit4]   2> 8422 T1691 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:57179/idf
[junit4:junit4]   2> 8422 T1691 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 8422 T1691 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 8422 T1691 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 8424 T1691 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 8425 T1648 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 8426 T1648 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 8426 T1648 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 8634 T1648 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 8637 T1648 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:18313
[junit4:junit4]   2> 8637 T1648 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 8638 T1648 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 8638 T1648 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1363332433963
[junit4:junit4]   2> 8639 T1648 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1363332433963/solr.xml
[junit4:junit4]   2> 8639 T1648 oasc.CoreContainer.<init> New CoreContainer 1170924492
[junit4:junit4]   2> 8640 T1648 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1363332433963/'
[junit4:junit4]   2> 8640 T1648 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1363332433963/'
[junit4:junit4]   2> 8695 T1648 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 8696 T1648 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 8696 T1648 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 8697 T1648 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 8697 T1648 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 8698 T1648 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 8698 T1648 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 8699 T1648 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 8699 T1648 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 8699 T1648 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 8722 T1648 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 8742 T1648 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:47880/solr
[junit4:junit4]   2> 8743 T1648 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 8743 T1648 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:47880 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@6152d6dd
[junit4:junit4]   2> 8744 T1648 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 8745 T1703 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:47880. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 8746 T1703 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:47880, initiating session
[junit4:junit4]   2> 8746 T1650 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:51634
[junit4:junit4]   2> 8746 T1650 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:51634
[junit4:junit4]   2> 8748 T1652 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6cf1e73b0007 with negotiated timeout 20000 for client /140.211.11.196:51634
[junit4:junit4]   2> 8748 T1703 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:47880, sessionid = 0x13d6cf1e73b0007, negotiated timeout = 20000
[junit4:junit4]   2> 8749 T1704 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6152d6dd name:ZooKeeperConnection Watcher:127.0.0.1:47880 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 8749 T1648 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 8750 T1653 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d6cf1e73b0007
[junit4:junit4]   2> 8764 T1704 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 8764 T1650 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:51634 which had sessionid 0x13d6cf1e73b0007
[junit4:junit4]   2> 8764 T1648 oaz.ZooKeeper.close Session: 0x13d6cf1e73b0007 closed
[junit4:junit4]   2> 8764 T1648 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 8778 T1648 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:47880/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@658a19e5
[junit4:junit4]   2> 8779 T1648 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 8779 T1705 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:47880. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 8779 T1705 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:47880, initiating session
[junit4:junit4]   2> 8780 T1650 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:64402
[junit4:junit4]   2> 8780 T1650 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:64402
[junit4:junit4]   2> 8781 T1652 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6cf1e73b0008 with negotiated timeout 20000 for client /140.211.11.196:64402
[junit4:junit4]   2> 8781 T1705 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:47880, sessionid = 0x13d6cf1e73b0008, negotiated timeout = 20000
[junit4:junit4]   2> 8782 T1706 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@658a19e5 name:ZooKeeperConnection Watcher:127.0.0.1:47880/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 8782 T1648 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 8783 T1653 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6cf1e73b0008 type:create cxid:0x1 zxid:0x6e txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 8784 T1653 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6cf1e73b0008 type:create cxid:0x2 zxid:0x6f txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 8793 T1648 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 9796 T1648 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:18313_idf
[junit4:junit4]   2> 9798 T1653 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6cf1e73b0008 type:delete cxid:0xb zxid:0x70 txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:18313_idf Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:18313_idf
[junit4:junit4]   2> 9799 T1648 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:18313_idf
[junit4:junit4]   2> 9802 T1670 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 9803 T1677 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 9803 T1670 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 9803 T1690 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 9803 T1677 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 9803 T1706 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 9803 T1690 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 9825 T1707 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1363332433963/collection1
[junit4:junit4]   2> 9825 T1707 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 9826 T1707 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 9826 T1707 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 9827 T1707 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1363332433963/collection1/'
[junit4:junit4]   2> 9829 T1707 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1363332433963/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 9829 T1707 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1363332433963/collection1/lib/README' to classloader
[junit4:junit4]   2> 9877 T1707 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 9913 T1671 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 9914 T1671 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"2",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:57179_idf",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:57179/idf"}
[junit4:junit4]   2> 9926 T1690 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 9926 T1677 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 9926 T1706 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 9926 T1670 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 9932 T1707 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 10033 T1707 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 10039 T1707 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 10622 T1707 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 10633 T1707 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 10637 T1707 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 10651 T1707 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 10656 T1707 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 10660 T1707 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 10660 T1707 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 10661 T1707 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 10661 T1707 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 10662 T1707 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 10662 T1707 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 10663 T1707 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 10663 T1707 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1363332433963/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363332425525/jetty2/
[junit4:junit4]   2> 10663 T1707 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@962522b
[junit4:junit4]   2> 10664 T1707 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 10664 T1707 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363332425525/jetty2 forceNew: false
[junit4:junit4]   2> 10664 T1707 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1363332425525/jetty2/index/
[junit4:junit4]   2> 10665 T1707 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1363332425525/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 10665 T1707 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363332425525/jetty2/index forceNew: false
[junit4:junit4]   2> 10668 T1707 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@46c101bf lockFactory=org.apache.lucene.store.NativeFSLockFactory@4510764a),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 10668 T1707 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 10670 T1707 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 10670 T1707 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 10671 T1707 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 10671 T1707 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 10672 T1707 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 10672 T1707 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 10672 T1707 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 10673 T1707 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 10673 T1707 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 10680 T1707 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 10686 T1707 oass.SolrIndexSearcher.<init> Opening Searcher@44f37b4 main
[junit4:junit4]   2> 10686 T1707 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363332425525/jetty2/tlog
[junit4:junit4]   2> 10687 T1707 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 10687 T1707 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 10691 T1708 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@44f37b4 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 10693 T1707 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 10693 T1707 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 11431 T1671 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 11432 T1671 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"3",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:18313_idf",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:18313/idf"}
[junit4:junit4]   2> 11432 T1671 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4]   2> 11432 T1671 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4:junit4]   2> 11448 T1706 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 11448 T1670 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 11448 T1677 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 11449 T1690 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 11696 T1707 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 11696 T1707 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:18313/idf collection:collection1 shard:shard2
[junit4:junit4]   2> 11697 T1707 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4:junit4]   2> 11715 T1653 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6cf1e73b0008 type:delete cxid:0x47 zxid:0x7f txntype:-1 reqpath:n/a Error Path:/solr/collections/collection1/leaders/shard2 Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders/shard2
[junit4:junit4]   2> 11716 T1707 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 11717 T1653 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6cf1e73b0008 type:create cxid:0x48 zxid:0x80 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 11728 T1707 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 11728 T1707 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 11728 T1707 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:18313/idf/collection1/
[junit4:junit4]   2> 11728 T1707 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 11728 T1707 oasc.SyncStrategy.syncToMe http://127.0.0.1:18313/idf/collection1/ has no replicas
[junit4:junit4]   2> 11729 T1707 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:18313/idf/collection1/
[junit4:junit4]   2> 11729 T1707 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4]   2> 11732 T1653 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6cf1e73b0008 type:create cxid:0x51 zxid:0x83 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 12955 T1671 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 13059 T1706 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 13059 T1677 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 13059 T1690 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 13059 T1670 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 13069 T1707 oasc.ZkController.register We are http://127.0.0.1:18313/idf/collection1/ and leader is http://127.0.0.1:18313/idf/collection1/
[junit4:junit4]   2> 13069 T1707 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:18313/idf
[junit4:junit4]   2> 13069 T1707 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 13069 T1707 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 13069 T1707 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 13092 T1707 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 13094 T1648 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 13094 T1648 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 13095 T1648 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 13432 T1648 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 13436 T1648 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:43388
[junit4:junit4]   2> 13437 T1648 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 13437 T1648 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 13438 T1648 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1363332438640
[junit4:junit4]   2> 13439 T1648 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1363332438640/solr.xml
[junit4:junit4]   2> 13439 T1648 oasc.CoreContainer.<init> New CoreContainer 28134358
[junit4:junit4]   2> 13440 T1648 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1363332438640/'
[junit4:junit4]   2> 13441 T1648 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1363332438640/'
[junit4:junit4]   2> 13518 T1648 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 13519 T1648 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 13520 T1648 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 13520 T1648 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 13521 T1648 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 13522 T1648 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 13522 T1648 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 13523 T1648 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 13524 T1648 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 13524 T1648 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 13544 T1648 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 13562 T1648 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:47880/solr
[junit4:junit4]   2> 13563 T1648 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 13563 T1648 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:47880 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@1c6f5871
[junit4:junit4]   2> 13564 T1648 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 13565 T1719 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:47880. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 13566 T1719 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:47880, initiating session
[junit4:junit4]   2> 13566 T1650 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:16438
[junit4:junit4]   2> 13566 T1650 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:16438
[junit4:junit4]   2> 13597 T1652 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6cf1e73b0009 with negotiated timeout 20000 for client /140.211.11.196:16438
[junit4:junit4]   2> 13597 T1719 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:47880, sessionid = 0x13d6cf1e73b0009, negotiated timeout = 20000
[junit4:junit4]   2> 13597 T1720 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c6f5871 name:ZooKeeperConnection Watcher:127.0.0.1:47880 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 13597 T1648 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 13599 T1653 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d6cf1e73b0009
[junit4:junit4]   2> 13611 T1720 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 13611 T1650 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:16438 which had sessionid 0x13d6cf1e73b0009
[junit4:junit4]   2> 13611 T1648 oaz.ZooKeeper.close Session: 0x13d6cf1e73b0009 closed
[junit4:junit4]   2> 13612 T1648 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 13623 T1648 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:47880/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@6f9c6850
[junit4:junit4]   2> 13625 T1648 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 13625 T1721 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:47880. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 13626 T1721 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:47880, initiating session
[junit4:junit4]   2> 13626 T1650 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:23201
[junit4:junit4]   2> 13626 T1650 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:23201
[junit4:junit4]   2> 13639 T1652 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6cf1e73b000a with negotiated timeout 20000 for client /140.211.11.196:23201
[junit4:junit4]   2> 13639 T1721 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:47880, sessionid = 0x13d6cf1e73b000a, negotiated timeout = 20000
[junit4:junit4]   2> 13639 T1722 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6f9c6850 name:ZooKeeperConnection Watcher:127.0.0.1:47880/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 13640 T1648 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 13641 T1653 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6cf1e73b000a type:create cxid:0x1 zxid:0x90 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 13642 T1653 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6cf1e73b000a type:create cxid:0x2 zxid:0x91 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 13645 T1648 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 14567 T1671 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 14567 T1671 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"3",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":"shard2",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:18313_idf",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:18313/idf"}
[junit4:junit4]   2> 14575 T1722 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 14575 T1706 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 14575 T1670 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 14575 T1677 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 14575 T1690 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 14648 T1648 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43388_idf
[junit4:junit4]   2> 14649 T1653 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6cf1e73b000a type:delete cxid:0xd zxid:0x96 txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:43388_idf Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:43388_idf
[junit4:junit4]   2> 14651 T1648 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:43388_idf
[junit4:junit4]   2> 14653 T1722 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 14653 T1706 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 14654 T1677 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 14654 T1670 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 14654 T1677 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 14654 T1690 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 14654 T1670 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 14654 T1690 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 14655 T1706 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 14655 T1722 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 14662 T1723 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1363332438640/collection1
[junit4:junit4]   2> 14662 T1723 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 14663 T1723 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 14663 T1723 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 14664 T1723 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1363332438640/collection1/'
[junit4:junit4]   2> 14666 T1723 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1363332438640/collection1/lib/README' to classloader
[junit4:junit4]   2> 14666 T1723 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1363332438640/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 14719 T1723 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 14781 T1723 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 14882 T1723 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 14888 T1723 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 15478 T1723 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 15485 T1723 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 15488 T1723 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 15499 T1723 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 15503 T1723 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 15506 T1723 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 15507 T1723 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 15508 T1723 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 15508 T1723 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 15509 T1723 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 15509 T1723 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 15509 T1723 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 15509 T1723 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1363332438640/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363332425525/jetty3/
[junit4:junit4]   2> 15510 T1723 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@962522b
[junit4:junit4]   2> 15510 T1723 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 15511 T1723 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363332425525/jetty3 forceNew: false
[junit4:junit4]   2> 15511 T1723 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1363332425525/jetty3/index/
[junit4:junit4]   2> 15511 T1723 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1363332425525/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 15511 T1723 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363332425525/jetty3/index forceNew: false
[junit4:junit4]   2> 15513 T1723 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5f9b82a7 lockFactory=org.apache.lucene.store.NativeFSLockFactory@712d4990),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 15513 T1723 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 15515 T1723 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 15515 T1723 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 15515 T1723 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 15516 T1723 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 15516 T1723 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 15516 T1723 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 15517 T1723 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 15517 T1723 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 15517 T1723 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 15522 T1723 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 15527 T1723 oass.SolrIndexSearcher.<init> Opening Searcher@5192a545 main
[junit4:junit4]   2> 15527 T1723 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363332425525/jetty3/tlog
[junit4:junit4]   2> 15528 T1723 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 15528 T1723 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 15532 T1724 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5192a545 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 15533 T1723 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 15533 T1723 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 16080 T1671 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 16081 T1671 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"4",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:43388_idf",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:43388/idf"}
[junit4:junit4]   2> 16081 T1671 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4]   2> 16082 T1671 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 16095 T1677 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 16095 T1722 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 16095 T1690 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 16095 T1670 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 16095 T1706 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 16536 T1723 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 16536 T1723 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:43388/idf collection:collection1 shard:shard1
[junit4:junit4]   2> 16540 T1723 oasc.ZkController.register We are http://127.0.0.1:43388/idf/collection1/ and leader is http://127.0.0.1:57179/idf/collection1/
[junit4:junit4]   2> 16540 T1723 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:43388/idf
[junit4:junit4]   2> 16540 T1723 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 16541 T1723 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C238 name=collection1 org.apache.solr.core.SolrCore@3c938f1c url=http://127.0.0.1:43388/idf/collection1 node=127.0.0.1:43388_idf C238_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:43388_idf, base_url=http://127.0.0.1:43388/idf}
[junit4:junit4]   2> 16541 T1725 C238 P43388 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 16542 T1725 C238 P43388 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 16542 T1723 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 16542 T1725 C238 P43388 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4]   2> 16542 T1725 C238 P43388 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 16543 T1648 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 16544 T1648 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 16544 T1648 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 16550 T1725 C238 P43388 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 16758 T1648 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 16761 T1648 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:57030
[junit4:junit4]   2> 16762 T1648 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 16762 T1648 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 16763 T1648 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1363332442084
[junit4:junit4]   2> 16763 T1648 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1363332442084/solr.xml
[junit4:junit4]   2> 16764 T1648 oasc.CoreContainer.<init> New CoreContainer 396692081
[junit4:junit4]   2> 16764 T1648 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1363332442084/'
[junit4:junit4]   2> 16765 T1648 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1363332442084/'
[junit4:junit4]   2> 16815 T1648 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 16816 T1648 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 16816 T1648 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 16817 T1648 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 16817 T1648 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 16817 T1648 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 16818 T1648 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 16818 T1648 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 16819 T1648 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 16819 T1648 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 16841 T1648 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 16864 T1648 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:47880/solr
[junit4:junit4]   2> 16865 T1648 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 16866 T1648 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:47880 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@385bbcb1
[junit4:junit4]   2> 16867 T1648 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 16867 T1736 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:47880. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 16868 T1736 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:47880, initiating session
[junit4:junit4]   2> 16868 T1650 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:39290
[junit4:junit4]   2> 16868 T1650 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:39290
[junit4:junit4]   2> 16870 T1652 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6cf1e73b000b with negotiated timeout 20000 for client /140.211.11.196:39290
[junit4:junit4]   2> 16870 T1736 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:47880, sessionid = 0x13d6cf1e73b000b, negotiated timeout = 20000
[junit4:junit4]   2> 16871 T1737 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@385bbcb1 name:ZooKeeperConnection Watcher:127.0.0.1:47880 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 16871 T1648 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 16872 T1653 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d6cf1e73b000b
[junit4:junit4]   2> 16874 T1737 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 16874 T1650 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:39290 which had sessionid 0x13d6cf1e73b000b
[junit4:junit4]   2> 16874 T1648 oaz.ZooKeeper.close Session: 0x13d6cf1e73b000b closed
[junit4:junit4]   2> 16874 T1648 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 16886 T1648 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:47880/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@4a9195e0
[junit4:junit4]   2> 16887 T1648 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 16888 T1738 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:47880. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 16888 T1738 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:47880, initiating session
[junit4:junit4]   2> 16888 T1650 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:32109
[junit4:junit4]   2> 16889 T1650 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:32109
[junit4:junit4]   2> 16890 T1652 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6cf1e73b000c with negotiated timeout 20000 for client /140.211.11.196:32109
[junit4:junit4]   2> 16890 T1738 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:47880, sessionid = 0x13d6cf1e73b000c, negotiated timeout = 20000
[junit4:junit4]   2> 16890 T1739 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4a9195e0 name:ZooKeeperConnection Watcher:127.0.0.1:47880/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 16891 T1648 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 16892 T1653 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6cf1e73b000c type:create cxid:0x1 zxid:0xa3 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 16893 T1653 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6cf1e73b000c type:create cxid:0x2 zxid:0xa4 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 16895 T1648 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 17599 T1671 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 17600 T1671 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"4",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:43388_idf",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:43388/idf"}
[junit4:junit4]   2> 17614 T1722 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 17614 T1690 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 17614 T1739 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 17614 T1706 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 17614 T1670 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 17614 T1677 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 17898 T1648 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57030_idf
[junit4:junit4]   2> 17899 T1653 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6cf1e73b000c type:delete cxid:0xd zxid:0xa9 txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:57030_idf Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:57030_idf
[junit4:junit4]   2> 17904 T1648 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:57030_idf
[junit4:junit4]   2> 17933 T1706 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 17933 T1722 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 17933 T1677 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 17934 T1690 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 17934 T1677 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 17934 T1690 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 17934 T1739 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 17934 T1670 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 17934 T1670 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 17934 T1739 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 17935 T1722 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 17935 T1706 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 17969 T1740 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1363332442084/collection1
[junit4:junit4]   2> 17969 T1740 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 17970 T1740 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 17970 T1740 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 17971 T1740 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1363332442084/collection1/'
[junit4:junit4]   2> 17972 T1740 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1363332442084/collection1/lib/README' to classloader
[junit4:junit4]   2> 17973 T1740 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1363332442084/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 18026 T1740 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 18089 T1740 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 18190 T1740 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 18196 T1740 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 18567 T1680 oass.SolrDispatchFilter.handleAdminRequest /admin/cores {coreNodeName=4&state=recovering&nodeName=127.0.0.1:43388_idf&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=2002 
[junit4:junit4]   2> 19476 T1740 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 19487 T1740 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 19492 T1740 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 19506 T1740 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 19511 T1740 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 19515 T1740 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 19516 T1740 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 19517 T1740 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 19517 T1740 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 19518 T1740 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 19518 T1740 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 19518 T1740 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 19518 T1740 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1363332442084/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363332425525/jetty4/
[junit4:junit4]   2> 19519 T1740 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@962522b
[junit4:junit4]   2> 19519 T1740 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 19520 T1740 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363332425525/jetty4 forceNew: false
[junit4:junit4]   2> 19520 T1740 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1363332425525/jetty4/index/
[junit4:junit4]   2> 19520 T1740 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1363332425525/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 19521 T1740 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363332425525/jetty4/index forceNew: false
[junit4:junit4]   2> 19524 T1740 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@78ffcbf7 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5d697666),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 19524 T1740 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 19526 T1740 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 19527 T1740 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 19527 T1740 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 19528 T1740 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 19528 T1740 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 19529 T1740 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 19529 T1740 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 19530 T1740 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 19530 T1740 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 19537 T1740 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 19543 T1740 oass.SolrIndexSearcher.<init> Opening Searcher@7f72b53c main
[junit4:junit4]   2> 19543 T1740 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363332425525/jetty4/tlog
[junit4:junit4]   2> 19544 T1740 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 19544 T1740 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 19548 T1741 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7f72b53c main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 19550 T1740 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 19550 T1740 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2>  C238_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:43388_idf, base_url=http://127.0.0.1:43388/idf}
[junit4:junit4]   2> 20570 T1725 C238 P43388 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:57179/idf/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4]   2> 20570 T1725 C238 P43388 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:43388/idf START replicas=[http://127.0.0.1:57179/idf/collection1/] nUpdates=100
[junit4:junit4]   2> 20570 T1725 C238 P43388 oasu.PeerSync.sync WARNING no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 20571 T1725 C238 P43388 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4]   2> 20571 T1725 C238 P43388 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4]   2> 20572 T1725 C238 P43388 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4]   2> 20572 T1725 C238 P43388 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4]   2> 20572 T1725 C238 P43388 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:57179/idf/collection1/. core=collection1
[junit4:junit4]   2> 20572 T1725 C238 P43388 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C239 name=collection1 org.apache.solr.core.SolrCore@68f3c7ae url=http://127.0.0.1:57179/idf/collection1 node=127.0.0.1:57179_idf C239_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:57179_idf, base_url=http://127.0.0.1:57179/idf, leader=true}
[junit4:junit4]   2> 20579 T1681 C239 P57179 REQ /get {getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0 
[junit4:junit4]   2> 20591 T1682 C239 P57179 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 20593 T1682 C239 P57179 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@45c8fad1 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7eeb6440),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 20593 T1682 C239 P57179 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 20594 T1682 C239 P57179 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@45c8fad1 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7eeb6440),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@45c8fad1 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7eeb6440),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 20594 T1682 C239 P57179 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[segments_2]
[junit4:junit4]   2> 20595 T1682 C239 P57179 oass.SolrIndexSearcher.<init> Opening Searcher@41f7dc64 realtime
[junit4:junit4]   2> 20595 T1682 C239 P57179 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 20596 T1682 C239 P57179 /update {waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 5
[junit4:junit4]   2> 20596 T1725 C238 P43388 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 20597 T1725 C238 P43388 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4]   2> 20598 T1683 C239 P57179 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 20599 T1683 C239 P57179 REQ /replication {command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 20599 T1725 C238 P43388 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4]   2> 20599 T1725 C238 P43388 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4]   2> 20599 T1725 C238 P43388 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4]   2> 20602 T1683 C239 P57179 REQ /replication {command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0 
[junit4:junit4]   2> 20602 T1725 C238 P43388 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4]   2> 20603 T1725 C238 P43388 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363332425525/jetty3/index.20130315122726124 forceNew: false
[junit4:junit4]   2> 20604 T1725 C238 P43388 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@4e6fe801 lockFactory=org.apache.lucene.store.NativeFSLockFactory@3eac2fd; maxCacheMB=0.826171875 maxMergeSizeMB=0.1025390625)) fullCopy=false
[junit4:junit4]   2> 20606 T1683 C239 P57179 REQ /replication {file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> 20607 T1725 C238 P43388 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4]   2> 20608 T1725 C238 P43388 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4]   2> 20608 T1725 C238 P43388 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4]   2> 20661 T1725 C238 P43388 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5f9b82a7 lockFactory=org.apache.lucene.store.NativeFSLockFactory@712d4990),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5f9b82a7 lockFactory=org.apache.lucene.store.NativeFSLockFactory@712d4990),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 20661 T1725 C238 P43388 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[segments_2]
[junit4:junit4]   2> 20662 T1725 C238 P43388 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4]   2> 20662 T1725 C238 P43388 oass.SolrIndexSearcher.<init> Opening Searcher@4142aef8 main
[junit4:junit4]   2> 20663 T1724 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4142aef8 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 20664 T1725 C238 P43388 oash.SnapPuller.fetchLatestIndex removing temporary index download directory files MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@4e6fe801 lockFactory=org.apache.lucene.store.NativeFSLockFactory@3eac2fd; maxCacheMB=0.826171875 maxMergeSizeMB=0.1025390625))
[junit4:junit4]   2> 20664 T1725 C238 P43388 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4]   2> 20664 T1725 C238 P43388 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4]   2> 20664 T1725 C238 P43388 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 20664 T1725 C238 P43388 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 20671 T1725 C238 P43388 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4]   2> 20759 T1671 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 20760 T1671 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"5",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:57030_idf",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:57030/idf"}
[junit4:junit4]   2> 20760 T1671 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4]   2> 20761 T1671 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4:junit4]   2> 20765 T1671 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"4",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:43388_idf",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:43388/idf"}
[junit4:junit4]   2> 20777 T1690 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 20777 T1670 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 20777 T1739 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 20777 T1722 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 20777 T1677 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 20777 T1706 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 21554 T1740 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 21554 T1740 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:57030/idf collection:collection1 shard:shard2
[junit4:junit4]   2> 21557 T1740 oasc.ZkController.register We are http://127.0.0.1:57030/idf/collection1/ and leader is http://127.0.0.1:18313/idf/collection1/
[junit4:junit4]   2> 21557 T1740 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:57030/idf
[junit4:junit4]   2> 21557 T1740 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 21558 T1740 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C240 name=collection1 org.apache.solr.core.SolrCore@1617f808 url=http://127.0.0.1:57030/idf/collection1 node=127.0.0.1:57030_idf C240_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:57030_idf, base_url=http://127.0.0.1:57030/idf}
[junit4:junit4]   2> 21558 T1744 C240 P57030 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 21559 T1744 C240 P57030 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 21559 T1740 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 21559 T1744 C240 P57030 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4]   2> 21560 T1744 C240 P57030 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 21560 T1648 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 21561 T1744 C240 P57030 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 21561 T1648 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 21562 T1648 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 21573 T1648 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 21575 T1648 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 21586 T1684 oasc.CoreContainer.create Creating SolrCore 'onenodecollectioncore' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363332429392/onenodecollectioncore
[junit4:junit4]   2> 21586 T1684 oasc.ZkController.createCollectionZkNode Check for collection zkNode:onenodecollection
[junit4:junit4]   2> 21587 T1684 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:onenodecollection
[junit4:junit4]   2> 21587 T1684 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4]   2> 21588 T1684 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
[junit4:junit4]   2> 21589 T1684 oascc.SolrZkClient.makePath makePath: /collections/onenodecollection
[junit4:junit4]   2> 21591 T1684 oasc.ZkController.readConfigName Load collection config from:/collections/onenodecollection
[junit4:junit4]   2> 21593 T1684 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363332429392/onenodecollectioncore/'
[junit4:junit4]   2> 21656 T1684 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 21716 T1684 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 21817 T1684 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 21823 T1684 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 22292 T1671 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 22292 T1671 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"5",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":"shard2",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:57030_idf",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:57030/idf"}
[junit4:junit4]   2> 22309 T1706 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 22309 T1722 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 22309 T1677 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 22309 T1739 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 22309 T1670 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 22309 T1690 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 22432 T1684 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 22439 T1684 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 22442 T1684 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 22454 T1684 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 22458 T1684 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 22461 T1684 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 22462 T1684 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 22462 T1684 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 22462 T1684 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 22463 T1684 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 22464 T1684 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 22464 T1684 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 22464 T1684 oasc.SolrCore.<init> [onenodecollectioncore] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363332429392/onenodecollectioncore/, dataDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1363332425525/onenodecollection/
[junit4:junit4]   2> 22464 T1684 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@962522b
[junit4:junit4]   2> 22465 T1684 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 22465 T1684 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1363332425525/onenodecollection forceNew: false
[junit4:junit4]   2> 22466 T1684 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1363332425525/onenodecollection/index/
[junit4:junit4]   2> 22466 T1684 oasc.SolrCore.initIndex WARNING [onenodecollectioncore] Solr index directory '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1363332425525/onenodecollection/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 22466 T1684 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1363332425525/onenodecollection/index forceNew: false
[junit4:junit4]   2> 22468 T1684 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@40fd1595 lockFactory=org.apache.lucene.store.NativeFSLockFactory@683f7b04),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 22469 T1684 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 22470 T1684 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 22471 T1684 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 22471 T1684 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 22471 T1684 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 22472 T1684 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 22472 T1684 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 22472 T1684 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 22473 T1684 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 22473 T1684 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 22478 T1684 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 22482 T1684 oass.SolrIndexSearcher.<init> Opening Searcher@6431156b main
[junit4:junit4]   2> 22482 T1684 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1363332425525/onenodecollection/tlog
[junit4:junit4]   2> 22483 T1684 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 22483 T1684 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 22485 T1746 oasc.SolrCore.registerSearcher [onenodecollectioncore] Registered new searcher Searcher@6431156b main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 22487 T1684 oasc.ZkController.publish publishing core=onenodecollectioncore state=down
[junit4:junit4]   2> 22574 T1696 oass.SolrDispatchFilter.handleAdminRequest /admin/cores {coreNodeName=5&state=recovering&nodeName=127.0.0.1:57030_idf&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001 
[junit4:junit4]   2> 23815 T1671 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 23815 T1671 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":"none",
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"onenodecollectioncore",
[junit4:junit4]   2> 	  "collection":"onenodecollection",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:57179_idf",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:57179/idf"}
[junit4:junit4]   2> 23816 T1671 oasc.Overseer$ClusterStateUpdater.createCollection Create collection onenodecollection with numShards 1
[junit4:junit4]   2> 23816 T1671 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 23857 T1706 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 23858 T1739 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 23857 T1690 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 23857 T1722 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 23857 T1670 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 23857 T1677 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 24490 T1684 oasc.CoreContainer.registerCore registering core: onenodecollectioncore
[junit4:junit4]   2> 24490 T1684 oasc.ZkController.register Register replica - core:onenodecollectioncore address:http://127.0.0.1:57179/idf collection:onenodecollection shard:shard1
[junit4:junit4]   2> 24491 T1684 oascc.SolrZkClient.makePath makePath: /collections/onenodecollection/leader_elect/shard1/election
[junit4:junit4]   2> 24507 T1653 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6cf1e73b0006 type:delete cxid:0xaa zxid:0xc5 txntype:-1 reqpath:n/a Error Path:/solr/collections/onenodecollection/leaders Error:KeeperErrorCode = NoNode for /solr/collections/onenodecollection/leaders
[junit4:junit4]   2> 24507 T1684 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 24508 T1653 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6cf1e73b0006 type:create cxid:0xab zxid:0xc6 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 24518 T1684 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 24518 T1684 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 24518 T1684 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:57179/idf/onenodecollectioncore/
[junit4:junit4]   2> 24519 T1684 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 24519 T1684 oasc.SyncStrategy.syncToMe http://127.0.0.1:57179/idf/onenodecollectioncore/ has no replicas
[junit4:junit4]   2> 24519 T1684 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:57179/idf/onenodecollectioncore/
[junit4:junit4]   2> 24519 T1684 oascc.SolrZkClient.makePath makePath: /collections/onenodecollection/leaders/shard1
[junit4:junit4]   2> 24523 T1653 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6cf1e73b0006 type:create cxid:0xb5 zxid:0xca txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2>  C240_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:57030_idf, base_url=http://127.0.0.1:57030/idf}
[junit4:junit4]   2> 24575 T1744 C240 P57030 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:18313/idf/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4]   2> 24575 T1744 C240 P57030 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:57030/idf START replicas=[http://127.0.0.1:18313/idf/collection1/] nUpdates=100
[junit4:junit4]   2> 24575 T1744 C240 P57030 oasu.PeerSync.sync WARNING no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 24575 T1744 C240 P57030 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4]   2> 24576 T1744 C240 P57030 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4]   2> 24576 T1744 C240 P57030 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4]   2> 24576 T1744 C240 P57030 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4]   2> 24577 T1744 C240 P57030 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:18313/idf/collection1/. core=collection1
[junit4:junit4]   2> 24577 T1744 C240 P57030 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C241 name=collection1 org.apache.solr.core.SolrCore@3b8569bf url=http://127.0.0.1:18313/idf/collection1 node=127.0.0.1:18313_idf C241_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:18313_idf, base_url=http://127.0.0.1:18313/idf, leader=true}
[junit4:junit4]   2> 24582 T1697 C241 P18313 REQ /get {getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0 
[junit4:junit4]   2> 24594 T1698 C241 P18313 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 24596 T1698 C241 P18313 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@46c101bf lockFactory=org.apache.lucene.store.NativeFSLockFactory@4510764a),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 24596 T1698 C241 P18313 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 24597 T1698 C241 P18313 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@46c101bf lockFactory=org.apache.lucene.store.NativeFSLockFactory@4510764a),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@46c101bf lockFactory=org.apache.lucene.store.NativeFSLockFactory@4510764a),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 24597 T1698 C241 P18313 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[segments_2]
[junit4:junit4]   2> 24598 T1698 C241 P18313 oass.SolrIndexSearcher.<init> Opening Searcher@39dcaf68 realtime
[junit4:junit4]   2> 24598 T1698 C241 P18313 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 24599 T1698 C241 P18313 /update {waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 5
[junit4:junit4]   2> 24599 T1744 C240 P57030 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 24599 T1744 C240 P57030 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4]   2> 24601 T1699 C241 P18313 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 24601 T1699 C241 P18313 REQ /replication {command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 24601 T1744 C240 P57030 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4]   2> 24602 T1744 C240 P57030 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4]   2> 24602 T1744 C240 P57030 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4]   2> 24604 T1699 C241 P18313 REQ /replication {command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0 
[junit4:junit4]   2> 24604 T1744 C240 P57030 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4]   2> 24605 T1744 C240 P57030 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363332425525/jetty4/index.20130315122730126 forceNew: false
[junit4:junit4]   2> 24605 T1744 C240 P57030 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@2ae2f8dd lockFactory=org.apache.lucene.store.NativeFSLockFactory@46d3b20d; maxCacheMB=0.826171875 maxMergeSizeMB=0.1025390625)) fullCopy=false
[junit4:junit4]   2> 24608 T1699 C241 P18313 REQ /replication {file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> 24609 T1744 C240 P57030 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4]   2> 24609 T1744 C240 P57030 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4]   2> 24610 T1744 C240 P57030 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4]   2> 24662 T1744 C240 P57030 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@78ffcbf7 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5d697666),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@78ffcbf7 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5d697666),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 24662 T1744 C240 P57030 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[segments_2]
[junit4:junit4]   2> 24662 T1744 C240 P57030 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4]   2> 24662 T1744 C240 P57030 oass.SolrIndexSearcher.<init> Opening Searcher@69de18a1 main
[junit4:junit4]   2> 24663 T1741 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@69de18a1 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 24664 T1744 C240 P57030 oash.SnapPuller.fetchLatestIndex removing temporary index download directory files MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@2ae2f8dd lockFactory=org.apache.lucene.store.NativeFSLockFactory@46d3b20d; maxCacheMB=0.826171875 maxMergeSizeMB=0.1025390625))
[junit4:junit4]   2> 24664 T1744 C240 P57030 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4]   2> 24664 T1744 C240 P57030 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4]   2> 24664 T1744 C240 P57030 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 24664 T1744 C240 P57030 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 24666 T1744 C240 P57030 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4]   2> 25362 T1671 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 25381 T1671 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"5",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":"shard2",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:57030_idf",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:57030/idf"}
[junit4:junit4]   2> 25393 T1706 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 25393 T1677 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 25393 T1739 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 25393 T1722 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 25393 T1670 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 25393 T1690 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 25442 T1684 oasc.ZkController.register We are http://127.0.0.1:57179/idf/onenodecollectioncore/ and leader is http://127.0.0.1:57179/idf/onenodecollectioncore/
[junit4:junit4]   2> 25442 T1684 oasc.ZkController.register No LogReplay needed for core=onenodecollectioncore baseURL=http://127.0.0.1:57179/idf
[junit4:junit4]   2> 25442 T1684 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 25443 T1684 oasc.ZkController.publish publishing core=onenodecollectioncore state=active
[junit4:junit4]   2> 25444 T1684 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 25445 T1684 oass.SolrDispatchFilter.handleAdminRequest /admin/cores {numShards=1&roles=none&name=onenodecollectioncore&action=CREATE&collection=onenodecollection&dataDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1363332425525/onenodecollection&wt=javabin&version=2} status=0 QTime=3859 
[junit4:junit4]   2> 25446 T1648 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: onenodecollection failOnTimeout:true timeout (sec):230
[junit4:junit4]   2> 25447 T1648 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 26448 T1648 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 26900 T1671 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 26901 T1671 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"127.0.0.1:57179_idf_onenodecollectioncore",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":"none",
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"onenodecollectioncore",
[junit4:junit4]   2> 	  "collection":"onenodecollection",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:57179_idf",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:57179/idf"}
[junit4:junit4]   2> 26921 T1690 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 26921 T1739 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 26921 T1722 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 26921 T1706 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 26921 T1670 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 26921 T1677 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 27450 T1648 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 27456 T1648 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: onenodecollection
[junit4:junit4]   2> 27457 T1648 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C242 name=onenodecollectioncore org.apache.solr.core.SolrCore@45b7003c url=http://127.0.0.1:57179/idf/onenodecollectioncore node=127.0.0.1:57179_idf C242_STATE=coll:onenodecollection core:onenodecollectioncore props:{shard=shard1, roles=none, state=active, core=onenodecollectioncore, collection=onenodecollection, node_name=127.0.0.1:57179_idf, base_url=http://127.0.0.1:57179/idf, leader=true}
[junit4:junit4]   2> 27487 T1685 C242 P57179 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@40fd1595 lockFactory=org.apache.lucene.store.NativeFSLockFactory@683f7b04),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 27488 T1685 C242 P57179 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 27492 T1685 C242 P57179 /update {wt=javabin&version=2} {add=[1 (1429557690243219456)]} 0 7
[junit4:junit4]   2> 27499 T1685 C242 P57179 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 27504 T1685 C242 P57179 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@40fd1595 lockFactory=org.apache.lucene.store.NativeFSLockFactory@683f7b04),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@40fd1595 lockFactory=org.apache.lucene.store.NativeFSLockFactory@683f7b04),segFN=segments_2,generation=2,filenames=[_0_TestBloomFilteredLucene41Postings_0.tim, _0_MockVariableIntBlock_0.doc, _0_MockVariableIntBlock_0.skp, _0_TestBloomFilteredLucene41Postings_0.tip, _0_TestBloomFilteredLucene41Postings_0.pos, _0_MockVariableIntBlock_0.pyl, _0_MockVariableIntBlock_0.tib, _0_Pulsing41_0.tim, _0_Pulsing41_0.tip, _0.nvd, _0_MockVariableIntBlock_0.pos, _0.si, _0_MockVariableIntBlock_0.tii, _0.fnm, _0_Disk_0.dvdd, _0_TestBloomFilteredLucene41Postings_0.blm, _0.nvm, _0_Disk_0.dvdm, _0_Direct_0.doc, _0_Pulsing41_0.pos, _0_Pulsing41_0.doc, _0_MockVariableIntBlock_0.frq, segments_2, _0_TestBloomFilteredLucene41Postings_0.doc, _0_Direct_0.pos, _0_Direct_0.tip, _0.fdx, _0_Direct_0.tim, _0.fdt]
[junit4:junit4]   2> 27504 T1685 C242 P57179 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[_0_TestBloomFilteredLucene41Postings_0.tim, _0_MockVariableIntBlock_0.doc, _0_MockVariableIntBlock_0.skp, _0_TestBloomFilteredLucene41Postings_0.tip, _0_TestBloomFilteredLucene41Postings_0.pos, _0_MockVariableIntBlock_0.pyl, _0_MockVariableIntBlock_0.tib, _0_Pulsing41_0.tim, _0_Pulsing41_0.tip, _0.nvd, _0_MockVariableIntBlock_0.pos, _0.si, _0_MockVariableIntBlock_0.tii, _0.fnm, _0_Disk_0.dvdd, _0_TestBloomFilteredLucene41Postings_0.blm, _0.nvm, _0_Disk_0.dvdm, _0_Direct_0.doc, _0_Pulsing41_0.pos, _0_Pulsing41_0.doc, _0_MockVariableIntBlock_0.frq, segments_2, _0_TestBloomFilteredLucene41Postings_0.doc, _0_Direct_0.pos, _0_Direct_0.tip, _0.fdx, _0_Direct_0.tim, _0.fdt]
[junit4:junit4]   2> 27554 T1685 C242 P57179 oass.SolrIndexSearcher.<init> Opening Searcher@5709cdc8 main
[junit4:junit4]   2> 27554 T1685 C242 P57179 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 27555 T1746 oasc.SolrCore.registerSearcher [onenodecollectioncore] Registered new searcher Searcher@5709cdc8 main{StandardDirectoryReader(segments_2:3 _0(5.0):C1)}
[junit4:junit4]   2> 27556 T1685 C242 P57179 /update {waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 57
[junit4:junit4]   2> 27560 T1685 C242 P57179 REQ /select {q=*:*&wt=javabin&version=2} hits=1 status=0 QTime=1 
[junit4:junit4]   2> 27561 T1648 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 27574 T1685 C242 P57179 REQ /select {q=*:*&wt=javabin&version=2} hits=1 status=0 QTime=0 
[junit4:junit4]   2> ASYNC  NEW_CORE C243 name=collection1 org.apache.solr.core.SolrCore@13eef332 url=http://127.0.0.1:65311/idf/collection1 node=127.0.0.1:65311_idf C243_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:65311_idf, base_url=http://127.0.0.1:65311/idf, leader=true}
[junit4:junit4]   2> 27584 T1660 C243 P65311 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@496cff2a lockFactory=org.apache.lucene.store.NativeFSLockFactory@48303e79),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 27585 T1660 C243 P65311 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 27588 T1660 C243 P65311 /update {wt=javabin&version=2} {add=[1 (1429557690344931328)]} 0 6
[junit4:junit4]   2>  C238_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:43388_idf, base_url=http://127.0.0.1:43388/idf}
[junit4:junit4]   2> 27611 T1712 C238 P43388 /update {distrib.from=http://127.0.0.1:57179/idf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1429557690362757120)]} 0 3
[junit4:junit4]   2> 27612 T1686 C239 P57179 /update {wt=javabin&version=2} {add=[1 (1429557690362757120)]} 0 13
[junit4:junit4]   2> 27614 T1660 C243 P65311 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 27619 T1660 C243 P65311 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@496cff2a lockFactory=org.apache.lucene.store.NativeFSLockFactory@48303e79),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@496cff2a lockFactory=org.apache.lucene.store.NativeFSLockFactory@48303e79),segFN=segments_2,generation=2,filenames=[_0_TestBloomFilteredLucene41Postings_0.tim, _0_MockVariableIntBlock_0.doc, _0_TestBloomFilteredLucene41Postings_0.tip, _0_MockVariableIntBlock_0.skp, _0_TestBloomFilteredLucene41Postings_0.pos, _0_MockVariableIntBlock_0.pyl, _0_Pulsing41_0.tim, _0_MockVariableIntBlock_0.tib, _0_Pulsing41_0.tip, _0.nvd, _0_MockVariableIntBlock_0.pos, _0.si, _0_MockVariableIntBlock_0.tii, _0.fnm, _0_Disk_0.dvdd, _0_TestBloomFilteredLucene41Postings_0.blm, _0.nvm, _0_Disk_0.dvdm, _0_Direct_0.doc, _0_Pulsing41_0.doc, _0_Pulsing41_0.pos, _0_MockVariableIntBlock_0.frq, segments_2, _0_TestBloomFilteredLucene41Postings_0.doc, _0_Direct_0.pos, _0_Direct_0.tip, _0.fdx, _0_Direct_0.tim, _0.fdt]
[junit4:junit4]   2> 27619 T1660 C243 P65311 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[_0_TestBloomFilteredLucene41Postings_0.tim, _0_MockVariableIntBlock_0.doc, _0_TestBloomFilteredLucene41Postings_0.tip, _0_MockVariableIntBlock_0.skp, _0_TestBloomFilteredLucene41Postings_0.pos, _0_MockVariableIntBlock_0.pyl, _0_Pulsing41_0.tim, _0_MockVariableIntBlock_0.tib, _0_Pulsing41_0.tip, _0.nvd, _0_MockVariableIntBlock_0.pos, _0.si, _0_MockVariableIntBlock_0.tii, _0.fnm, _0_Disk_0.dvdd, _0_TestBloomFilteredLucene41Postings_0.blm, _0.nvm, _0_Disk_0.dvdm, _0_Direct_0.doc, _0_Pulsing41_0.doc, _0_Pulsing41_0.pos, _0_MockVariableIntBlock_0.frq, segments_2, _0_TestBloomFilteredLucene41Postings_0.doc, _0_Direct_0.pos, _0_Direct_0.tip, _0.fdx, _0_Direct_0.tim, _0.fdt]
[junit4:junit4]   2> 27621 T1660 C243 P65311 oass.SolrIndexSearcher.<init> Opening Searcher@47906e1d main
[junit4:junit4]   2> 27622 T1660 C243 P65311 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 27623 T1674 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@47906e1d main{StandardDirectoryReader(segments_2:3 _0(5.0):C1)}
[junit4:junit4]   2> 27623 T1660 C243 P65311 /update {waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 9
[junit4:junit4]   2> 27626 T1686 C239 P57179 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 27630 T1686 C239 P57179 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@45c8fad1 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7eeb6440),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@45c8fad1 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7eeb6440),segFN=segments_3,generation=3,filenames=[_0_TestBloomFilteredLucene41Postings_0.tim, _0_MockVariableIntBlock_0.doc, _0_TestBloomFilteredLucene41Postings_0.tip, _0_MockVariableIntBlock_0.skp, _0_TestBloomFilteredLucene41Postings_0.pos, _0_MockVariableIntBlock_0.pyl, _0_Pulsing41_0.tim, _0_MockVariableIntBlock_0.tib, _0_Pulsing41_0.tip, _0.nvd, _0_MockVariableIntBlock_0.pos, _0.si, _0_MockVariableIntBlock_0.tii, _0.fnm, _0_Disk_0.dvdd, _0_TestBloomFilteredLucene41Postings_0.blm, _0.nvm, _0_Disk_0.dvdm, _0_Direct_0.doc, _0_Pulsing41_0.doc, _0_Pulsing41_0.pos, _0_MockVariableIntBlock_0.frq, _0_TestBloomFilteredLucene41Postings_0.doc, _0_Direct_0.pos, _0_Direct_0.tip, _0.fdx, _0_Direct_0.tim, _0.fdt, segments_3]
[junit4:junit4]   2> 27631 T1686 C239 P57179 oasc.SolrDeletionPolicy.updateCommits newest commit = 3[_0_TestBloomFilteredLucene41Postings_0.tim, _0_MockVariableIntBlock_0.doc, _0_TestBloomFilteredLucene41Postings_0.tip, _0_MockVariableIntBlock_0.skp, _0_TestBloomFilteredLucene41Postings_0.pos, _0_MockVariableIntBlock_0.pyl, _0_Pulsing41_0.tim, _0_MockVariableIntBlock_0.tib, _0_Pulsing41_0.tip, _0.nvd, _0_MockVariableIntBlock_0.pos, _0.si, _0_MockVariableIntBlock_0.tii, _0.fnm, _0_Disk_0.dvdd, _0_TestBloomFilteredLucene41Postings_0.blm, _0.nvm, _0_Disk_0.dvdm, _0_Direct_0.doc, _0_Pulsing41_0.doc, _0_Pulsing41_0.pos, _0_MockVariableIntBlock_0.frq, _0_TestBloomFilteredLucene41Postings_0.doc, _0_Direct_0.pos, _0_Direct_0.tip, _0.fdx, _0_Direct_0.tim, _0.fdt, segments_3]
[junit4:junit4]   2> 27633 T1686 C239 P57179 oass.SolrIndexSearcher.<init> Opening Searcher@3ec0e689 main
[junit4:junit4]   2> 27633 T1686 C239 P57179 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 27634 T1692 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3ec0e689 main{StandardDirectoryReader(segments_3:3:nrt _0(5.0):C1)}
[junit4:junit4]   2> 27634 T1686 C239 P57179 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:43388/idf/collection1/, StdNode: http://127.0.0.1:18313/idf/collection1/, StdNode: http://127.0.0.1:57030/idf/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4]   2> 27636 T1702 C241 P18313 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 27637 T1712 C238 P43388 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 27637 T1702 C241 P18313 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@46c101bf lockFactory=org.apache.lucene.store.NativeFSLockFactory@4510764a),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@46c101bf lockFactory=org.apache.lucene.store.NativeFSLockFactory@4510764a),segFN=segments_3,generation=3,filenames=[segments_3]
[junit4:junit4]   2> 27638 T1702 C241 P18313 oasc.SolrDeletionPolicy.updateCommits newest commit = 3[segments_3]
[junit4:junit4]   2> 27638 T1702 C241 P18313 oass.SolrIndexSearcher.<init> Opening Searcher@47d7353b main
[junit4:junit4]   2> 27638 T1702 C241 P18313 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 27639 T1708 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@47d7353b main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 27639 T1702 C241 P18313 /update {waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 3
[junit4:junit4]   2> 27642 T1712 C238 P43388 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5f9b82a7 lockFactory=org.apache.lucene.store.NativeFSLockFactory@712d4990),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5f9b82a7 lockFactory=org.apache.lucene.store.NativeFSLockFactory@712d4990),segFN=segments_3,generation=3,filenames=[_0_TestBloomFilteredLucene41Postings_0.tim, _0_MockVariableIntBlock_0.doc, _0_TestBloomFilteredLucene41Postings_0.tip, _0_MockVariableIntBlock_0.skp, _0_TestBloomFilteredLucene41Postings_0.pos, _0_MockVariableIntBlock_0.pyl, _0_Pulsing41_0.tim, _0_MockVariableIntBlock_0.tib, _0_Pulsing41_0.tip, _0.nvd, _0_MockVariableIntBlock_0.pos, _0.si, _0_MockVariableIntBlock_0.tii, _0.fnm, _0_Disk_0.dvdd, _0_TestBloomFilteredLucene41Postings_0.blm, _0.nvm, _0_Disk_0.dvdm, _0_Direct_0.doc, _0_Pulsing41_0.doc, _0_Pulsing41_0.pos, _0_MockVariableIntBlock_0.frq, _0_TestBloomFilteredLucene41Postings_0.doc, _0_Direct_0.pos, _0_Direct_0.tip, _0.fdx, _0_Direct_0.tim, _0.fdt, segments_3]
[junit4:junit4]   2>  C240_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:57030_idf, base_url=http://127.0.0.1:57030/idf}
[junit4:junit4]   2> 27642 T1729 C240 P57030 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 27642 T1712 C238 P43388 oasc.SolrDeletionPolicy.updateCommits newest commit = 3[_0_TestBloomFilteredLucene41Postings_0.tim, _0_MockVariableIntBlock_0.doc, _0_TestBloomFilteredLucene41Postings_0.tip, _0_MockVariableIntBlock_0.skp, _0_TestBloomFilteredLucene41Postings_0.pos, _0_MockVariableIntBlock_0.pyl, _0_Pulsing41_0.tim, _0_MockVariableIntBlock_0.tib, _0_Pulsing41_0.tip, _0.nvd, _0_MockVariableIntBlock_0.pos, _0.si, _0_MockVariableIntBlock_0.tii, _0.fnm, _0_Disk_0.dvdd, _0_TestBloomFilteredLucene41Postings_0.blm, _0.nvm, _0_Disk_0.dvdm, _0_Direct_0.doc, _0_Pulsing41_0.doc, _0_Pulsing41_0.pos, _0_MockVariableIntBlock_0.frq, _0_TestBloomFilteredLucene41Postings_0.doc, _0_Direct_0.pos, _0_Direct_0.tip, _0.fdx, _0_Direct_0.tim, _0.fdt, segments_3]
[junit4:junit4]   2> 27643 T1729 C240 P57030 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@78ffcbf7 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5d697666),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@78ffcbf7 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5d697666),segFN=segments_3,generation=3,filenames=[segments_3]
[junit4:junit4]   2> 27643 T1729 C240 P57030 oasc.SolrDeletionPolicy.updateCommits newest commit = 3[segments_3]
[junit4:junit4]   2> 27644 T1729 C240 P57030 oass.SolrIndexSearcher.<init> Opening Searcher@63b3c0be main
[junit4:junit4]   2> 27644 T1729 C240 P57030 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 27645 T1741 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@63b3c0be main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 27645 T1729 C240 P57030 /update {waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 3
[junit4:junit4]   2> 27695 T1712 C238 P43388 oass.SolrIndexSearcher.<init> Opening Searcher@7eac872f main
[junit4:junit4]   2> 27696 T1712 C238 P43388 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 27697 T1724 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7eac872f main{StandardDirectoryReader(segments_3:3:nrt _0(5.0):C1)}
[junit4:junit4]   2> 27697 T1712 C238 P43388 /update {waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 60
[junit4:junit4]   2> 27698 T1686 C239 P57179 /update {waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 72
[junit4:junit4]   2> 27698 T1648 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):230
[junit4:junit4]   2> 27699 T1648 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 27700 T1648 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4]   2> 27703 T1660 C243 P65311 REQ /select {q=*:*&wt=javabin&version=2} hits=1 status=0 QTime=0 
[junit4:junit4]   2> 27704 T1648 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:47880/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@70df5fa1
[junit4:junit4]   2> 27705 T1648 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 27706 T1753 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:47880. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 27706 T1753 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:47880, initiating session
[junit4:junit4]   2> 27706 T1650 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:32138
[junit4:junit4]   2> 27707 T1650 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:32138
[junit4:junit4]   2> 27708 T1652 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6cf1e73b000d with negotiated timeout 10000 for client /140.211.11.196:32138
[junit4:junit4]   2> 27708 T1753 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:47880, sessionid = 0x13d6cf1e73b000d, negotiated timeout = 10000
[junit4:junit4]   2> 27709 T1

[...truncated too long message...]

esById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4]   2> 155822 T1739 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4]   2> 155822 T1739 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4]   2> 155823 T1739 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4]   2> 155826 T1739 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4]   2> 155827 T1739 oasc.CachingDirectoryFactory.close Closing directory when closing factory: ./org.apache.solr.cloud.BasicDistributedZk2Test-1363332425525/jetty4/index.20130315122730126
[junit4:junit4]   2> 155827 T1739 oasc.CachingDirectoryFactory.closeDirectory Closing directory: ./org.apache.solr.cloud.BasicDistributedZk2Test-1363332425525/jetty4/index.20130315122730126
[junit4:junit4]   2> 155827 T1739 oasc.CachingDirectoryFactory.close Closing directory when closing factory: ./org.apache.solr.cloud.BasicDistributedZk2Test-1363332425525/jetty4
[junit4:junit4]   2> 155827 T1739 oasc.CachingDirectoryFactory.closeDirectory Closing directory: ./org.apache.solr.cloud.BasicDistributedZk2Test-1363332425525/jetty4
[junit4:junit4]   2> 155828 T1739 oasc.CachingDirectoryFactory.close Closing directory when closing factory: ./org.apache.solr.cloud.BasicDistributedZk2Test-1363332425525/jetty4/index
[junit4:junit4]   2> 155828 T1739 oasc.CachingDirectoryFactory.closeDirectory Closing directory: ./org.apache.solr.cloud.BasicDistributedZk2Test-1363332425525/jetty4/index
[junit4:junit4]   2> 155828 T1739 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4]   2> 155829 T1739 oasc.LeaderElector$1.process WARNING  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard2/election/89347353839599628-5-n_0000000001
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4:junit4]   2> 	at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:152)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:149)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.delete(SolrZkClient.java:149)
[junit4:junit4]   2> 	at org.apache.solr.cloud.ElectionContext.cancelElection(ElectionContext.java:63)
[junit4:junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:246)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:156)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:100)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:129)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
[junit4:junit4]   2> 
[junit4:junit4]   2> 155830 T1739 oascc.ZkStateReader$3.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 155830 T1739 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 155830 T1739 oascc.ZkStateReader$2.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 155830 T1739 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 155831 T1739 oascc.ZkStateReader$2.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 155831 T1739 oasc.LeaderElector$1.process WARNING  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4:junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:235)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:232)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:232)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:84)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:129)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
[junit4:junit4]   2> 
[junit4:junit4]   2> 155832 T1739 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> NOTE: test params are: codec=Lucene42: {foo_b=PostingsFormat(name=Direct), n_f1=MockVariableIntBlock(baseBlockSize=41), foo_d=Pulsing41(freqCutoff=1 minBlockSize=41 maxBlockSize=111), foo_f=PostingsFormat(name=Direct), n_tl1=MockVariableIntBlock(baseBlockSize=41), n_d1=PostingsFormat(name=TestBloomFilteredLucene41Postings), rnd_b=Pulsing41(freqCutoff=1 minBlockSize=41 maxBlockSize=111), intDefault=PostingsFormat(name=TestBloomFilteredLucene41Postings), n_td1=MockVariableIntBlock(baseBlockSize=41), timestamp=PostingsFormat(name=TestBloomFilteredLucene41Postings), id=Pulsing41(freqCutoff=1 minBlockSize=41 maxBlockSize=111), range_facet_sl=PostingsFormat(name=Direct), range_facet_si=MockVariableIntBlock(baseBlockSize=41), oddField_s=PostingsFormat(name=Direct), multiDefault=MockVariableIntBlock(baseBlockSize=41), n_tf1=PostingsFormat(name=TestBloomFilteredLucene41Postings), n_dt1=MockVariableIntBlock(baseBlockSize=41), range_facet_l=PostingsFormat(name=TestBloomFilteredLucene41Postings), n_ti1=Pulsing41(freqCutoff=1 minBlockSize=41 maxBlockSize=111), text=PostingsFormat(name=Direct), _version_=PostingsFormat(name=TestBloomFilteredLucene41Postings), SubjectTerms_mfacet=PostingsFormat(name=TestBloomFilteredLucene41Postings), a_t=PostingsFormat(name=TestBloomFilteredLucene41Postings), n_tdt1=PostingsFormat(name=TestBloomFilteredLucene41Postings), other_tl1=PostingsFormat(name=TestBloomFilteredLucene41Postings), n_l1=PostingsFormat(name=TestBloomFilteredLucene41Postings), a_si=MockVariableIntBlock(baseBlockSize=41)}, docValues:{timestamp=DocValuesFormat(name=Disk)}, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=is, timezone=Asia/Ashgabat
[junit4:junit4]   2> NOTE: FreeBSD 9.0-RELEASE amd64/Sun Microsystems Inc. 1.6.0_32 (64-bit)/cpus=16,threads=2,free=89818736,total=393805824
[junit4:junit4]   2> NOTE: All tests run in this JVM: [TestRangeQuery, TestFiltering, OpenCloseCoreStressTest, TestMultiCoreConfBootstrap, ChaosMonkeySafeLeaderTest, TestUtils, TestLMJelinekMercerSimilarityFactory, SolrCoreCheckLockOnStartupTest, UpdateParamsTest, StandardRequestHandlerTest, TestDistributedSearch, JSONWriterTest, TestDFRSimilarityFactory, TestBinaryField, TestBinaryResponseWriter, TestNumberUtils, ShardRoutingCustomTest, SolrRequestParserTest, MultiTermTest, PathHierarchyTokenizerFactoryTest, BadIndexSchemaTest, TestFoldingMultitermQuery, TestRTGBase, SolrCmdDistributorTest, TestLuceneMatchVersion, TestAnalyzedSuggestions, SuggesterTSTTest, DirectUpdateHandlerTest, TestPartialUpdateDeduplication, OpenExchangeRatesOrgProviderTest, CollectionsAPIDistributedZkTest, DocumentAnalysisRequestHandlerTest, HighlighterTest, UniqFieldsUpdateProcessorFactoryTest, TestStressReorder, TestDynamicFieldCollectionResource, TestQueryTypes, IndexBasedSpellCheckerTest, BasicDistributedZk2Test]
[junit4:junit4] Completed on J1 in 156.85s, 1 test, 1 error <<< FAILURES!

[...truncated 515 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/build.xml:381: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/build.xml:361: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build.xml:183: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/common-build.xml:449: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/lucene/common-build.xml:1213: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/lucene/common-build.xml:877: There were test failures: 272 suites, 1144 tests, 1 error, 27 ignored (8 assumptions)

Total time: 53 minutes 5 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-Tests-trunk-Java6 - Build # 16004 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-trunk-Java6/16004/

1 tests failed.
FAILED:  org.apache.solr.cloud.BasicDistributedZk2Test.testDistribSearch

Error Message:
No registered leader was found, collection:collection1 slice:shard1

Stack Trace:
org.apache.solr.common.SolrException: No registered leader was found, collection:collection1 slice:shard1
	at __randomizedtesting.SeedInfo.seed([5949ACF497AB2B3F:D8AF22ECE0F44B03]:0)
	at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:484)
	at org.apache.solr.cloud.BasicDistributedZk2Test.brindDownShardIndexSomeDocsAndRecover(BasicDistributedZk2Test.java:291)
	at org.apache.solr.cloud.BasicDistributedZk2Test.doTest(BasicDistributedZk2Test.java:116)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:806)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:679)




Build Log:
[...truncated 8744 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.BasicDistributedZk2Test
[junit4:junit4]   2> 0 T2030 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4:junit4]   2> 6 T2030 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4]   2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./solrtest-BasicDistributedZk2Test-1363304656708
[junit4:junit4]   2> 7 T2030 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 8 T2031 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4]   2> 11 T2031 oazs.ZooKeeperServer.setTickTime tickTime set to 1000
[junit4:junit4]   2> 11 T2031 oazs.ZooKeeperServer.setMinSessionTimeout minSessionTimeout set to -1
[junit4:junit4]   2> 11 T2031 oazs.ZooKeeperServer.setMaxSessionTimeout maxSessionTimeout set to -1
[junit4:junit4]   2> 12 T2031 oazs.NIOServerCnxnFactory.configure binding to port 0.0.0.0/0.0.0.0:0
[junit4:junit4]   2> 13 T2031 oazsp.FileTxnSnapLog.save Snapshotting: 0x0 to /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-1363304656708/zookeeper/server1/data/version-2/snapshot.0
[junit4:junit4]   2> 108 T2030 oasc.ZkTestServer.run start zk server on port:53052
[junit4:junit4]   2> 108 T2030 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:53052 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@3a748d4d
[junit4:junit4]   2> 109 T2030 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 110 T2036 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:53052. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 110 T2036 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:53052, initiating session
[junit4:junit4]   2> 111 T2032 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:53053
[junit4:junit4]   2> 111 T2032 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:53053
[junit4:junit4]   2> 112 T2034 oazsp.FileTxnLog.append Creating new log file: log.1
[junit4:junit4]   2> 140 T2034 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6b4a2f4c0000 with negotiated timeout 10000 for client /140.211.11.196:53053
[junit4:junit4]   2> 140 T2036 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:53052, sessionid = 0x13d6b4a2f4c0000, negotiated timeout = 10000
[junit4:junit4]   2> 141 T2037 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3a748d4d name:ZooKeeperConnection Watcher:127.0.0.1:53052 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 141 T2030 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 142 T2030 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4]   2> 151 T2035 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d6b4a2f4c0000
[junit4:junit4]   2> 152 T2037 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 152 T2030 oaz.ZooKeeper.close Session: 0x13d6b4a2f4c0000 closed
[junit4:junit4]   2> 153 T2032 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:53053 which had sessionid 0x13d6b4a2f4c0000
[junit4:junit4]   2> 153 T2030 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:53052/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@23e31e9e
[junit4:junit4]   2> 154 T2030 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 154 T2038 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:53052. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 155 T2038 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:53052, initiating session
[junit4:junit4]   2> 155 T2032 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:53054
[junit4:junit4]   2> 156 T2032 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:53054
[junit4:junit4]   2> 157 T2034 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6b4a2f4c0001 with negotiated timeout 10000 for client /140.211.11.196:53054
[junit4:junit4]   2> 157 T2038 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:53052, sessionid = 0x13d6b4a2f4c0001, negotiated timeout = 10000
[junit4:junit4]   2> 157 T2039 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@23e31e9e name:ZooKeeperConnection Watcher:127.0.0.1:53052/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 157 T2030 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 158 T2030 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4]   2> 167 T2030 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4]   2> 170 T2030 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4]   2> 172 T2030 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4]   2> 180 T2030 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 181 T2030 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 193 T2030 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4]   2> 194 T2030 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4]   2> 298 T2030 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4]   2> 299 T2030 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4]   2> 302 T2030 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4]   2> 303 T2030 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4]   2> 313 T2030 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4]   2> 313 T2030 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4]   2> 316 T2030 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 317 T2030 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 320 T2030 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 320 T2030 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 323 T2030 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 324 T2030 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 327 T2030 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4]   2> 327 T2030 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4]   2> 335 T2035 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d6b4a2f4c0001
[junit4:junit4]   2> 335 T2039 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 336 T2032 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:53054 which had sessionid 0x13d6b4a2f4c0001
[junit4:junit4]   2> 335 T2030 oaz.ZooKeeper.close Session: 0x13d6b4a2f4c0001 closed
[junit4:junit4]   2> 524 T2030 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 528 T2030 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:53055
[junit4:junit4]   2> 529 T2030 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 529 T2030 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 530 T2030 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1363304657039
[junit4:junit4]   2> 530 T2030 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1363304657039/solr.xml
[junit4:junit4]   2> 531 T2030 oasc.CoreContainer.<init> New CoreContainer 1017836972
[junit4:junit4]   2> 531 T2030 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1363304657039/'
[junit4:junit4]   2> 532 T2030 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1363304657039/'
[junit4:junit4]   2> 586 T2030 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 586 T2030 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 587 T2030 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 587 T2030 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 588 T2030 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 588 T2030 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 589 T2030 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 589 T2030 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 590 T2030 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 590 T2030 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 613 T2030 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 633 T2030 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:53052/solr
[junit4:junit4]   2> 634 T2030 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 634 T2030 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:53052 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@7e5d2763
[junit4:junit4]   2> 635 T2030 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 636 T2049 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:53052. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 636 T2049 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:53052, initiating session
[junit4:junit4]   2> 637 T2032 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:53056
[junit4:junit4]   2> 637 T2032 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:53056
[junit4:junit4]   2> 639 T2034 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6b4a2f4c0002 with negotiated timeout 20000 for client /140.211.11.196:53056
[junit4:junit4]   2> 639 T2049 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:53052, sessionid = 0x13d6b4a2f4c0002, negotiated timeout = 20000
[junit4:junit4]   2> 640 T2050 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7e5d2763 name:ZooKeeperConnection Watcher:127.0.0.1:53052 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 640 T2030 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 641 T2035 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d6b4a2f4c0002
[junit4:junit4]   2> 643 T2050 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 643 T2032 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:53056 which had sessionid 0x13d6b4a2f4c0002
[junit4:junit4]   2> 643 T2030 oaz.ZooKeeper.close Session: 0x13d6b4a2f4c0002 closed
[junit4:junit4]   2> 643 T2030 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 656 T2030 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:53052/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@6d10d74c
[junit4:junit4]   2> 656 T2030 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 657 T2051 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:53052. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 657 T2051 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:53052, initiating session
[junit4:junit4]   2> 658 T2032 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:53057
[junit4:junit4]   2> 658 T2032 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:53057
[junit4:junit4]   2> 659 T2034 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6b4a2f4c0003 with negotiated timeout 20000 for client /140.211.11.196:53057
[junit4:junit4]   2> 659 T2051 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:53052, sessionid = 0x13d6b4a2f4c0003, negotiated timeout = 20000
[junit4:junit4]   2> 660 T2052 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6d10d74c name:ZooKeeperConnection Watcher:127.0.0.1:53052/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 660 T2030 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 662 T2035 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6b4a2f4c0003 type:create cxid:0x2 zxid:0x1a txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 671 T2030 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4]   2> 673 T2030 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53055_
[junit4:junit4]   2> 674 T2035 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6b4a2f4c0003 type:delete cxid:0x8 zxid:0x1c txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:53055_ Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:53055_
[junit4:junit4]   2> 675 T2030 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53055_
[junit4:junit4]   2> 685 T2030 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4]   2> 698 T2035 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6b4a2f4c0003 type:delete cxid:0x16 zxid:0x21 txntype:-1 reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4]   2> 699 T2030 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 702 T2030 oasc.Overseer.start Overseer (id=89345533982539779-127.0.0.1:53055_-n_0000000000) starting
[junit4:junit4]   2> 702 T2035 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6b4a2f4c0003 type:create cxid:0x1b zxid:0x23 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 703 T2035 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6b4a2f4c0003 type:create cxid:0x1c zxid:0x24 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 714 T2035 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6b4a2f4c0003 type:create cxid:0x1d zxid:0x25 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 715 T2035 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6b4a2f4c0003 type:create cxid:0x1e zxid:0x26 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 717 T2054 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4]   2> 718 T2030 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4]   2> 730 T2030 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4]   2> 732 T2030 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 736 T2053 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 738 T2055 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1363304657039/collection1
[junit4:junit4]   2> 738 T2055 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4]   2> 739 T2055 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 739 T2055 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4]   2> 740 T2055 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1363304657039/collection1/'
[junit4:junit4]   2> 741 T2055 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1363304657039/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 742 T2055 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1363304657039/collection1/lib/README' to classloader
[junit4:junit4]   2> 792 T2055 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 854 T2055 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 955 T2055 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 965 T2055 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 1619 T2055 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 1628 T2055 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 1632 T2055 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 1641 T2055 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 1644 T2055 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 1648 T2055 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 1648 T2055 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 1649 T2055 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 1649 T2055 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 1650 T2055 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 1650 T2055 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 1650 T2055 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 1650 T2055 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1363304657039/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363304656708/control/data/
[junit4:junit4]   2> 1651 T2055 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@561777b1
[junit4:junit4]   2> 1651 T2055 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 1651 T2055 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363304656708/control/data forceNew: false
[junit4:junit4]   2> 1652 T2055 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1363304656708/control/data/index/
[junit4:junit4]   2> 1652 T2055 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1363304656708/control/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 1652 T2055 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363304656708/control/data/index forceNew: false
[junit4:junit4]   2> 1654 T2055 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=BaseDirectoryWrapper(org.apache.lucene.store.RAMDirectory@3ed8833f lockFactory=org.apache.lucene.store.NativeFSLockFactory@667e13a2),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 1655 T2055 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 1656 T2055 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 1657 T2055 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 1657 T2055 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 1657 T2055 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 1658 T2055 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 1658 T2055 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 1658 T2055 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 1658 T2055 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 1659 T2055 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 1664 T2055 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 1668 T2055 oass.SolrIndexSearcher.<init> Opening Searcher@fbe338f main
[junit4:junit4]   2> 1669 T2055 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363304656708/control/data/tlog
[junit4:junit4]   2> 1670 T2055 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 1670 T2055 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 1672 T2056 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@fbe338f main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 1673 T2055 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 1673 T2055 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 1674 T2035 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6b4a2f4c0003 type:create cxid:0x55 zxid:0x2a txntype:-1 reqpath:n/a Error Path:/solr/overseer/queue Error:KeeperErrorCode = NoNode for /solr/overseer/queue
[junit4:junit4]   2> 2239 T2053 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2240 T2053 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"control_collection",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:53055_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:53055"}
[junit4:junit4]   2> 2240 T2053 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4]   2> 2240 T2053 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 2240 T2035 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6b4a2f4c0003 type:create cxid:0x5d zxid:0x2d txntype:-1 reqpath:n/a Error Path:/solr/overseer/queue-work Error:KeeperErrorCode = NoNode for /solr/overseer/queue-work
[junit4:junit4]   2> 2254 T2052 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 2687 T2055 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 2687 T2055 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:53055 collection:control_collection shard:shard1
[junit4:junit4]   2> 2688 T2055 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4]   2> 2704 T2035 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6b4a2f4c0003 type:delete cxid:0x77 zxid:0x37 txntype:-1 reqpath:n/a Error Path:/solr/collections/control_collection/leaders Error:KeeperErrorCode = NoNode for /solr/collections/control_collection/leaders
[junit4:junit4]   2> 2705 T2055 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 2705 T2035 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6b4a2f4c0003 type:create cxid:0x78 zxid:0x38 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 2708 T2055 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 2708 T2055 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 2708 T2055 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:53055/collection1/
[junit4:junit4]   2> 2708 T2055 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 2709 T2055 oasc.SyncStrategy.syncToMe http://127.0.0.1:53055/collection1/ has no replicas
[junit4:junit4]   2> 2709 T2055 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:53055/collection1/
[junit4:junit4]   2> 2709 T2055 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4]   2> 2722 T2035 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6b4a2f4c0003 type:create cxid:0x82 zxid:0x3c txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 3764 T2053 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 3815 T2052 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 3897 T2055 oasc.ZkController.register We are http://127.0.0.1:53055/collection1/ and leader is http://127.0.0.1:53055/collection1/
[junit4:junit4]   2> 3897 T2055 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:53055
[junit4:junit4]   2> 3897 T2055 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 3897 T2055 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 3897 T2055 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 3921 T2055 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 3922 T2030 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0
[junit4:junit4]   2> 3922 T2030 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 3923 T2030 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 3931 T2030 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4]   2> 3932 T2030 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:53052/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@fad72f7
[junit4:junit4]   2> 3933 T2030 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 3933 T2058 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:53052. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 3934 T2058 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:53052, initiating session
[junit4:junit4]   2> 3934 T2032 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:53058
[junit4:junit4]   2> 3934 T2032 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:53058
[junit4:junit4]   2> 3937 T2034 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6b4a2f4c0004 with negotiated timeout 10000 for client /140.211.11.196:53058
[junit4:junit4]   2> 3937 T2058 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:53052, sessionid = 0x13d6b4a2f4c0004, negotiated timeout = 10000
[junit4:junit4]   2> 3938 T2059 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@fad72f7 name:ZooKeeperConnection Watcher:127.0.0.1:53052/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 3938 T2030 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 3940 T2030 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 3942 T2030 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4]   2> 4080 T2030 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 4082 T2030 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:53059
[junit4:junit4]   2> 4083 T2030 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 4083 T2030 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 4083 T2030 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363304660644
[junit4:junit4]   2> 4084 T2030 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363304660644/solr.xml
[junit4:junit4]   2> 4084 T2030 oasc.CoreContainer.<init> New CoreContainer 183606507
[junit4:junit4]   2> 4084 T2030 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363304660644/'
[junit4:junit4]   2> 4085 T2030 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363304660644/'
[junit4:junit4]   2> 4122 T2030 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 4122 T2030 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 4122 T2030 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 4123 T2030 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 4123 T2030 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 4123 T2030 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 4124 T2030 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 4124 T2030 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 4124 T2030 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 4125 T2030 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 4138 T2030 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 4152 T2030 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:53052/solr
[junit4:junit4]   2> 4152 T2030 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 4153 T2030 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:53052 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@68e47bd4
[junit4:junit4]   2> 4153 T2030 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 4154 T2069 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:53052. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 4154 T2069 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:53052, initiating session
[junit4:junit4]   2> 4155 T2032 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:53060
[junit4:junit4]   2> 4155 T2032 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:53060
[junit4:junit4]   2> 4160 T2034 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6b4a2f4c0005 with negotiated timeout 20000 for client /140.211.11.196:53060
[junit4:junit4]   2> 4160 T2069 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:53052, sessionid = 0x13d6b4a2f4c0005, negotiated timeout = 20000
[junit4:junit4]   2> 4161 T2070 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@68e47bd4 name:ZooKeeperConnection Watcher:127.0.0.1:53052 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 4161 T2030 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 4162 T2035 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d6b4a2f4c0005
[junit4:junit4]   2> 4177 T2070 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 4177 T2032 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:53060 which had sessionid 0x13d6b4a2f4c0005
[junit4:junit4]   2> 4177 T2030 oaz.ZooKeeper.close Session: 0x13d6b4a2f4c0005 closed
[junit4:junit4]   2> 4178 T2030 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 4188 T2030 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:53052/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@20b405ff
[junit4:junit4]   2> 4189 T2030 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 4189 T2071 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:53052. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 4190 T2071 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:53052, initiating session
[junit4:junit4]   2> 4190 T2032 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:53061
[junit4:junit4]   2> 4190 T2032 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:53061
[junit4:junit4]   2> 4192 T2034 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6b4a2f4c0006 with negotiated timeout 20000 for client /140.211.11.196:53061
[junit4:junit4]   2> 4192 T2071 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:53052, sessionid = 0x13d6b4a2f4c0006, negotiated timeout = 20000
[junit4:junit4]   2> 4192 T2072 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@20b405ff name:ZooKeeperConnection Watcher:127.0.0.1:53052/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 4193 T2030 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 4194 T2035 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6b4a2f4c0006 type:create cxid:0x1 zxid:0x4a txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 4196 T2035 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6b4a2f4c0006 type:create cxid:0x2 zxid:0x4b txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 4198 T2030 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 5202 T2030 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53059_
[junit4:junit4]   2> 5202 T2035 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6b4a2f4c0006 type:delete cxid:0xb zxid:0x4c txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:53059_ Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:53059_
[junit4:junit4]   2> 5206 T2030 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53059_
[junit4:junit4]   2> 5216 T2052 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 5216 T2072 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 5216 T2059 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 5216 T2052 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 5221 T2073 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363304660644/collection1
[junit4:junit4]   2> 5221 T2073 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 5222 T2073 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 5222 T2073 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 5223 T2073 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363304660644/collection1/'
[junit4:junit4]   2> 5224 T2073 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363304660644/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 5224 T2073 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363304660644/collection1/lib/README' to classloader
[junit4:junit4]   2> 5257 T2073 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 5297 T2073 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 5367 T2053 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 5368 T2053 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"control_collection",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:53055_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:53055"}
[junit4:junit4]   2> 5371 T2059 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 5372 T2052 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 5398 T2073 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 5398 T2072 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 5402 T2073 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 5820 T2073 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 5828 T2073 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 5831 T2073 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 5843 T2073 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 5846 T2073 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 5849 T2073 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 5850 T2073 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 5851 T2073 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 5851 T2073 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 5852 T2073 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 5852 T2073 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 5853 T2073 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 5853 T2073 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363304660644/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363304656708/jetty1/
[junit4:junit4]   2> 5853 T2073 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@561777b1
[junit4:junit4]   2> 5853 T2073 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 5854 T2073 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363304656708/jetty1 forceNew: false
[junit4:junit4]   2> 5854 T2073 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1363304656708/jetty1/index/
[junit4:junit4]   2> 5854 T2073 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1363304656708/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 5855 T2073 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363304656708/jetty1/index forceNew: false
[junit4:junit4]   2> 5863 T2073 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=BaseDirectoryWrapper(org.apache.lucene.store.RAMDirectory@1c5fdc11 lockFactory=org.apache.lucene.store.NativeFSLockFactory@780b7ed7),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 5863 T2073 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 5865 T2073 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 5865 T2073 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 5865 T2073 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 5866 T2073 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 5866 T2073 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 5866 T2073 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 5867 T2073 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 5867 T2073 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 5867 T2073 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 5872 T2073 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 5876 T2073 oass.SolrIndexSearcher.<init> Opening Searcher@55a68f4c main
[junit4:junit4]   2> 5876 T2073 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363304656708/jetty1/tlog
[junit4:junit4]   2> 5877 T2073 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 5877 T2073 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 5879 T2074 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@55a68f4c main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 5881 T2073 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 5881 T2073 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 6875 T2053 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 6876 T2053 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:53059_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:53059"}
[junit4:junit4]   2> 6876 T2053 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 2
[junit4:junit4]   2> 6877 T2053 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 6895 T2072 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 6895 T2052 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 6895 T2059 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 7885 T2073 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 7885 T2073 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:53059 collection:collection1 shard:shard1
[junit4:junit4]   2> 7886 T2073 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 7909 T2035 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6b4a2f4c0006 type:delete cxid:0x48 zxid:0x5c txntype:-1 reqpath:n/a Error Path:/solr/collections/collection1/leaders Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders
[junit4:junit4]   2> 7915 T2073 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 7915 T2035 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6b4a2f4c0006 type:create cxid:0x49 zxid:0x5d txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 7918 T2073 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 7918 T2073 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 7919 T2073 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:53059/collection1/
[junit4:junit4]   2> 7919 T2073 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 7919 T2073 oasc.SyncStrategy.syncToMe http://127.0.0.1:53059/collection1/ has no replicas
[junit4:junit4]   2> 7919 T2073 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:53059/collection1/
[junit4:junit4]   2> 7920 T2073 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4]   2> 7929 T2035 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6b4a2f4c0006 type:create cxid:0x53 zxid:0x61 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 8400 T2053 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 8416 T2052 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 8416 T2072 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 8416 T2059 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 8447 T2073 oasc.ZkController.register We are http://127.0.0.1:53059/collection1/ and leader is http://127.0.0.1:53059/collection1/
[junit4:junit4]   2> 8447 T2073 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:53059
[junit4:junit4]   2> 8447 T2073 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 8447 T2073 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 8448 T2073 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 8450 T2073 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 8451 T2030 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0
[junit4:junit4]   2> 8452 T2030 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 8452 T2030 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 8659 T2030 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 8662 T2030 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:43851
[junit4:junit4]   2> 8663 T2030 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 8663 T2030 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 8664 T2030 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1363304665170
[junit4:junit4]   2> 8664 T2030 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1363304665170/solr.xml
[junit4:junit4]   2> 8665 T2030 oasc.CoreContainer.<init> New CoreContainer 638331777
[junit4:junit4]   2> 8665 T2030 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1363304665170/'
[junit4:junit4]   2> 8666 T2030 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1363304665170/'
[junit4:junit4]   2> 8714 T2030 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 8715 T2030 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 8715 T2030 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 8716 T2030 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 8716 T2030 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 8716 T2030 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 8717 T2030 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 8717 T2030 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 8718 T2030 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 8718 T2030 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 8744 T2030 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 8766 T2030 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:53052/solr
[junit4:junit4]   2> 8767 T2030 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 8768 T2030 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:53052 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@62761bf
[junit4:junit4]   2> 8769 T2030 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 8769 T2085 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:53052. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 8770 T2085 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:53052, initiating session
[junit4:junit4]   2> 8771 T2032 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:47433
[junit4:junit4]   2> 8771 T2032 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:47433
[junit4:junit4]   2> 8818 T2034 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6b4a2f4c0007 with negotiated timeout 20000 for client /140.211.11.196:47433
[junit4:junit4]   2> 8818 T2085 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:53052, sessionid = 0x13d6b4a2f4c0007, negotiated timeout = 20000
[junit4:junit4]   2> 8819 T2086 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@62761bf name:ZooKeeperConnection Watcher:127.0.0.1:53052 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 8819 T2030 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 8821 T2035 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d6b4a2f4c0007
[junit4:junit4]   2> 8846 T2086 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 8846 T2032 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:47433 which had sessionid 0x13d6b4a2f4c0007
[junit4:junit4]   2> 8846 T2030 oaz.ZooKeeper.close Session: 0x13d6b4a2f4c0007 closed
[junit4:junit4]   2> 8847 T2030 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 8860 T2030 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:53052/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@3fe8f8e7
[junit4:junit4]   2> 8861 T2030 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 8862 T2087 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:53052. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 8862 T2087 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:53052, initiating session
[junit4:junit4]   2> 8863 T2032 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:25259
[junit4:junit4]   2> 8863 T2032 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:25259
[junit4:junit4]   2> 8871 T2034 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6b4a2f4c0008 with negotiated timeout 20000 for client /140.211.11.196:25259
[junit4:junit4]   2> 8871 T2087 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:53052, sessionid = 0x13d6b4a2f4c0008, negotiated timeout = 20000
[junit4:junit4]   2> 8871 T2088 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3fe8f8e7 name:ZooKeeperConnection Watcher:127.0.0.1:53052/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 8872 T2030 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 8873 T2035 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6b4a2f4c0008 type:create cxid:0x1 zxid:0x6e txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 8920 T2035 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6b4a2f4c0008 type:create cxid:0x2 zxid:0x6f txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 8922 T2030 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 9926 T2030 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43851_
[junit4:junit4]   2> 9927 T2035 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6b4a2f4c0008 type:delete cxid:0xb zxid:0x70 txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:43851_ Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:43851_
[junit4:junit4]   2> 9929 T2030 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:43851_
[junit4:junit4]   2> 9932 T2053 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 9932 T2059 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 9932 T2088 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 9932 T2059 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 9932 T2072 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 9933 T2053 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:53059_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:53059"}
[junit4:junit4]   2> 9933 T2072 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 9946 T2089 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1363304665170/collection1
[junit4:junit4]   2> 9946 T2089 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 9948 T2089 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 9949 T2089 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 9950 T2072 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 9950 T2088 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 9950 T2059 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 9950 T2089 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1363304665170/collection1/'
[junit4:junit4]   2> 9952 T2089 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1363304665170/collection1/lib/README' to classloader
[junit4:junit4]   2> 9952 T2089 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1363304665170/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 9953 T2052 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 9953 T2052 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 9954 T2052 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 9999 T2089 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 10055 T2089 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 10156 T2089 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 10162 T2089 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 10770 T2089 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 10782 T2089 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 10786 T2089 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 10800 T2089 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 10805 T2089 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 10810 T2089 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 10811 T2089 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 10812 T2089 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 10812 T2089 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 10813 T2089 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 10814 T2089 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 10814 T2089 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 10814 T2089 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1363304665170/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363304656708/jetty2/
[junit4:junit4]   2> 10815 T2089 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@561777b1
[junit4:junit4]   2> 10815 T2089 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 10816 T2089 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363304656708/jetty2 forceNew: false
[junit4:junit4]   2> 10816 T2089 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1363304656708/jetty2/index/
[junit4:junit4]   2> 10816 T2089 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1363304656708/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 10817 T2089 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363304656708/jetty2/index forceNew: false
[junit4:junit4]   2> 10819 T2089 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=BaseDirectoryWrapper(org.apache.lucene.store.RAMDirectory@2074bda8 lockFactory=org.apache.lucene.store.NativeFSLockFactory@f9384fc),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 10820 T2089 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 10822 T2089 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 10822 T2089 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 10823 T2089 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 10824 T2089 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 10825 T2089 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 10825 T2089 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 10826 T2089 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 10827 T2089 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 10827 T2089 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 10840 T2089 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 10849 T2089 oass.SolrIndexSearcher.<init> Opening Searcher@62dce75 main
[junit4:junit4]   2> 10849 T2089 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363304656708/jetty2/tlog
[junit4:junit4]   2> 10851 T2089 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 10851 T2089 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 10857 T2090 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@62dce75 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 10860 T2089 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 10861 T2089 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 11456 T2053 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 11457 T2053 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:43851_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:43851"}
[junit4:junit4]   2> 11457 T2053 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4]   2> 11457 T2053 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4:junit4]   2> 11473 T2059 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 11473 T2052 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 11473 T2072 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 11473 T2088 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 11864 T2089 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 11864 T2089 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:43851 collection:collection1 shard:shard2
[junit4:junit4]   2> 11865 T2089 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4:junit4]   2> 11870 T2035 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6b4a2f4c0008 type:delete cxid:0x47 zxid:0x7f txntype:-1 reqpath:n/a Error Path:/solr/collections/collection1/leaders/shard2 Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders/shard2
[junit4:junit4]   2> 11871 T2089 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 11871 T2035 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6b4a2f4c0008 type:create cxid:0x48 zxid:0x80 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 11873 T2089 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 11874 T2089 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 11874 T2089 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:43851/collection1/
[junit4:junit4]   2> 11874 T2089 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 11874 T2089 oasc.SyncStrategy.syncToMe http://127.0.0.1:43851/collection1/ has no replicas
[junit4:junit4]   2> 11874 T2089 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:43851/collection1/
[junit4:junit4]   2> 11874 T2089 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4]   2> 11877 T2035 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6b4a2f4c0008 type:create cxid:0x51 zxid:0x83 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 12978 T2053 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 12994 T2059 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 12994 T2052 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 12994 T2072 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 12994 T2088 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 13010 T2089 oasc.ZkController.register We are http://127.0.0.1:43851/collection1/ and leader is http://127.0.0.1:43851/collection1/
[junit4:junit4]   2> 13010 T2089 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:43851
[junit4:junit4]   2> 13010 T2089 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 13010 T2089 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 13011 T2089 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 13012 T2089 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 13013 T2030 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0
[junit4:junit4]   2> 13014 T2030 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 13014 T2030 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 13215 T2030 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 13217 T2030 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:19225
[junit4:junit4]   2> 13218 T2030 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 13218 T2030 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 13219 T2030 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1363304669730
[junit4:junit4]   2> 13219 T2030 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1363304669730/solr.xml
[junit4:junit4]   2> 13220 T2030 oasc.CoreContainer.<init> New CoreContainer 1014281166
[junit4:junit4]   2> 13220 T2030 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1363304669730/'
[junit4:junit4]   2> 13221 T2030 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1363304669730/'
[junit4:junit4]   2> 13269 T2030 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 13270 T2030 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 13270 T2030 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 13271 T2030 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 13271 T2030 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 13272 T2030 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 13272 T2030 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 13273 T2030 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 13273 T2030 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 13274 T2030 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 13293 T2030 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 13311 T2030 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:53052/solr
[junit4:junit4]   2> 13312 T2030 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 13312 T2030 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:53052 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@10c1b65b
[junit4:junit4]   2> 13313 T2030 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 13314 T2101 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:53052. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 13314 T2101 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:53052, initiating session
[junit4:junit4]   2> 13315 T2032 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:35736
[junit4:junit4]   2> 13315 T2032 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:35736
[junit4:junit4]   2> 13317 T2034 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6b4a2f4c0009 with negotiated timeout 20000 for client /140.211.11.196:35736
[junit4:junit4]   2> 13317 T2101 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:53052, sessionid = 0x13d6b4a2f4c0009, negotiated timeout = 20000
[junit4:junit4]   2> 13317 T2102 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@10c1b65b name:ZooKeeperConnection Watcher:127.0.0.1:53052 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 13318 T2030 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 13319 T2035 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d6b4a2f4c0009
[junit4:junit4]   2> 13320 T2102 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 13321 T2032 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:35736 which had sessionid 0x13d6b4a2f4c0009
[junit4:junit4]   2> 13320 T2030 oaz.ZooKeeper.close Session: 0x13d6b4a2f4c0009 closed
[junit4:junit4]   2> 13321 T2030 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 13331 T2030 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:53052/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@43bdb9f3
[junit4:junit4]   2> 13332 T2030 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 13332 T2103 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:53052. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 13333 T2103 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:53052, initiating session
[junit4:junit4]   2> 13333 T2032 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:14388
[junit4:junit4]   2> 13333 T2032 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:14388
[junit4:junit4]   2> 13334 T2034 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6b4a2f4c000a with negotiated timeout 20000 for client /140.211.11.196:14388
[junit4:junit4]   2> 13334 T2103 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:53052, sessionid = 0x13d6b4a2f4c000a, negotiated timeout = 20000
[junit4:junit4]   2> 13335 T2104 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@43bdb9f3 name:ZooKeeperConnection Watcher:127.0.0.1:53052/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 13335 T2030 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 13336 T2035 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6b4a2f4c000a type:create cxid:0x1 zxid:0x90 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 13337 T2035 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6b4a2f4c000a type:create cxid:0x2 zxid:0x91 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 13339 T2030 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 14343 T2030 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:19225_
[junit4:junit4]   2> 14344 T2035 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6b4a2f4c000a type:delete cxid:0xb zxid:0x92 txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:19225_ Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:19225_
[junit4:junit4]   2> 14348 T2030 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:19225_
[junit4:junit4]   2> 14352 T2072 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 14352 T2059 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 14352 T2088 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 14352 T2072 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 14352 T2104 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 14352 T2052 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 14353 T2059 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 14352 T2088 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 14353 T2052 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 14360 T2105 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1363304669730/collection1
[junit4:junit4]   2> 14360 T2105 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 14361 T2105 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 14361 T2105 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 14362 T2105 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1363304669730/collection1/'
[junit4:junit4]   2> 14363 T2105 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1363304669730/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 14364 T2105 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1363304669730/collection1/lib/README' to classloader
[junit4:junit4]   2> 14411 T2105 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 14468 T2105 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 14508 T2053 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 14509 T2053 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":"shard2",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:43851_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:43851"}
[junit4:junit4]   2> 14521 T2052 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 14521 T2059 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 14521 T2088 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 14521 T2072 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 14569 T2105 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 14569 T2104 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 14575 T2105 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 15643 T2105 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 15660 T2105 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 15666 T2105 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 15681 T2105 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 15688 T2105 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 15694 T2105 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 15696 T2105 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 15697 T2105 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 15697 T2105 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 15698 T2105 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 15698 T2105 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 15699 T2105 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 15699 T2105 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1363304669730/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363304656708/jetty3/
[junit4:junit4]   2> 15699 T2105 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@561777b1
[junit4:junit4]   2> 15700 T2105 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 15701 T2105 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363304656708/jetty3 forceNew: false
[junit4:junit4]   2> 15701 T2105 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1363304656708/jetty3/index/
[junit4:junit4]   2> 15701 T2105 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1363304656708/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 15701 T2105 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363304656708/jetty3/index forceNew: false
[junit4:junit4]   2> 15706 T2105 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=BaseDirectoryWrapper(org.apache.lucene.store.RAMDirectory@431ca8ce lockFactory=org.apache.lucene.store.NativeFSLockFactory@31417dd9),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 15706 T2105 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 15709 T2105 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 15710 T2105 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 15710 T2105 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 15711 T2105 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 15712 T2105 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 15712 T2105 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 15712 T2105 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 15713 T2105 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 15713 T2105 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 15728 T2105 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 15738 T2105 oass.SolrIndexSearcher.<init> Opening Searcher@e2554ff main
[junit4:junit4]   2> 15739 T2105 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363304656708/jetty3/tlog
[junit4:junit4]   2> 15740 T2105 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 15740 T2105 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 15744 T2106 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@e2554ff main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 15746 T2105 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 15746 T2105 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 16026 T2053 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 16027 T2053 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:19225_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:19225"}
[junit4:junit4]   2> 16027 T2053 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4]   2> 16028 T2053 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 16043 T2059 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 16043 T2052 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 16043 T2088 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 16043 T2072 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 16045 T2104 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 16774 T2105 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 16774 T2105 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:19225 collection:collection1 shard:shard1
[junit4:junit4]   2> 16777 T2105 oasc.ZkController.register We are http://127.0.0.1:19225/collection1/ and leader is http://127.0.0.1:53059/collection1/
[junit4:junit4]   2> 16778 T2105 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:19225
[junit4:junit4]   2> 16778 T2105 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 16778 T2105 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C115 name=collection1 org.apache.solr.core.SolrCore@8e20c43 url=http://127.0.0.1:19225/collection1 node=127.0.0.1:19225_ C115_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:19225_, base_url=http://127.0.0.1:19225}
[junit4:junit4]   2> 16778 T2107 C115 P19225 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 16779 T2105 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 16780 T2107 C115 P19225 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 16780 T2107 C115 P19225 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4]   2> 16780 T2107 C115 P19225 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 16781 T2030 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0
[junit4:junit4]   2> 16781 T2030 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 16782 T2107 C115 P19225 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 16782 T2030 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 16988 T2030 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 16991 T2030 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:31182
[junit4:junit4]   2> 16992 T2030 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 16992 T2030 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 16993 T2030 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1363304673499
[junit4:junit4]   2> 16993 T2030 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1363304673499/solr.xml
[junit4:junit4]   2> 16994 T2030 oasc.CoreContainer.<init> New CoreContainer 1142042302
[junit4:junit4]   2> 16994 T2030 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1363304673499/'
[junit4:junit4]   2> 16995 T2030 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1363304673499/'
[junit4:junit4]   2> 17043 T2030 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 17043 T2030 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 17044 T2030 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 17044 T2030 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 17044 T2030 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 17045 T2030 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 17045 T2030 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 17046 T2030 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 17046 T2030 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 17047 T2030 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 17066 T2030 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 17084 T2030 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:53052/solr
[junit4:junit4]   2> 17084 T2030 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 17085 T2030 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:53052 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@48129807
[junit4:junit4]   2> 17086 T2030 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 17087 T2118 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:53052. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 17088 T2118 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:53052, initiating session
[junit4:junit4]   2> 17088 T2032 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:31183
[junit4:junit4]   2> 17088 T2032 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:31183
[junit4:junit4]   2> 17090 T2034 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6b4a2f4c000b with negotiated timeout 20000 for client /140.211.11.196:31183
[junit4:junit4]   2> 17090 T2118 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:53052, sessionid = 0x13d6b4a2f4c000b, negotiated timeout = 20000
[junit4:junit4]   2> 17091 T2119 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@48129807 name:ZooKeeperConnection Watcher:127.0.0.1:53052 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 17091 T2030 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 17093 T2035 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d6b4a2f4c000b
[junit4:junit4]   2> 17104 T2119 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 17104 T2032 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:31183 which had sessionid 0x13d6b4a2f4c000b
[junit4:junit4]   2> 17104 T2030 oaz.ZooKeeper.close Session: 0x13d6b4a2f4c000b closed
[junit4:junit4]   2> 17105 T2030 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 17114 T2030 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:53052/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@26a17f04
[junit4:junit4]   2> 17115 T2030 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 17115 T2120 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:53052. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 17116 T2120 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:53052, initiating session
[junit4:junit4]   2> 17116 T2032 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:31184
[junit4:junit4]   2> 17117 T2032 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:31184
[junit4:junit4]   2> 17118 T2034 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6b4a2f4c000c with negotiated timeout 20000 for client /140.211.11.196:31184
[junit4:junit4]   2> 17118 T2120 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:53052, sessionid = 0x13d6b4a2f4c000c, negotiated timeout = 20000
[junit4:junit4]   2> 17118 T2121 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@26a17f04 name:ZooKeeperConnection Watcher:127.0.0.1:53052/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 17119 T2030 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 17120 T2035 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6b4a2f4c000c type:create cxid:0x1 zxid:0xa3 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 17125 T2035 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6b4a2f4c000c type:create cxid:0x2 zxid:0xa4 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 17128 T2030 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 17548 T2053 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 17549 T2053 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:19225_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:19225"}
[junit4:junit4]   2> 17562 T2121 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 17562 T2052 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 17562 T2072 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 17562 T2059 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 17562 T2088 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 17562 T2104 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 17800 T2062 oass.SolrDispatchFilter.handleAdminRequest /admin/cores {coreNodeName=127.0.0.1:19225__collection1&state=recovering&nodeName=127.0.0.1:19225_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001 
[junit4:junit4]   2> 18131 T2030 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:31182_
[junit4:junit4]   2> 18132 T2035 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6b4a2f4c000c type:delete cxid:0xd zxid:0xa9 txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:31182_ Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:31182_
[junit4:junit4]   2> 18134 T2030 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:31182_
[junit4:junit4]   2> 18137 T2072 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 18138 T2121 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 18138 T2052 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 18138 T2072 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 18138 T2088 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 18138 T2059 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 18137 T2104 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 18138 T2088 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 18139 T2104 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 18138 T2121 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 18138 T2052 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 18139 T2059 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 18152 T2122 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1363304673499/collection1
[junit4:junit4]   2> 18153 T2122 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 18153 T2122 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 18154 T2122 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 18155 T2122 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1363304673499/collection1/'
[junit4:junit4]   2> 18156 T2122 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1363304673499/collection1/lib/README' to classloader
[junit4:junit4]   2> 18157 T2122 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1363304673499/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 18215 T2122 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 18284 T2122 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 18385 T2122 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 18392 T2122 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 19132 T2122 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 19140 T2122 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 19143 T2122 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 19153 T2122 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 19156 T2122 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 19159 T2122 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 19160 T2122 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 19161 T2122 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 19161 T2122 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 19162 T2122 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 19162 T2122 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 19162 T2122 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 19162 T2122 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1363304673499/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363304656708/jetty4/
[junit4:junit4]   2> 19162 T2122 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@561777b1
[junit4:junit4]   2> 19163 T2122 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 19163 T2122 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363304656708/jetty4 forceNew: false
[junit4:junit4]   2> 19163 T2122 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1363304656708/jetty4/index/
[junit4:junit4]   2> 19164 T2122 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1363304656708/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 19164 T2122 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363304656708/jetty4/index forceNew: false
[junit4:junit4]   2> 19166 T2122 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=BaseDirectoryWrapper(org.apache.lucene.store.RAMDirectory@6c155fc lockFactory=org.apache.lucene.store.NativeFSLockFactory@4b088353),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 19166 T2122 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 19168 T2122 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 19168 T2122 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 19168 T2122 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 19169 T2122 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 19169 T2122 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 19169 T2122 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 19170 T2122 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 19170 T2122 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 19170 T2122 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 19175 T2122 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 19179 T2122 oass.SolrIndexSearcher.<init> Opening Searcher@63efc277 main
[junit4:junit4]   2> 19179 T2122 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363304656708/jetty4/tlog
[junit4:junit4]   2> 19180 T2122 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 19180 T2122 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 19183 T2123 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@63efc277 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 19184 T2122 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 19184 T2122 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2>  C115_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:19225_, base_url=http://127.0.0.1:19225}
[junit4:junit4]   2> 19802 T2107 C115 P19225 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:53059/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4]   2> 19802 T2107 C115 P19225 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:19225 START replicas=[http://127.0.0.1:53059/collection1/] nUpdates=100
[junit4:junit4]   2> 19802 T2107 C115 P19225 oasu.PeerSync.sync WARNING no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 19803 T2107 C115 P19225 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4]   2> 19803 T2107 C115 P19225 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4]   2> 19803 T2107 C115 P19225 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4]   2> 19803 T2107 C115 P19225 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4]   2> 19804 T2107 C115 P19225 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:53059/collection1/. core=collection1
[junit4:junit4]   2> 19804 T2107 C115 P19225 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C116 name=collection1 org.apache.solr.core.SolrCore@4c40db3a url=http://127.0.0.1:53059/collection1 node=127.0.0.1:53059_ C116_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:53059_, base_url=http://127.0.0.1:53059, leader=true}
[junit4:junit4]   2> 19810 T2063 C116 P53059 REQ /get {getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0 
[junit4:junit4]   2> 19814 T2064 C116 P53059 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 19815 T2064 C116 P53059 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=BaseDirectoryWrapper(org.apache.lucene.store.RAMDirectory@1c5fdc11 lockFactory=org.apache.lucene.store.NativeFSLockFactory@780b7ed7),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 19816 T2064 C116 P53059 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 19817 T2064 C116 P53059 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=BaseDirectoryWrapper(org.apache.lucene.store.RAMDirectory@1c5fdc11 lockFactory=org.apache.lucene.store.NativeFSLockFactory@780b7ed7),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=BaseDirectoryWrapper(org.apache.lucene.store.RAMDirectory@1c5fdc11 lockFactory=org.apache.lucene.store.NativeFSLockFactory@780b7ed7),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 19817 T2064 C116 P53059 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[segments_2]
[junit4:junit4]   2> 19817 T2064 C116 P53059 oass.SolrIndexSearcher.<init> Opening Searcher@5a102144 realtime
[junit4:junit4]   2> 19818 T2064 C116 P53059 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 19818 T2064 C116 P53059 /update {waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 4
[junit4:junit4]   2> 19819 T2107 C115 P19225 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 19819 T2107 C115 P19225 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4]   2> 19821 T2065 C116 P53059 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 19821 T2065 C116 P53059 REQ /replication {command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 19822 T2107 C115 P19225 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4]   2> 19822 T2107 C115 P19225 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4]   2> 19822 T2107 C115 P19225 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4]   2> 19824 T2065 C116 P53059 REQ /replication {command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0 
[junit4:junit4]   2> 19825 T2107 C115 P19225 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4]   2> 19826 T2107 C115 P19225 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363304656708/jetty3/index.20130315104436527 forceNew: false
[junit4:junit4]   2> 19826 T2107 C115 P19225 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@32a07412 lockFactory=org.apache.lucene.store.NativeFSLockFactory@310e2306) fullCopy=false
[junit4:junit4]   2> 19829 T2065 C116 P53059 REQ /replication {file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> 19830 T2107 C115 P19225 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4]   2> 19831 T2107 C115 P19225 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4]   2> 19831 T2107 C115 P19225 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4]   2> 19832 T2107 C115 P19225 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=2
[junit4:junit4]   2> 		commit{dir=BaseDirectoryWrapper(org.apache.lucene.store.RAMDirectory@431ca8ce lockFactory=org.apache.lucene.store.NativeFSLockFactory@31417dd9),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=BaseDirectoryWrapper(org.apache.lucene.store.RAMDirectory@431ca8ce lockFactory=org.apache.lucene.store.NativeFSLockFactory@31417dd9),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 19832 T2107 C115 P19225 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[segments_2]
[junit4:junit4]   2> 19832 T2107 C115 P19225 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4]   2> 19832 T2107 C115 P19225 oass.SolrIndexSearcher.<init> Opening Searcher@10c7ac00 main
[junit4:junit4]   2> 19833 T2106 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@10c7ac00 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 19833 T2107 C115 P19225 oash.SnapPuller.fetchLatestIndex removing temporary index download directory files MockDirWrapper(org.apache.lucene.store.RAMDirectory@32a07412 lockFactory=org.apache.lucene.store.NativeFSLockFactory@310e2306)
[junit4:junit4]   2> 19833 T2107 C115 P19225 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4]   2> 19833 T2107 C115 P19225 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4]   2> 19834 T2107 C115 P19225 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 19834 T2107 C115 P19225 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 19840 T2107 C115 P19225 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4]   2> 20569 T2053 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 20570 T2053 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:31182_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:31182"}
[junit4:junit4]   2> 20570 T2053 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4]   2> 20571 T2053 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4:junit4]   2> 20599 T2053 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:19225_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:19225"}
[junit4:junit4]   2> 20609 T2052 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 20609 T2104 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 20609 T2059 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 20609 T2088 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 20609 T2121 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 20609 T2072 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 21188 T2122 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 21188 T2122 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:31182 collection:collection1 shard:shard2
[junit4:junit4]   2> 21192 T2122 oasc.ZkController.register We are http://127.0.0.1:31182/collection1/ and leader is http://127.0.0.1:43851/collection1/
[junit4:junit4]   2> 21192 T2122 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:31182
[junit4:junit4]   2> 21193 T2122 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 21193 T2122 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C117 name=collection1 org.apache.solr.core.SolrCore@58b736b url=http://127.0.0.1:31182/collection1 node=127.0.0.1:31182_ C117_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:31182_, base_url=http://127.0.0.1:31182}
[junit4:junit4]   2> 21193 T2126 C117 P31182 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 21194 T2122 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 21194 T2126 C117 P31182 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 21195 T2126 C117 P31182 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4]   2> 21195 T2126 C117 P31182 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 21195 T2030 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0
[junit4:junit4]   2> 21196 T2030 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 21196 T2126 C117 P31182 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 21197 T2030 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 21212 T2030 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 21213 T2030 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 21224 T2066 oasc.CoreContainer.create Creating SolrCore 'onenodecollectioncore' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363304660644/onenodecollectioncore
[junit4:junit4]   2> 21224 T2066 oasc.ZkController.createCollectionZkNode Check for collection zkNode:onenodecollection
[junit4:junit4]   2> 21225 T2066 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:onenodecollection
[junit4:junit4]   2> 21225 T2066 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4]   2> 21226 T2066 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
[junit4:junit4]   2> 21226 T2066 oascc.SolrZkClient.makePath makePath: /collections/onenodecollection
[junit4:junit4]   2> 21229 T2066 oasc.ZkController.readConfigName Load collection config from:/collections/onenodecollection
[junit4:junit4]   2> 21231 T2066 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363304660644/onenodecollectioncore/'
[junit4:junit4]   2> 21291 T2066 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 21347 T2066 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 21448 T2066 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 21454 T2066 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 22068 T2066 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 22080 T2066 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 22085 T2066 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 22099 T2066 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 22104 T2066 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 22109 T2066 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 22110 T2066 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 22111 T2066 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 22111 T2066 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 22112 T2066 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 22113 T2066 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 22113 T2066 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 22113 T2066 oasc.SolrCore.<init> [onenodecollectioncore] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363304660644/onenodecollectioncore/, dataDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./solrtest-BasicDistributedZk2Test-1363304656708/onenodecollection/
[junit4:junit4]   2> 22114 T2066 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@561777b1
[junit4:junit4]   2> 22114 T2066 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 22115 T2066 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./solrtest-BasicDistributedZk2Test-1363304656708/onenodecollection forceNew: false
[junit4:junit4]   2> 22116 T2066 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./solrtest-BasicDistributedZk2Test-1363304656708/onenodecollection/index/
[junit4:junit4]   2> 22116 T2066 oasc.SolrCore.initIndex WARNING [onenodecollectioncore] Solr index directory '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./solrtest-BasicDistributedZk2Test-1363304656708/onenodecollection/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 22117 T2066 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./solrtest-BasicDistributedZk2Test-1363304656708/onenodecollection/index forceNew: false
[junit4:junit4]   2> 22119 T2066 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=BaseDirectoryWrapper(org.apache.lucene.store.RAMDirectory@3c88f6b0 lockFactory=org.apache.lucene.store.NativeFSLockFactory@43ea8b62),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 22120 T2066 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 22122 T2066 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 22122 T2053 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 22122 T2066 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 22123 T2053 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":"shard2",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:31182_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:31182"}
[junit4:junit4]   2> 22123 T2066 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 22124 T2066 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 22124 T2066 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 22125 T2066 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 22125 T2066 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 22126 T2066 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 22126 T2066 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 22127 T2052 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 22127 T2059 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 22128 T2088 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 22128 T2104 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 22128 T2121 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 22127 T2072 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 22133 T2066 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 22138 T2066 oass.SolrIndexSearcher.<init> Opening Searcher@6c7cb8d2 main
[junit4:junit4]   2> 22139 T2066 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./solrtest-BasicDistributedZk2Test-1363304656708/onenodecollection/tlog
[junit4:junit4]   2> 22140 T2066 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 22140 T2066 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 22143 T2128 oasc.SolrCore.registerSearcher [onenodecollectioncore] Registered new searcher Searcher@6c7cb8d2 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 22145 T2066 oasc.ZkController.publish publishing core=onenodecollectioncore state=down
[junit4:junit4]   2> 22214 T2078 oass.SolrDispatchFilter.handleAdminRequest /admin/cores {coreNodeName=127.0.0.1:31182__collection1&state=recovering&nodeName=127.0.0.1:31182_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001 
[junit4:junit4]   2> 23632 T2053 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 23633 T2053 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":"none",
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"onenodecollectioncore",
[junit4:junit4]   2> 	  "collection":"onenodecollection",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:53059_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:53059"}
[junit4:junit4]   2> 23634 T2053 oasc.Overseer$ClusterStateUpdater.createCollection Create collection onenodecollection with numShards 1
[junit4:junit4]   2> 23634 T2053 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 23638 T2052 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 23638 T2104 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 23638 T2088 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 23638 T2059 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 23638 T2072 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 23638 T2121 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 24148 T2066 oasc.CoreContainer.registerCore registering core: onenodecollectioncore
[junit4:junit4]   2> 24149 T2066 oasc.ZkController.register Register replica - core:onenodecollectioncore address:http://127.0.0.1:53059 collection:onenodecollection shard:shard1
[junit4:junit4]   2> 24150 T2066 oascc.SolrZkClient.makePath makePath: /collections/onenodecollection/leader_elect/shard1/election
[junit4:junit4]   2> 24197 T2035 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6b4a2f4c0006 type:delete cxid:0xaa zxid:0xc5 txntype:-1 reqpath:n/a Error Path:/solr/collections/onenodecollection/leaders Error:KeeperErrorCode = NoNode for /solr/collections/onenodecollection/leaders
[junit4:junit4]   2>  C117_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:31182_, base_url=http://127.0.0.1:31182}
[junit4:junit4]   2> 24215 T2126 C117 P31182 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:43851/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4]   2> 24215 T2126 C117 P31182 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:31182 START replicas=[http://127.0.0.1:43851/collection1/] nUpdates=100
[junit4:junit4]   2> 24216 T2126 C117 P31182 oasu.PeerSync.sync WARNING no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 24216 T2126 C117 P31182 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4]   2> 24216 T2126 C117 P31182 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4]   2> 24216 T2126 C117 P31182 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4]   2> 24216 T2126 C117 P31182 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4]   2> 24217 T2126 C117 P31182 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:43851/collection1/. core=collection1
[junit4:junit4]   2> 24217 T2126 C117 P31182 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C118 name=collection1 org.apache.solr.core.SolrCore@43f14fd8 url=http://127.0.0.1:43851/collection1 node=127.0.0.1:43851_ C118_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:43851_, base_url=http://127.0.0.1:43851, leader=true}
[junit4:junit4]   2> 24223 T2079 C118 P43851 REQ /get {getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1 
[junit4:junit4]   2> 24223 T2066 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 24224 T2035 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6b4a2f4c0006 type:create cxid:0xab zxid:0xc6 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 24226 T2066 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 24226 T2066 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 24227 T2066 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:53059/onenodecollectioncore/
[junit4:junit4]   2> 24227 T2066 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 24227 T2066 oasc.SyncStrategy.syncToMe http://127.0.0.1:53059/onenodecollectioncore/ has no replicas
[junit4:junit4]   2> 24228 T2066 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:53059/onenodecollectioncore/
[junit4:junit4]   2> 24228 T2066 oascc.SolrZkClient.makePath makePath: /collections/onenodecollection/leaders/shard1
[junit4:junit4]   2> 24232 T2080 C118 P43851 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 24232 T2035 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6b4a2f4c0006 type:create cxid:0xb5 zxid:0xca txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 24234 T2080 C118 P43851 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=BaseDirectoryWrapper(org.apache.lucene.store.RAMDirectory@2074bda8 lockFactory=org.apache.lucene.store.NativeFSLockFactory@f9384fc),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 24234 T2080 C118 P43851 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 24235 T2080 C118 P43851 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=BaseDirectoryWrapper(org.apache.lucene.store.RAMDirectory@2074bda8 lockFactory=org.apache.lucene.store.NativeFSLockFactory@f9384fc),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=BaseDirectoryWrapper(org.apache.lucene.store.RAMDirectory@2074bda8 lockFactory=org.apache.lucene.store.NativeFSLockFactory@f9384fc),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 24236 T2080 C118 P43851 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[segments_2]
[junit4:junit4]   2> 24236 T2080 C118 P43851 oass.SolrIndexSearcher.<init> Opening Searcher@655304f4 realtime
[junit4:junit4]   2> 24237 T2080 C118 P43851 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 24237 T2080 C118 P43851 /update {waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 5
[junit4:junit4]   2> 24238 T2126 C117 P31182 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 24238 T2126 C117 P31182 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4]   2> 24239 T2081 C118 P43851 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 24240 T2081 C118 P43851 REQ /replication {command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 24240 T2126 C117 P31182 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4]   2> 24240 T2126 C117 P31182 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4]   2> 24241 T2126 C117 P31182 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4]   2> 24243 T2081 C118 P43851 REQ /replication {command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0 
[junit4:junit4]   2> 24244 T2126 C117 P31182 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4]   2> 24245 T2126 C117 P31182 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363304656708/jetty4/index.20130315104440946 forceNew: false
[junit4:junit4]   2> 24245 T2126 C117 P31182 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@3efb3cc9 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1e9893dc) fullCopy=false
[junit4:junit4]   2> 24248 T2081 C118 P43851 REQ /replication {file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> 24249 T2126 C117 P31182 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4]   2> 24250 T2126 C117 P31182 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4]   2> 24250 T2126 C117 P31182 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4]   2> 24251 T2126 C117 P31182 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=2
[junit4:junit4]   2> 		commit{dir=BaseDirectoryWrapper(org.apache.lucene.store.RAMDirectory@6c155fc lockFactory=org.apache.lucene.store.NativeFSLockFactory@4b088353),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=BaseDirectoryWrapper(org.apache.lucene.store.RAMDirectory@6c155fc lockFactory=org.apache.lucene.store.NativeFSLockFactory@4b088353),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 24252 T2126 C117 P31182 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[segments_2]
[junit4:junit4]   2> 24252 T2126 C117 P31182 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4]   2> 24252 T2126 C117 P31182 oass.SolrIndexSearcher.<init> Opening Searcher@7f7d44a8 main
[junit4:junit4]   2> 24253 T2123 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7f7d44a8 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 24254 T2126 C117 P31182 oash.SnapPuller.fetchLatestIndex removing temporary index download directory files MockDirWrapper(org.apache.lucene.store.RAMDirectory@3efb3cc9 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1e9893dc)
[junit4:junit4]   2> 24254 T2126 C117 P31182 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4]   2> 24254 T2126 C117 P31182 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4]   2> 24254 T2126 C117 P31182 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 24254 T2126 C117 P31182 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 24261 T2126 C117 P31182 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4]   2> 25142 T2053 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 25157 T2053 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":"shard2",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:31182_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:31182"}
[junit4:junit4]   2> 25169 T2052 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 25169 T2104 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 25169 T2072 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 25169 T2088 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 25169 T2059 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 25169 T2121 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 25179 T2066 oasc.ZkController.register We are http://127.0.0.1:53059/onenodecollectioncore/ and leader is http://127.0.0.1:53059/onenodecollectioncore/
[junit4:junit4]   2> 25179 T2066 oasc.ZkController.register No LogReplay needed for core=onenodecollectioncore baseURL=http://127.0.0.1:53059
[junit4:junit4]   2> 25179 T2066 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 25180 T2066 oasc.ZkController.publish publishing core=onenodecollectioncore state=active
[junit4:junit4]   2> 25182 T2066 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 25183 T2066 oass.SolrDispatchFilter.handleAdminRequest /admin/cores {numShards=1&roles=none&name=onenodecollectioncore&action=CREATE&collection=onenodecollection&dataDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./solrtest-BasicDistributedZk2Test-1363304656708/onenodecollection&wt=javabin&version=2} status=0 QTime=3960 
[junit4:junit4]   2> 25183 T2030 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: onenodecollection failOnTimeout:true timeout (sec):230
[junit4:junit4]   2> 25184 T2030 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 26186 T2030 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 26685 T2053 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 26686 T2053 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":"none",
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"onenodecollectioncore",
[junit4:junit4]   2> 	  "collection":"onenodecollection",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:53059_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:53059"}
[junit4:junit4]   2> 26699 T2052 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 26699 T2088 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 26699 T2072 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 26699 T2121 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 26699 T2104 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 26699 T2059 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 27189 T2030 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 27190 T2030 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: onenodecollection
[junit4:junit4]   2> 27191 T2030 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C119 name=onenodecollectioncore org.apache.solr.core.SolrCore@3735df92 url=http://127.0.0.1:53059/onenodecollectioncore node=127.0.0.1:53059_ C119_STATE=coll:onenodecollection core:onenodecollectioncore props:{shard=shard1, roles=none, state=active, core=onenodecollectioncore, collection=onenodecollection, node_name=127.0.0.1:53059_, base_url=http://127.0.0.1:53059, leader=true}
[junit4:junit4]   2> 27206 T2067 C119 P53059 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=BaseDirectoryWrapper(org.apache.lucene.store.RAMDirectory@3c88f6b0 lockFactory=org.apache.lucene.store.NativeFSLockFactory@43ea8b62),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 27206 T2067 C119 P53059 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 27208 T2067 C119 P53059 /update {wt=javabin&version=2} {add=[1 (1429528572231417856)]} 0 4
[junit4:junit4]   2> 27211 T2067 C119 P53059 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 27213 T2067 C119 P53059 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=BaseDirectoryWrapper(org.apache.lucene.store.RAMDirectory@3c88f6b0 lockFactory=org.apache.lucene.store.NativeFSLockFactory@43ea8b62),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=BaseDirectoryWrapper(org.apache.lucene.store.RAMDirectory@3c88f6b0 lockFactory=org.apache.lucene.store.NativeFSLockFactory@43ea8b62),segFN=segments_2,generation=2,filenames=[_0.fnm, _0.dvd, _0.nvm, _0.dvm, _0.pos, _0.nvd, _0.tim, segments_2, _0.fdx, _0.si, _0.tip, _0.doc, _0.fdt]
[junit4:junit4]   2> 27213 T2067 C119 P53059 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[_0.fnm, _0.dvd, _0.nvm, _0.dvm, _0.pos, _0.nvd, _0.tim, segments_2, _0.fdx, _0.si, _0.tip, _0.doc, _0.fdt]
[junit4:junit4]   2> 27214 T2067 C119 P53059 oass.SolrIndexSearcher.<init> Opening Searcher@5f4e1839 main
[junit4:junit4]   2> 27214 T2067 C119 P53059 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 27215 T2128 oasc.SolrCore.registerSearcher [onenodecollectioncore] Registered new searcher Searcher@5f4e1839 main{StandardDirectoryReader(segments_2:3 _0(5.0):C1)}
[junit4:junit4]   2> 27215 T2067 C119 P53059 /update {waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 4
[junit4:junit4]   2> 27219 T2067 C119 P53059 REQ /select {q=*:*&wt=javabin&version=2} hits=1 status=0 QTime=1 
[junit4:junit4]   2> 27219 T2030 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 27228 T2067 C119 P53059 REQ /select {q=*:*&wt=javabin&version=2} hits=1 status=0 QTime=0 
[junit4:junit4]   2> ASYNC  NEW_CORE C120 name=collection1 org.apache.solr.core.SolrCore@79ab0dc3 url=http://127.0.0.1:53055/collection1 node=127.0.0.1:53055_ C120_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:53055_, base_url=http://127.0.0.1:53055, leader=true}
[junit4:junit4]   2> 27235 T2042 C120 P53055 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=BaseDirectoryWrapper(org.apache.lucene.store.RAMDirectory@3ed8833f lockFactory=org.apache.lucene.store.NativeFSLockFactory@667e13a2),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 27235 T2042 C120 P53055 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 27238 T2042 C120 P53055 /update {wt=javabin&version=2} {add=[1 (1429528572262875136)]} 0 5
[junit4:junit4]   2>  C115_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:19225_, base_url=http://127.0.0.1:19225}
[junit4:junit4]   2> 27256 T2094 C115 P19225 /update {distrib.from=http://127.0.0.1:53059/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1429528572274409472)]} 0 2
[junit4:junit4]   2> 27257 T2068 C116 P53059 /update {wt=javabin&version=2} {add=[1 (1429528572274409472)]} 0 12
[junit4:junit4]   2> 27260 T2042 C120 P53055 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 27263 T2042 C120 P53055 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=BaseDirectoryWrapper(org.apache.lucene.store.RAMDirectory@3ed8833f lockFactory=org.apache.lucene.store.NativeFSLockFactory@667e13a2),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=BaseDirectoryWrapper(org.apache.lucene.store.RAMDirectory@3ed8833f lockFactory=org.apache.lucene.store.NativeFSLockFactory@667e13a2),segFN=segments_2,generation=2,filenames=[_0.fnm, _0.dvd, _0.nvm, _0.dvm, _0.pos, _0.nvd, _0.tim, segments_2, _0.fdx, _0.si, _0.tip, _0.doc, _0.fdt]
[junit4:junit4]   2> 27263 T2042 C120 P53055 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[_0.fnm, _0.dvd, _0.nvm, _0.dvm, _0.pos, _0.nvd, _0.tim, segments_2, _0.fdx, _0.si, _0.tip, _0.doc, _0.fdt]
[junit4:junit4]   2> 27264 T2042 C120 P53055 oass.SolrIndexSearcher.<init> Opening Searcher@4101c2fa main
[junit4:junit4]   2> 27264 T2042 C120 P53055 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 27265 T2056 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4101c2fa main{StandardDirectoryReader(segments_2:3 _0(5.0):C1)}
[junit4:junit4]   2> 27265 T2042 C120 P53055 /update {waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 5
[junit4:junit4]   2> 27266 T2084 C118 P43851 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 27267 T2084 C118 P43851 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=BaseDirectoryWrapper(org.apache.lucene.store.RAMDirectory@2074bda8 lockFactory=org.apache.lucene.store.NativeFSLockFactory@f9384fc),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 		commit{dir=BaseDirectoryWrapper(org.apache.lucene.store.RAMDirectory@2074bda8 lockFactory=org.apache.lucene.store.NativeFSLockFactory@f9384fc),segFN=segments_3,generation=3,filenames=[segments_3]
[junit4:junit4]   2> 27268 T2084 C118 P43851 oasc.SolrDeletionPolicy.updateCommits newest commit = 3[segments_3]
[junit4:junit4]   2> 27268 T2084 C118 P43851 oass.SolrIndexSearcher.<init> Opening Searcher@215dbf46 main
[junit4:junit4]   2> 27268 T2084 C118 P43851 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 27269 T2090 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@215dbf46 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 27270 T2084 C118 P43851 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:53059/collection1/, StdNode: http://127.0.0.1:19225/collection1/, StdNode: http://127.0.0.1:31182/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4]   2> 27272 T2095 C115 P19225 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 27272 T2062 C116 P53059 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 27275 T2095 C115 P19225 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=BaseDirectoryWrapper(org.apache.lucene.store.RAMDirectory@431ca8ce lockFactory=org.apache.lucene.store.NativeFSLockFactory@31417dd9),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 		commit{dir=BaseDirectoryWrapper(org.apache.lucene.store.RAMDirectory@431ca8ce lockFactory=org.apache.lucene.store.NativeFSLockFactory@31417dd9),segFN=segments_3,generation=3,filenames=[_0.fnm, _0.dvd, _0.nvm, _0.dvm, _0.pos, _0.nvd, _0.tim, _0.fdx, _0.si, _0.tip, _0.doc, segments_3, _0.fdt]
[junit4:junit4]   2> 27276 T2095 C115 P19225 oasc.SolrDeletionPolicy.updateCommits newest commit = 3[_0.fnm, _0.dvd, _0.nvm, _0.dvm, _0.pos, _0.nvd, _0.tim, _0.fdx, _0.si, _0.tip, _0.doc, segments_3, _0.fdt]
[junit4:junit4]   2> 27276 T2062 C116 P53059 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=BaseDirectoryWrapper(org.apache.lucene.store.RAMDirectory@1c5fdc11 lockFactory=org.apache.lucene.store.NativeFSLockFactory@780b7ed7),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 		commit{dir=BaseDirectoryWrapper(org.apache.lucene.store.RAMDirectory@1c5fdc11 lockFactory=org.apache.lucene.store.NativeFSLockFactory@780b7ed7),segFN=segments_3,generation=3,filenames=[_0.fnm, _0.dvd, _0.nvm, _0.dvm, _0.pos, _0.nvd, _0.tim, _0.fdx, _0.si, _0.tip, _0.doc, segments_3, _0.fdt]
[junit4:junit4]   2> 27276 T2062 C116 P53059 oasc.SolrDeletionPolicy.updateCommits newest commit = 3[_0.fnm, _0.dvd, _0.nvm, _0.dvm, _0.pos, _0.nvd, _0.tim, _0.fdx, _0.si, _0.tip, _0.doc, segments_3, _0.fdt]
[junit4:junit4]   2> 27277 T2095 C115 P19225 oass.SolrIndexSearcher.<init> Opening Searcher@3ee530a6 main
[junit4:junit4]   2>  C117_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:31182_, base_url=http://127.0.0.1:31182}
[junit4:junit4]   2> 27277 T2111 C117 P31182 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 27277 T2062 C116 P53059 oass.SolrIndexSearcher.<init> Opening Searcher@325e9cd main
[junit4:junit4]   2> 27277 T2095 C115 P19225 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 27278 T2111 C117 P31182 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=BaseDirectoryWrapper(org.apache.lucene.store.RAMDirectory@6c155fc lockFactory=org.apache.lucene.store.NativeFSLockFactory@4b088353),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 		commit{dir=BaseDirectoryWrapper(org.apache.lucene.store.RAMDirectory@6c155fc lockFactory=org.apache.lucene.store.NativeFSLockFactory@4b088353),segFN=segments_3,generation=3,filenames=[segments_3]
[junit4:junit4]   2> 27278 T2106 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3ee530a6 main{StandardDirectoryReader(segments_3:3:nrt _0(5.0):C1)}
[junit4:junit4]   2> 27278 T2062 C116 P53059 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 27278 T2074 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@325e9cd main{StandardDirectoryReader(segments_3:3:nrt _0(5.0):C1)}
[junit4:junit4]   2> 27279 T2111 C117 P31182 oasc.SolrDeletionPolicy.updateCommits newest commit = 3[segments_3]
[junit4:junit4]   2> 27279 T2095 C115 P19225 /update {waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 7
[junit4:junit4]   2> 27279 T2062 C116 P53059 /update {waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 7
[junit4:junit4]   2> 27279 T2111 C117 P31182 oass.SolrIndexSearcher.<init> Opening Searcher@169347e0 main
[junit4:junit4]   2> 27280 T2111 C117 P31182 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 27280 T2123 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@169347e0 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 27281 T2111 C117 P31182 /update {waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 4
[junit4:junit4]   2> 27281 T2084 C118 P43851 /update {waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 15
[junit4:junit4]   2> 27281 T2030 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):230
[junit4:junit4]   2> 27282 T2030 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 27283 T2030 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4]   2> 27285 T2042 C120 P53055 REQ /select {q=*:*&wt=javabin&version=2} hits=1 status=0 QTime=0 
[junit4:junit4]   2> 27286 T2030 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:53052/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@1f53ceb4
[junit4:junit4]   2> 27287 T2030 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 27287 T2136 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:53052. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 27287 T2136 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:53052, initiating session
[junit4:junit4]   2> 27287 T2032 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:31252
[junit4:junit4]   2> 27288 T2032 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:31252
[junit4:junit4]   2> 27289 T2034 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6b4a2f4c000d with negotiated timeout 10000 for client /140.211.11.196:31252
[junit4:junit4]   2> 27289 T2136 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:53052, sessionid = 0x13d6b4a2f4c000d, negotiated timeout = 10000
[junit4:junit4]   2> 27289 T2137 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1f53ceb4 name:ZooKeeperConnection Watcher:127.0.0.1:53052/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 27289 T2030 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 27290 T2030 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 27292 T2035 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d6b4a2f4c000d
[junit4:junit4]   2> 27293 T2137 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 27293 T2032 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:31252 which had sessionid 0x13d6b4a2f4c000d
[junit4:junit4]   2> 27293 T2030 oaz.ZooKeeper.close Session: 0x13d6b4a2f4c000d closed
[junit4:junit4]   2> 27294 T2064 C116 P53059 REQ /select {q=*:*&distrib=false&wt=javabin&version=2} hits=1 status=0 QTime=0 
[junit4:junit4]   2> 27296 T2078 C118 P43851 REQ /select {q=*:*&distrib=false&wt=javabin&version=2} hits=0 status=0 QTime=0 
[junit4:junit4]   2> 27297 T2096 C115 P19225 REQ /select {q=*:*&distrib=false&wt=javabin&version=2} hits=1 status=0 QTime=0 
[junit4:junit4]   2> 27301 T2140 C116 P53059 REQ /select {NOW=1363304684001&shard.url=127.0.0.1:53059/collection1/|127.0.0.1:19225/collection1/&fl=id,score&start=0&q=*:*&distrib=false&wt=javabin&isShard=true&fsv=true&version=2&rows=10} hits=1 status=0 QTime=0 
[junit4:junit4]   2> 27301 T2113 C117 P31182 REQ /select {NOW=1363304684001&shard.url=127.0.0.1:43851/collection1/|127.0.0.1:31182/collection1/&fl=id,score&start=0&q=*:*&distrib=false&wt=javabin&isShard=true&fsv=true&version=2&rows=10} hits=0 status=0 QTime=0 
[junit4:junit4]   2> 27304 T2140 C116 P53059 REQ /select {NOW=1363304684001&shard.url=127.0.0.1:53059/collection1/|127.0.0.1:19225/collection1/&ids=1&q=*:*&distrib=false&wt=javabin&isShard=true&version=2} status=0 QTime=0 
[junit4:junit4]   2> 27305 T2112 C117 P31182 REQ /select {q=*:*&wt=javabin&version=2} hits=1 status=0 QTime=6 
[junit4:junit4]   2> 27309 T2042 C120 P53055 /update {wt=javabin&version=2} {add=[2 (1429528572339421184)]} 0 2
[junit4:junit4]   2> 27318 T2111 C117 P31182 /update {distrib.from=http://127.0.0.1:43851/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[2 (1429528572344664064)]} 0 2
[junit4:junit4]   2> 27318 T2084 C118 P43851 /update {wt=javabin&version=2} {add=[2 (1429528572344664064)]} 0 6
[junit4:junit4]   2> 27322 T2042 C120 P53055 /update {wt=javabin&version=2} {add=[3 (1429528572354101248)]} 0 1
[junit4:junit4]   2> 27331 T2111 C117 P31182 /update {distrib.from=http://127.0.0.1:43851/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[3 (1429528572359344128)]} 0 1
[junit4:junit4]   2> 27331 T2080 C118 P43851 /update {distrib.from=http://127.0.0.1:53059/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[3 (1429528572359344128)]} 0 5
[junit4:junit4]   2> 27332 T2068 C116 P53059 /update {wt=javabin&version=2} {add=[3]} 0 7
[junit4:junit4]   2> 27336 T2042 C120 P53055 /update {wt=javabin&version=2} {add=[4 (1429528572368781312)]} 0 1
[junit4:junit4]   2> 27348 T2094 C115 P19225 /update {distrib.from=http://127.0.0.1:53059/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[4 (1429528572375072768)]} 0 2
[junit4:junit4]   2> 27349 T2062 C116 P53059 /update {distrib.from=http://127.0.0.1:43851/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[4 (1429528572375072768)]} 0 8
[junit4:junit4]   2> 27349 T2084 C118 P43851 /update {wt=javabin&version=2} {add=[4]} 0 10
[junit4:junit4]   2> 27353 T2042 C120 P53055 /update {wt=javabin&version=2} {add=[5 (1429528572386607104)]} 0 1
[junit4:junit4]   2> 27360 T2111 C117 P31182 /update {distrib.from=http://127.0.0.1:43851/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[5 (1429528572390801408)]} 0 1
[junit4:junit4]   2> 27361 T2084 C118 P43851 /update {wt=javabin&version=2} {add=[5 (1429528572390801408)]} 0 5
[junit4:junit4]   2> 27366 T2042 C120 P53055 /update {wt=javabin&version=2} {add=[6 (1429528572399190016)]} 0 2
[junit4:junit4]   2> 27376 T2111 C117 P31182 /update {distrib.from=http://127.0.0.1:43851/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[6 (1429528572406530048)]} 0 1
[junit4:junit4]   2> 27377 T2080 C118 P43851 /update {distrib.from=http://127.0.0.1:53059/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[6 (1429528572406530048)]} 0 6
[junit4:junit4]   2> 27377 T2068 C116 P53059 /update {wt=javabin&version=2} {add=[6]} 0 8
[junit4:junit4]   2> 27382 T2042 C120 P53055 /update {wt=javabin&version=2} {add=[7 (1429528572415967232)]} 0 2
[junit4:junit4]   2> 27394 T2111 C117 P31182 /update {distrib.from=http://127.0.0.1:43851/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[7 (1429528572424355840)]} 0 2
[junit4:junit4]   2> 27395 T2080 C118 P43851 /update {distrib.from=http://127.0.0.1:53059/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[7 (1429528572424355840)]} 0 7
[junit4:junit4]   2> 27396 T2068 C116 P53059 /update {wt=javabin&version=2} {add=[7]} 0 11
[junit4:junit4]   2> 27400 T2042 C120 P5

[...truncated too long message...]

per (1)
[junit4:junit4]   1>  DATA:
[junit4:junit4]   1>      
[junit4:junit4]   1> 
[junit4:junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=BasicDistributedZk2Test -Dtests.method=testDistribSearch -Dtests.seed=5949ACF497AB2B3F -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=bg -Dtests.timezone=Pacific/Ponape -Dtests.file.encoding=ISO-8859-1
[junit4:junit4] ERROR    139s J0 | BasicDistributedZk2Test.testDistribSearch <<<
[junit4:junit4]    > Throwable #1: org.apache.solr.common.SolrException: No registered leader was found, collection:collection1 slice:shard1
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([5949ACF497AB2B3F:D8AF22ECE0F44B03]:0)
[junit4:junit4]    > 	at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:484)
[junit4:junit4]    > 	at org.apache.solr.cloud.BasicDistributedZk2Test.brindDownShardIndexSomeDocsAndRecover(BasicDistributedZk2Test.java:291)
[junit4:junit4]    > 	at org.apache.solr.cloud.BasicDistributedZk2Test.doTest(BasicDistributedZk2Test.java:116)
[junit4:junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:806)
[junit4:junit4]    > 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit4:junit4]    > 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
[junit4:junit4]    > 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit4:junit4]    > 	at java.lang.reflect.Method.invoke(Method.java:616)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
[junit4:junit4]    > 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
[junit4:junit4]    > 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
[junit4:junit4]    > 	at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 139045 T2030 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4]   2> 139051 T2029 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 2 leaked thread(s).
[junit4:junit4]   2> 139299 T2033 oazs.SessionTrackerImpl.run SessionTrackerImpl exited loop!
[junit4:junit4]   2> 140061 T2121 oascc.ZkStateReader$3.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 140061 T2121 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 140061 T2121 oascc.ZkStateReader$2.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 140061 T2121 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> NOTE: test params are: codec=Asserting, sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=bg, timezone=Pacific/Ponape
[junit4:junit4]   2> NOTE: FreeBSD 9.0-RELEASE amd64/Sun Microsystems Inc. 1.6.0_32 (64-bit)/cpus=16,threads=1,free=333712880,total=478281728
[junit4:junit4]   2> NOTE: All tests run in this JVM: [TestLRUCache, SystemInfoHandlerTest, PreAnalyzedFieldTest, TestReversedWildcardFilterFactory, TestStressRecovery, OverseerCollectionProcessorTest, SpatialFilterTest, AutoCommitTest, DocValuesTest, TestFunctionQuery, TestDocSet, TestTrie, TestPluginEnable, AnalysisAfterCoreReloadTest, TestJoin, CSVRequestHandlerTest, TestDynamicFieldResource, ZkCLITest, TermsComponentTest, UnloadDistributedZkTest, BadComponentTest, TestFastLRUCache, TestGroupingSearch, TestPHPSerializedResponseWriter, SearchHandlerTest, DirectSolrConnectionTest, TestCoreContainer, CacheHeaderTest, TestRandomDVFaceting, CoreAdminHandlerTest, BasicDistributedZkTest, TestSolrIndexConfig, ZkControllerTest, SchemaVersionSpecificBehaviorTest, TestWriterPerf, DistributedTermsComponentTest, AliasIntegrationTest, TestSort, TestSearchPerf, TestQueryUtils, TestBadConfig, LoggingHandlerTest, TestLFUCache, CircularListTest, DateFieldTest, FileBasedSpellCheckerTest, ResourceLoaderTest, ShardRoutingTest, DistributedQueryElevationComponentTest, CachingDirectoryFactoryTest, RegexBoostProcessorTest, TestPerFieldSimilarity, TestSolrDeletionPolicy2, TestIBSimilarityFactory, UUIDFieldTest, SoftAutoCommitTest, TestDocumentBuilder, SuggesterFSTTest, ShowFileRequestHandlerTest, TestSurroundQueryParser, TestSweetSpotSimilarityFactory, UpdateRequestProcessorFactoryTest, TestCopyFieldCollectionResource, SliceStateUpdateTest, SolrIndexSplitterTest, FieldMutatingUpdateProcessorTest, TestRemoteStreaming, BasicDistributedZk2Test]
[junit4:junit4] Completed on J0 in 140.33s, 1 test, 1 error <<< FAILURES!

[...truncated 423 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/build.xml:381: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/build.xml:361: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build.xml:183: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/common-build.xml:449: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/lucene/common-build.xml:1213: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/lucene/common-build.xml:877: There were test failures: 272 suites, 1144 tests, 1 error, 27 ignored (8 assumptions)

Total time: 52 minutes 33 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-Tests-trunk-Java6 - Build # 16003 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-trunk-Java6/16003/

1 tests failed.
REGRESSION:  org.apache.solr.cloud.BasicDistributedZk2Test.testDistribSearch

Error Message:
No registered leader was found, collection:collection1 slice:shard1

Stack Trace:
org.apache.solr.common.SolrException: No registered leader was found, collection:collection1 slice:shard1
	at __randomizedtesting.SeedInfo.seed([77ED1EE1D70B422A:F60B90F9A0542216]:0)
	at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:484)
	at org.apache.solr.cloud.BasicDistributedZk2Test.brindDownShardIndexSomeDocsAndRecover(BasicDistributedZk2Test.java:291)
	at org.apache.solr.cloud.BasicDistributedZk2Test.doTest(BasicDistributedZk2Test.java:116)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:806)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:679)




Build Log:
[...truncated 8620 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.BasicDistributedZk2Test
[junit4:junit4]   2> 0 T1616 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /i/
[junit4:junit4]   2> 4 T1616 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4]   2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1363281315308
[junit4:junit4]   2> 6 T1616 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 6 T1617 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4]   2> 8 T1617 oazs.ZooKeeperServer.setTickTime tickTime set to 1000
[junit4:junit4]   2> 8 T1617 oazs.ZooKeeperServer.setMinSessionTimeout minSessionTimeout set to -1
[junit4:junit4]   2> 8 T1617 oazs.ZooKeeperServer.setMaxSessionTimeout maxSessionTimeout set to -1
[junit4:junit4]   2> 9 T1617 oazs.NIOServerCnxnFactory.configure binding to port 0.0.0.0/0.0.0.0:0
[junit4:junit4]   2> 9 T1617 oazsp.FileTxnSnapLog.save Snapshotting: 0x0 to /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-1363281315308/zookeeper/server1/data/version-2/snapshot.0
[junit4:junit4]   2> 107 T1616 oasc.ZkTestServer.run start zk server on port:41586
[junit4:junit4]   2> 107 T1616 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:41586 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@4f2029df
[junit4:junit4]   2> 108 T1616 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 108 T1622 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:41586. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 109 T1622 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:41586, initiating session
[junit4:junit4]   2> 109 T1618 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:40029
[junit4:junit4]   2> 110 T1618 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:40029
[junit4:junit4]   2> 110 T1620 oazsp.FileTxnLog.append Creating new log file: log.1
[junit4:junit4]   2> 113 T1620 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d69e605f10000 with negotiated timeout 10000 for client /140.211.11.196:40029
[junit4:junit4]   2> 113 T1622 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:41586, sessionid = 0x13d69e605f10000, negotiated timeout = 10000
[junit4:junit4]   2> 114 T1623 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4f2029df name:ZooKeeperConnection Watcher:127.0.0.1:41586 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 114 T1616 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 114 T1616 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4]   2> 132 T1621 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d69e605f10000
[junit4:junit4]   2> 133 T1623 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 133 T1616 oaz.ZooKeeper.close Session: 0x13d69e605f10000 closed
[junit4:junit4]   2> 133 T1618 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:40029 which had sessionid 0x13d69e605f10000
[junit4:junit4]   2> 133 T1616 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:41586/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@76076e71
[junit4:junit4]   2> 134 T1616 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 134 T1624 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:41586. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 135 T1624 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:41586, initiating session
[junit4:junit4]   2> 135 T1618 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:61696
[junit4:junit4]   2> 135 T1618 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:61696
[junit4:junit4]   2> 137 T1620 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d69e605f10001 with negotiated timeout 10000 for client /140.211.11.196:61696
[junit4:junit4]   2> 137 T1624 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:41586, sessionid = 0x13d69e605f10001, negotiated timeout = 10000
[junit4:junit4]   2> 137 T1625 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@76076e71 name:ZooKeeperConnection Watcher:127.0.0.1:41586/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 137 T1616 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 138 T1616 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4]   2> 147 T1616 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4]   2> 150 T1616 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4]   2> 157 T1616 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4]   2> 160 T1616 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 160 T1616 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 172 T1616 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4]   2> 173 T1616 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4]   2> 277 T1616 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4]   2> 277 T1616 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4]   2> 281 T1616 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4]   2> 281 T1616 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4]   2> 289 T1616 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4]   2> 290 T1616 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4]   2> 293 T1616 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 293 T1616 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 296 T1616 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 297 T1616 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 300 T1616 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 300 T1616 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 308 T1616 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4]   2> 308 T1616 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4]   2> 311 T1621 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d69e605f10001
[junit4:junit4]   2> 312 T1625 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 312 T1616 oaz.ZooKeeper.close Session: 0x13d69e605f10001 closed
[junit4:junit4]   2> 312 T1618 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:61696 which had sessionid 0x13d69e605f10001
[junit4:junit4]   2> 438 T1616 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 442 T1616 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:35834
[junit4:junit4]   2> 442 T1616 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 442 T1616 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 443 T1616 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1363281315616
[junit4:junit4]   2> 443 T1616 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1363281315616/solr.xml
[junit4:junit4]   2> 443 T1616 oasc.CoreContainer.<init> New CoreContainer 42445612
[junit4:junit4]   2> 444 T1616 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1363281315616/'
[junit4:junit4]   2> 444 T1616 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1363281315616/'
[junit4:junit4]   2> 476 T1616 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 477 T1616 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 477 T1616 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 477 T1616 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 477 T1616 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 478 T1616 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 478 T1616 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 478 T1616 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 479 T1616 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 479 T1616 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 493 T1616 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 505 T1616 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:41586/solr
[junit4:junit4]   2> 506 T1616 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 506 T1616 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:41586 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@2a5f0a0e
[junit4:junit4]   2> 507 T1616 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 507 T1635 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:41586. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 508 T1635 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:41586, initiating session
[junit4:junit4]   2> 508 T1618 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:62777
[junit4:junit4]   2> 508 T1618 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:62777
[junit4:junit4]   2> 510 T1620 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d69e605f10002 with negotiated timeout 20000 for client /140.211.11.196:62777
[junit4:junit4]   2> 510 T1635 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:41586, sessionid = 0x13d69e605f10002, negotiated timeout = 20000
[junit4:junit4]   2> 510 T1636 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2a5f0a0e name:ZooKeeperConnection Watcher:127.0.0.1:41586 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 511 T1616 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 512 T1621 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d69e605f10002
[junit4:junit4]   2> 513 T1636 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 513 T1618 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:62777 which had sessionid 0x13d69e605f10002
[junit4:junit4]   2> 513 T1616 oaz.ZooKeeper.close Session: 0x13d69e605f10002 closed
[junit4:junit4]   2> 513 T1616 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 521 T1616 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:41586/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@31da2502
[junit4:junit4]   2> 521 T1616 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 522 T1637 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:41586. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 522 T1637 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:41586, initiating session
[junit4:junit4]   2> 522 T1618 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:26351
[junit4:junit4]   2> 522 T1618 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:26351
[junit4:junit4]   2> 523 T1620 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d69e605f10003 with negotiated timeout 20000 for client /140.211.11.196:26351
[junit4:junit4]   2> 523 T1637 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:41586, sessionid = 0x13d69e605f10003, negotiated timeout = 20000
[junit4:junit4]   2> 524 T1638 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@31da2502 name:ZooKeeperConnection Watcher:127.0.0.1:41586/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 524 T1616 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 526 T1621 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d69e605f10003 type:create cxid:0x2 zxid:0x1a txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 535 T1616 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4]   2> 537 T1616 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35834_i
[junit4:junit4]   2> 538 T1621 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d69e605f10003 type:delete cxid:0x8 zxid:0x1c txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:35834_i Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:35834_i
[junit4:junit4]   2> 539 T1616 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:35834_i
[junit4:junit4]   2> 549 T1616 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4]   2> 660 T1621 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d69e605f10003 type:delete cxid:0x16 zxid:0x21 txntype:-1 reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4]   2> 661 T1616 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 677 T1616 oasc.Overseer.start Overseer (id=89344004280352771-127.0.0.1:35834_i-n_0000000000) starting
[junit4:junit4]   2> 677 T1621 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d69e605f10003 type:create cxid:0x1b zxid:0x23 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 679 T1621 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d69e605f10003 type:create cxid:0x1c zxid:0x24 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 720 T1621 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d69e605f10003 type:create cxid:0x1d zxid:0x25 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 721 T1621 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d69e605f10003 type:create cxid:0x1e zxid:0x26 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 722 T1640 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4]   2> 723 T1616 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4]   2> 776 T1616 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4]   2> 778 T1616 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 780 T1639 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 783 T1641 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1363281315616/collection1
[junit4:junit4]   2> 783 T1641 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4]   2> 784 T1641 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 784 T1641 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4]   2> 785 T1641 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1363281315616/collection1/'
[junit4:junit4]   2> 786 T1641 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1363281315616/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 787 T1641 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1363281315616/collection1/lib/README' to classloader
[junit4:junit4]   2> 832 T1641 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 888 T1641 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 989 T1641 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 995 T1641 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 1585 T1641 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 1596 T1641 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 1600 T1641 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 1616 T1641 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 1621 T1641 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 1625 T1641 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 1626 T1641 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 1626 T1641 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 1626 T1641 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 1627 T1641 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 1628 T1641 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 1628 T1641 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 1628 T1641 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1363281315616/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363281315308/control/data/
[junit4:junit4]   2> 1628 T1641 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2e097617
[junit4:junit4]   2> 1629 T1641 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 1629 T1641 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363281315308/control/data forceNew: false
[junit4:junit4]   2> 1630 T1641 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1363281315308/control/data/index/
[junit4:junit4]   2> 1630 T1641 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1363281315308/control/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 1631 T1641 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363281315308/control/data/index forceNew: false
[junit4:junit4]   2> 1664 T1641 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.SimpleFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/index4947649487tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@13718f92; maxCacheMB=0.599609375 maxMergeSizeMB=0.6162109375)),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 1665 T1641 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 1667 T1641 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 1667 T1641 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 1668 T1641 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 1668 T1641 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 1669 T1641 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 1669 T1641 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 1669 T1641 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 1670 T1641 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 1670 T1641 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 1677 T1641 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 1684 T1641 oass.SolrIndexSearcher.<init> Opening Searcher@2a56ba98 main
[junit4:junit4]   2> 1684 T1641 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363281315308/control/data/tlog
[junit4:junit4]   2> 1685 T1641 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 1685 T1641 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 1688 T1642 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2a56ba98 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 1689 T1641 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 1689 T1641 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 1690 T1621 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d69e605f10003 type:create cxid:0x55 zxid:0x2a txntype:-1 reqpath:n/a Error Path:/solr/overseer/queue Error:KeeperErrorCode = NoNode for /solr/overseer/queue
[junit4:junit4]   2> 2283 T1639 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2284 T1639 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"control_collection",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:35834_i",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:35834/i"}
[junit4:junit4]   2> 2284 T1639 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4]   2> 2285 T1639 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 2285 T1621 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d69e605f10003 type:create cxid:0x5d zxid:0x2d txntype:-1 reqpath:n/a Error Path:/solr/overseer/queue-work Error:KeeperErrorCode = NoNode for /solr/overseer/queue-work
[junit4:junit4]   2> 2370 T1638 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 2702 T1641 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 2702 T1641 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:35834/i collection:control_collection shard:shard1
[junit4:junit4]   2> 2703 T1641 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4]   2> 2720 T1621 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d69e605f10003 type:delete cxid:0x77 zxid:0x37 txntype:-1 reqpath:n/a Error Path:/solr/collections/control_collection/leaders Error:KeeperErrorCode = NoNode for /solr/collections/control_collection/leaders
[junit4:junit4]   2> 2722 T1641 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 2722 T1621 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d69e605f10003 type:create cxid:0x78 zxid:0x38 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 2730 T1641 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 2730 T1641 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 2730 T1641 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:35834/i/collection1/
[junit4:junit4]   2> 2730 T1641 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 2731 T1641 oasc.SyncStrategy.syncToMe http://127.0.0.1:35834/i/collection1/ has no replicas
[junit4:junit4]   2> 2731 T1641 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:35834/i/collection1/
[junit4:junit4]   2> 2731 T1641 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4]   2> 2736 T1621 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d69e605f10003 type:create cxid:0x82 zxid:0x3c txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 3876 T1639 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 3919 T1638 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 3970 T1641 oasc.ZkController.register We are http://127.0.0.1:35834/i/collection1/ and leader is http://127.0.0.1:35834/i/collection1/
[junit4:junit4]   2> 3970 T1641 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:35834/i
[junit4:junit4]   2> 3970 T1641 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 3970 T1641 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 3970 T1641 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 3972 T1641 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 3973 T1616 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 3974 T1616 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 3974 T1616 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 3987 T1616 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4]   2> 3988 T1616 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:41586/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@6ff361bc
[junit4:junit4]   2> 3989 T1616 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 3989 T1644 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:41586. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 3990 T1644 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:41586, initiating session
[junit4:junit4]   2> 3990 T1618 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:12613
[junit4:junit4]   2> 3990 T1618 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:12613
[junit4:junit4]   2> 3991 T1620 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d69e605f10004 with negotiated timeout 10000 for client /140.211.11.196:12613
[junit4:junit4]   2> 3992 T1644 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:41586, sessionid = 0x13d69e605f10004, negotiated timeout = 10000
[junit4:junit4]   2> 3992 T1645 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6ff361bc name:ZooKeeperConnection Watcher:127.0.0.1:41586/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 3992 T1616 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 3994 T1616 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 3996 T1616 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4]   2> 4180 T1616 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 4183 T1616 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:37767
[junit4:junit4]   2> 4183 T1616 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 4184 T1616 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 4184 T1616 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363281319300
[junit4:junit4]   2> 4185 T1616 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363281319300/solr.xml
[junit4:junit4]   2> 4185 T1616 oasc.CoreContainer.<init> New CoreContainer 1929133135
[junit4:junit4]   2> 4186 T1616 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363281319300/'
[junit4:junit4]   2> 4186 T1616 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363281319300/'
[junit4:junit4]   2> 4242 T1616 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 4243 T1616 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 4243 T1616 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 4244 T1616 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 4244 T1616 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 4244 T1616 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 4245 T1616 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 4245 T1616 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 4246 T1616 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 4246 T1616 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 4264 T1616 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 4282 T1616 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:41586/solr
[junit4:junit4]   2> 4283 T1616 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 4283 T1616 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:41586 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@3af6017
[junit4:junit4]   2> 4284 T1616 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 4285 T1655 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:41586. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 4285 T1655 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:41586, initiating session
[junit4:junit4]   2> 4286 T1618 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:50966
[junit4:junit4]   2> 4286 T1618 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:50966
[junit4:junit4]   2> 4288 T1620 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d69e605f10005 with negotiated timeout 20000 for client /140.211.11.196:50966
[junit4:junit4]   2> 4288 T1655 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:41586, sessionid = 0x13d69e605f10005, negotiated timeout = 20000
[junit4:junit4]   2> 4288 T1656 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3af6017 name:ZooKeeperConnection Watcher:127.0.0.1:41586 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 4288 T1616 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 4290 T1621 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d69e605f10005
[junit4:junit4]   2> 4301 T1656 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 4301 T1618 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:50966 which had sessionid 0x13d69e605f10005
[junit4:junit4]   2> 4301 T1616 oaz.ZooKeeper.close Session: 0x13d69e605f10005 closed
[junit4:junit4]   2> 4302 T1616 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 4311 T1616 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:41586/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@7b927ef9
[junit4:junit4]   2> 4312 T1616 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 4312 T1657 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:41586. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 4313 T1657 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:41586, initiating session
[junit4:junit4]   2> 4313 T1618 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:31051
[junit4:junit4]   2> 4313 T1618 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:31051
[junit4:junit4]   2> 4314 T1620 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d69e605f10006 with negotiated timeout 20000 for client /140.211.11.196:31051
[junit4:junit4]   2> 4314 T1657 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:41586, sessionid = 0x13d69e605f10006, negotiated timeout = 20000
[junit4:junit4]   2> 4315 T1658 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7b927ef9 name:ZooKeeperConnection Watcher:127.0.0.1:41586/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 4315 T1616 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 4316 T1621 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d69e605f10006 type:create cxid:0x1 zxid:0x4a txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 4318 T1621 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d69e605f10006 type:create cxid:0x2 zxid:0x4b txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 4320 T1616 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 5323 T1616 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37767_i
[junit4:junit4]   2> 5324 T1621 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d69e605f10006 type:delete cxid:0xb zxid:0x4c txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:37767_i Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:37767_i
[junit4:junit4]   2> 5352 T1616 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:37767_i
[junit4:junit4]   2> 5355 T1638 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 5356 T1645 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 5356 T1658 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 5358 T1638 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 5363 T1659 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363281319300/collection1
[junit4:junit4]   2> 5364 T1659 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 5364 T1659 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 5364 T1659 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 5366 T1659 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363281319300/collection1/'
[junit4:junit4]   2> 5367 T1659 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363281319300/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 5367 T1659 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363281319300/collection1/lib/README' to classloader
[junit4:junit4]   2> 5412 T1659 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 5432 T1639 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 5433 T1639 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"control_collection",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:35834_i",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:35834/i"}
[junit4:junit4]   2> 5438 T1645 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 5438 T1638 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 5438 T1658 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 5468 T1659 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 5569 T1659 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 5576 T1659 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 6196 T1659 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 6207 T1659 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 6211 T1659 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 6227 T1659 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 6231 T1659 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 6236 T1659 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 6237 T1659 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 6237 T1659 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 6237 T1659 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 6238 T1659 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 6238 T1659 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 6239 T1659 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 6239 T1659 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363281319300/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363281315308/jetty1/
[junit4:junit4]   2> 6239 T1659 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2e097617
[junit4:junit4]   2> 6240 T1659 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 6240 T1659 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363281315308/jetty1 forceNew: false
[junit4:junit4]   2> 6241 T1659 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1363281315308/jetty1/index/
[junit4:junit4]   2> 6241 T1659 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1363281315308/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 6242 T1659 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363281315308/jetty1/index forceNew: false
[junit4:junit4]   2> 6274 T1659 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.SimpleFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/index4947649488tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@2d955f79; maxCacheMB=0.599609375 maxMergeSizeMB=0.6162109375)),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 6274 T1659 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 6277 T1659 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 6277 T1659 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 6277 T1659 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 6278 T1659 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 6278 T1659 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 6279 T1659 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 6279 T1659 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 6279 T1659 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 6280 T1659 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 6287 T1659 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 6294 T1659 oass.SolrIndexSearcher.<init> Opening Searcher@332a3b24 main
[junit4:junit4]   2> 6294 T1659 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363281315308/jetty1/tlog
[junit4:junit4]   2> 6295 T1659 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 6296 T1659 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 6299 T1660 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@332a3b24 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 6301 T1659 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 6301 T1659 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 6951 T1639 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 6951 T1639 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:37767_i",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:37767/i"}
[junit4:junit4]   2> 6952 T1639 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 2
[junit4:junit4]   2> 6952 T1639 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 6966 T1638 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 6966 T1658 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 6966 T1645 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 7303 T1659 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 7303 T1659 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:37767/i collection:collection1 shard:shard1
[junit4:junit4]   2> 7304 T1659 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 7318 T1621 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d69e605f10006 type:delete cxid:0x48 zxid:0x5c txntype:-1 reqpath:n/a Error Path:/solr/collections/collection1/leaders Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders
[junit4:junit4]   2> 7325 T1659 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 7325 T1621 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d69e605f10006 type:create cxid:0x49 zxid:0x5d txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 7327 T1659 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 7327 T1659 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 7328 T1659 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:37767/i/collection1/
[junit4:junit4]   2> 7328 T1659 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 7328 T1659 oasc.SyncStrategy.syncToMe http://127.0.0.1:37767/i/collection1/ has no replicas
[junit4:junit4]   2> 7328 T1659 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:37767/i/collection1/
[junit4:junit4]   2> 7329 T1659 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4]   2> 7339 T1621 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d69e605f10006 type:create cxid:0x53 zxid:0x61 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 8479 T1639 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 8495 T1638 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 8495 T1645 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 8495 T1658 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 8515 T1659 oasc.ZkController.register We are http://127.0.0.1:37767/i/collection1/ and leader is http://127.0.0.1:37767/i/collection1/
[junit4:junit4]   2> 8515 T1659 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:37767/i
[junit4:junit4]   2> 8515 T1659 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 8515 T1659 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 8515 T1659 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 8517 T1659 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 8519 T1616 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 8520 T1616 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 8520 T1616 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 8716 T1616 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 8719 T1616 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:45216
[junit4:junit4]   2> 8720 T1616 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 8720 T1616 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 8721 T1616 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1363281323836
[junit4:junit4]   2> 8721 T1616 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1363281323836/solr.xml
[junit4:junit4]   2> 8721 T1616 oasc.CoreContainer.<init> New CoreContainer 1693107756
[junit4:junit4]   2> 8722 T1616 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1363281323836/'
[junit4:junit4]   2> 8723 T1616 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1363281323836/'
[junit4:junit4]   2> 8771 T1616 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 8772 T1616 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 8772 T1616 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 8773 T1616 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 8773 T1616 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 8773 T1616 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 8774 T1616 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 8774 T1616 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 8775 T1616 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 8775 T1616 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 8794 T1616 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 8812 T1616 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:41586/solr
[junit4:junit4]   2> 8812 T1616 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 8813 T1616 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:41586 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@27f8c72d
[junit4:junit4]   2> 8814 T1616 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 8814 T1671 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:41586. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 8815 T1671 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:41586, initiating session
[junit4:junit4]   2> 8815 T1618 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:19061
[junit4:junit4]   2> 8815 T1618 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:19061
[junit4:junit4]   2> 8817 T1620 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d69e605f10007 with negotiated timeout 20000 for client /140.211.11.196:19061
[junit4:junit4]   2> 8817 T1671 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:41586, sessionid = 0x13d69e605f10007, negotiated timeout = 20000
[junit4:junit4]   2> 8818 T1672 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@27f8c72d name:ZooKeeperConnection Watcher:127.0.0.1:41586 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 8818 T1616 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 8819 T1621 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d69e605f10007
[junit4:junit4]   2> 8820 T1672 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 8820 T1618 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:19061 which had sessionid 0x13d69e605f10007
[junit4:junit4]   2> 8820 T1616 oaz.ZooKeeper.close Session: 0x13d69e605f10007 closed
[junit4:junit4]   2> 8821 T1616 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 8830 T1616 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:41586/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@3c2b4560
[junit4:junit4]   2> 8831 T1616 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 8831 T1673 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:41586. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 8832 T1673 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:41586, initiating session
[junit4:junit4]   2> 8832 T1618 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:27083
[junit4:junit4]   2> 8832 T1618 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:27083
[junit4:junit4]   2> 8833 T1620 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d69e605f10008 with negotiated timeout 20000 for client /140.211.11.196:27083
[junit4:junit4]   2> 8833 T1673 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:41586, sessionid = 0x13d69e605f10008, negotiated timeout = 20000
[junit4:junit4]   2> 8834 T1674 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3c2b4560 name:ZooKeeperConnection Watcher:127.0.0.1:41586/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 8834 T1616 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 8835 T1621 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d69e605f10008 type:create cxid:0x1 zxid:0x6e txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 8836 T1621 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d69e605f10008 type:create cxid:0x2 zxid:0x6f txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 8846 T1616 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 9849 T1616 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45216_i
[junit4:junit4]   2> 9850 T1621 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d69e605f10008 type:delete cxid:0xb zxid:0x70 txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:45216_i Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:45216_i
[junit4:junit4]   2> 9854 T1616 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:45216_i
[junit4:junit4]   2> 9856 T1658 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 9856 T1638 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 9856 T1645 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 9858 T1674 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 9859 T1658 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 9859 T1638 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 9859 T1645 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 9870 T1675 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1363281323836/collection1
[junit4:junit4]   2> 9870 T1675 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 9871 T1675 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 9871 T1675 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 9873 T1675 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1363281323836/collection1/'
[junit4:junit4]   2> 9874 T1675 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1363281323836/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 9874 T1675 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1363281323836/collection1/lib/README' to classloader
[junit4:junit4]   2> 9919 T1675 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 9974 T1675 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 10009 T1639 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 10010 T1639 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:37767_i",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:37767/i"}
[junit4:junit4]   2> 10019 T1658 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 10019 T1638 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 10019 T1645 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 10075 T1675 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 10075 T1674 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 10081 T1675 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 10752 T1675 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 10764 T1675 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 10768 T1675 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 10786 T1675 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 10791 T1675 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 10796 T1675 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 10797 T1675 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 10797 T1675 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 10797 T1675 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 10799 T1675 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 10799 T1675 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 10799 T1675 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 10799 T1675 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1363281323836/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363281315308/jetty2/
[junit4:junit4]   2> 10800 T1675 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2e097617
[junit4:junit4]   2> 10800 T1675 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 10872 T1675 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363281315308/jetty2 forceNew: false
[junit4:junit4]   2> 10873 T1675 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1363281315308/jetty2/index/
[junit4:junit4]   2> 10873 T1675 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1363281315308/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 10876 T1675 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363281315308/jetty2/index forceNew: false
[junit4:junit4]   2> 10917 T1675 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.SimpleFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/index4947649489tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@5b07c310; maxCacheMB=0.599609375 maxMergeSizeMB=0.6162109375)),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 10917 T1675 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 10920 T1675 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 10920 T1675 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 10921 T1675 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 10922 T1675 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 10922 T1675 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 10922 T1675 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 10923 T1675 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 10923 T1675 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 10924 T1675 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 10931 T1675 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 10938 T1675 oass.SolrIndexSearcher.<init> Opening Searcher@e9e8d25 main
[junit4:junit4]   2> 10939 T1675 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363281315308/jetty2/tlog
[junit4:junit4]   2> 10940 T1675 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 10940 T1675 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 10944 T1676 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@e9e8d25 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 10946 T1675 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 10947 T1675 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 11525 T1639 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 11525 T1639 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:45216_i",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:45216/i"}
[junit4:junit4]   2> 11526 T1639 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4]   2> 11526 T1639 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4:junit4]   2> 11566 T1645 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 11566 T1674 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 11566 T1638 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 11566 T1658 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 11953 T1675 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 11953 T1675 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:45216/i collection:collection1 shard:shard2
[junit4:junit4]   2> 11954 T1675 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4:junit4]   2> 11968 T1621 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d69e605f10008 type:delete cxid:0x47 zxid:0x7f txntype:-1 reqpath:n/a Error Path:/solr/collections/collection1/leaders/shard2 Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders/shard2
[junit4:junit4]   2> 11969 T1675 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 11969 T1621 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d69e605f10008 type:create cxid:0x48 zxid:0x80 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 11978 T1675 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 11978 T1675 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 11978 T1675 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:45216/i/collection1/
[junit4:junit4]   2> 11978 T1675 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 11979 T1675 oasc.SyncStrategy.syncToMe http://127.0.0.1:45216/i/collection1/ has no replicas
[junit4:junit4]   2> 11979 T1675 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:45216/i/collection1/
[junit4:junit4]   2> 11979 T1675 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4]   2> 11982 T1621 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d69e605f10008 type:create cxid:0x51 zxid:0x83 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 13072 T1639 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 13125 T1645 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 13125 T1674 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 13125 T1658 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 13125 T1638 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 13166 T1675 oasc.ZkController.register We are http://127.0.0.1:45216/i/collection1/ and leader is http://127.0.0.1:45216/i/collection1/
[junit4:junit4]   2> 13166 T1675 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:45216/i
[junit4:junit4]   2> 13166 T1675 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 13166 T1675 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 13166 T1675 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 13168 T1675 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 13169 T1616 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 13170 T1616 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 13171 T1616 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 13364 T1616 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 13367 T1616 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:49442
[junit4:junit4]   2> 13368 T1616 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 13368 T1616 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 13369 T1616 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1363281328487
[junit4:junit4]   2> 13369 T1616 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1363281328487/solr.xml
[junit4:junit4]   2> 13370 T1616 oasc.CoreContainer.<init> New CoreContainer 1540230540
[junit4:junit4]   2> 13370 T1616 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1363281328487/'
[junit4:junit4]   2> 13371 T1616 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1363281328487/'
[junit4:junit4]   2> 13420 T1616 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 13421 T1616 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 13421 T1616 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 13422 T1616 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 13422 T1616 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 13422 T1616 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 13423 T1616 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 13423 T1616 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 13424 T1616 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 13424 T1616 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 13443 T1616 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 13462 T1616 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:41586/solr
[junit4:junit4]   2> 13462 T1616 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 13463 T1616 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:41586 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@3b27e142
[junit4:junit4]   2> 13464 T1616 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 13464 T1687 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:41586. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 13465 T1687 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:41586, initiating session
[junit4:junit4]   2> 13465 T1618 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:38216
[junit4:junit4]   2> 13465 T1618 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:38216
[junit4:junit4]   2> 13468 T1620 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d69e605f10009 with negotiated timeout 20000 for client /140.211.11.196:38216
[junit4:junit4]   2> 13468 T1687 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:41586, sessionid = 0x13d69e605f10009, negotiated timeout = 20000
[junit4:junit4]   2> 13468 T1688 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3b27e142 name:ZooKeeperConnection Watcher:127.0.0.1:41586 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 13469 T1616 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 13470 T1621 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d69e605f10009
[junit4:junit4]   2> 13481 T1688 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 13481 T1618 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:38216 which had sessionid 0x13d69e605f10009
[junit4:junit4]   2> 13481 T1616 oaz.ZooKeeper.close Session: 0x13d69e605f10009 closed
[junit4:junit4]   2> 13482 T1616 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 13491 T1616 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:41586/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@61c8a4cb
[junit4:junit4]   2> 13492 T1616 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 13492 T1689 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:41586. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 13493 T1689 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:41586, initiating session
[junit4:junit4]   2> 13493 T1618 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:17351
[junit4:junit4]   2> 13493 T1618 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:17351
[junit4:junit4]   2> 13495 T1620 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d69e605f1000a with negotiated timeout 20000 for client /140.211.11.196:17351
[junit4:junit4]   2> 13495 T1689 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:41586, sessionid = 0x13d69e605f1000a, negotiated timeout = 20000
[junit4:junit4]   2> 13495 T1690 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@61c8a4cb name:ZooKeeperConnection Watcher:127.0.0.1:41586/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 13495 T1616 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 13498 T1621 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d69e605f1000a type:create cxid:0x1 zxid:0x90 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 13500 T1621 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d69e605f1000a type:create cxid:0x2 zxid:0x91 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 13502 T1616 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 14505 T1616 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49442_i
[junit4:junit4]   2> 14506 T1621 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d69e605f1000a type:delete cxid:0xb zxid:0x92 txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:49442_i Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:49442_i
[junit4:junit4]   2> 14517 T1616 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49442_i
[junit4:junit4]   2> 14520 T1658 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 14520 T1638 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 14520 T1645 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 14521 T1690 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 14521 T1674 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 14521 T1674 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 14522 T1658 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 14523 T1645 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 14523 T1638 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 14529 T1691 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1363281328487/collection1
[junit4:junit4]   2> 14529 T1691 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 14530 T1691 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 14530 T1691 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 14531 T1691 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1363281328487/collection1/'
[junit4:junit4]   2> 14532 T1691 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1363281328487/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 14533 T1691 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1363281328487/collection1/lib/README' to classloader
[junit4:junit4]   2> 14579 T1691 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 14636 T1691 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 14639 T1639 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 14640 T1639 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":"shard2",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:45216_i",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:45216/i"}
[junit4:junit4]   2> 14646 T1645 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 14646 T1638 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 14646 T1674 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 14646 T1658 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 14737 T1691 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 14737 T1690 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 14743 T1691 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 15338 T1691 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 15349 T1691 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 15353 T1691 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 15369 T1691 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 15374 T1691 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 15378 T1691 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 15379 T1691 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 15379 T1691 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 15379 T1691 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 15380 T1691 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 15381 T1691 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 15381 T1691 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 15381 T1691 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1363281328487/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363281315308/jetty3/
[junit4:junit4]   2> 15381 T1691 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2e097617
[junit4:junit4]   2> 15382 T1691 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 15383 T1691 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363281315308/jetty3 forceNew: false
[junit4:junit4]   2> 15383 T1691 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1363281315308/jetty3/index/
[junit4:junit4]   2> 15383 T1691 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1363281315308/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 15385 T1691 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363281315308/jetty3/index forceNew: false
[junit4:junit4]   2> 15390 T1691 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.SimpleFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/index4947649490tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@9f96e06; maxCacheMB=0.599609375 maxMergeSizeMB=0.6162109375)),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 15390 T1691 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 15393 T1691 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 15393 T1691 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 15394 T1691 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 15394 T1691 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 15395 T1691 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 15395 T1691 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 15395 T1691 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 15396 T1691 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 15396 T1691 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 15403 T1691 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 15410 T1691 oass.SolrIndexSearcher.<init> Opening Searcher@32328a53 main
[junit4:junit4]   2> 15410 T1691 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363281315308/jetty3/tlog
[junit4:junit4]   2> 15411 T1691 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 15411 T1691 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 15419 T1692 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@32328a53 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 15420 T1691 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 15421 T1691 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 16159 T1639 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 16160 T1639 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:49442_i",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:49442/i"}
[junit4:junit4]   2> 16160 T1639 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4]   2> 16160 T1639 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 16165 T1690 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 16165 T1645 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 16165 T1674 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 16165 T1638 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 16165 T1658 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 16423 T1691 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 16423 T1691 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:49442/i collection:collection1 shard:shard1
[junit4:junit4]   2> 16426 T1691 oasc.ZkController.register We are http://127.0.0.1:49442/i/collection1/ and leader is http://127.0.0.1:37767/i/collection1/
[junit4:junit4]   2> 16426 T1691 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:49442/i
[junit4:junit4]   2> 16427 T1691 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 16427 T1691 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C290 name=collection1 org.apache.solr.core.SolrCore@2f012cfd url=http://127.0.0.1:49442/i/collection1 node=127.0.0.1:49442_i C290_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:49442_i, base_url=http://127.0.0.1:49442/i}
[junit4:junit4]   2> 16428 T1693 C290 P49442 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 16428 T1691 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 16428 T1693 C290 P49442 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 16429 T1693 C290 P49442 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4]   2> 16429 T1693 C290 P49442 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 16429 T1616 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 16430 T1616 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 16430 T1616 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 16518 T1693 C290 P49442 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 16622 T1616 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 16625 T1616 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:51048
[junit4:junit4]   2> 16626 T1616 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 16626 T1616 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 16626 T1616 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1363281331747
[junit4:junit4]   2> 16627 T1616 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1363281331747/solr.xml
[junit4:junit4]   2> 16627 T1616 oasc.CoreContainer.<init> New CoreContainer 1481093797
[junit4:junit4]   2> 16628 T1616 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1363281331747/'
[junit4:junit4]   2> 16628 T1616 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1363281331747/'
[junit4:junit4]   2> 16676 T1616 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 16677 T1616 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 16677 T1616 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 16678 T1616 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 16678 T1616 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 16679 T1616 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 16679 T1616 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 16679 T1616 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 16680 T1616 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 16680 T1616 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 16700 T1616 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 16718 T1616 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:41586/solr
[junit4:junit4]   2> 16719 T1616 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 16719 T1616 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:41586 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@b83462f
[junit4:junit4]   2> 16720 T1616 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 16721 T1704 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:41586. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 16721 T1704 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:41586, initiating session
[junit4:junit4]   2> 16722 T1618 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:60488
[junit4:junit4]   2> 16722 T1618 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:60488
[junit4:junit4]   2> 16724 T1620 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d69e605f1000b with negotiated timeout 20000 for client /140.211.11.196:60488
[junit4:junit4]   2> 16724 T1704 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:41586, sessionid = 0x13d69e605f1000b, negotiated timeout = 20000
[junit4:junit4]   2> 16724 T1705 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@b83462f name:ZooKeeperConnection Watcher:127.0.0.1:41586 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 16724 T1616 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 16726 T1621 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d69e605f1000b
[junit4:junit4]   2> 16727 T1705 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 16727 T1618 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:60488 which had sessionid 0x13d69e605f1000b
[junit4:junit4]   2> 16727 T1616 oaz.ZooKeeper.close Session: 0x13d69e605f1000b closed
[junit4:junit4]   2> 16728 T1616 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 16739 T1616 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:41586/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@5311dd0f
[junit4:junit4]   2> 16739 T1616 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 16740 T1706 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:41586. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 16740 T1706 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:41586, initiating session
[junit4:junit4]   2> 16740 T1618 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:39007
[junit4:junit4]   2> 16741 T1618 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:39007
[junit4:junit4]   2> 16742 T1620 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d69e605f1000c with negotiated timeout 20000 for client /140.211.11.196:39007
[junit4:junit4]   2> 16742 T1706 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:41586, sessionid = 0x13d69e605f1000c, negotiated timeout = 20000
[junit4:junit4]   2> 16742 T1707 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5311dd0f name:ZooKeeperConnection Watcher:127.0.0.1:41586/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 16743 T1616 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 16744 T1621 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d69e605f1000c type:create cxid:0x1 zxid:0xa3 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 16745 T1621 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d69e605f1000c type:create cxid:0x2 zxid:0xa4 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 16754 T1616 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 17684 T1639 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 17685 T1639 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:49442_i",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:49442/i"}
[junit4:junit4]   2> 17690 T1690 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 17690 T1674 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 17690 T1658 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 17690 T1638 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 17690 T1707 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 17690 T1645 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 17757 T1616 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51048_i
[junit4:junit4]   2> 17758 T1621 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d69e605f1000c type:delete cxid:0xd zxid:0xa9 txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:51048_i Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:51048_i
[junit4:junit4]   2> 17759 T1616 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:51048_i
[junit4:junit4]   2> 17761 T1658 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 17761 T1638 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 17761 T1645 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 17762 T1690 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 17762 T1707 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 17762 T1674 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 17762 T1690 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 17762 T1707 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 17763 T1674 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 17764 T1638 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 17764 T1658 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 17765 T1645 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 17770 T1708 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1363281331747/collection1
[junit4:junit4]   2> 17770 T1708 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 17771 T1708 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 17771 T1708 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 17772 T1708 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1363281331747/collection1/'
[junit4:junit4]   2> 17773 T1708 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1363281331747/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 17774 T1708 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1363281331747/collection1/lib/README' to classloader
[junit4:junit4]   2> 17819 T1708 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 17874 T1708 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 17974 T1708 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 17980 T1708 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 18531 T1648 oass.SolrDispatchFilter.handleAdminRequest /admin/cores {coreNodeName=127.0.0.1:49442_i_collection1&state=recovering&nodeName=127.0.0.1:49442_i&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=2002 
[junit4:junit4]   2> 18570 T1708 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 18581 T1708 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 18585 T1708 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 18601 T1708 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 18606 T1708 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 18610 T1708 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 18611 T1708 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 18611 T1708 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 18612 T1708 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 18613 T1708 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 18613 T1708 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 18613 T1708 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 18613 T1708 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1363281331747/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363281315308/jetty4/
[junit4:junit4]   2> 18614 T1708 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2e097617
[junit4:junit4]   2> 18614 T1708 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 18615 T1708 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363281315308/jetty4 forceNew: false
[junit4:junit4]   2> 18615 T1708 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1363281315308/jetty4/index/
[junit4:junit4]   2> 18615 T1708 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1363281315308/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 18617 T1708 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363281315308/jetty4/index forceNew: false
[junit4:junit4]   2> 18622 T1708 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.SimpleFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/index4947649491tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@3a607bbb; maxCacheMB=0.599609375 maxMergeSizeMB=0.6162109375)),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 18622 T1708 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 18625 T1708 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 18625 T1708 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 18626 T1708 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 18626 T1708 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 18627 T1708 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 18627 T1708 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 18627 T1708 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 18628 T1708 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 18628 T1708 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 18635 T1708 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 18642 T1708 oass.SolrIndexSearcher.<init> Opening Searcher@33eedda4 main
[junit4:junit4]   2> 18642 T1708 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363281315308/jetty4/tlog
[junit4:junit4]   2> 18643 T1708 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 18643 T1708 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 18647 T1709 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@33eedda4 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 18649 T1708 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 18649 T1708 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 19202 T1639 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 19202 T1639 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:51048_i",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:51048/i"}
[junit4:junit4]   2> 19203 T1639 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4]   2> 19203 T1639 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4:junit4]   2> 19208 T1690 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 19208 T1658 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 19208 T1674 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 19208 T1645 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 19208 T1638 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 19208 T1707 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 19652 T1708 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 19652 T1708 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:51048/i collection:collection1 shard:shard2
[junit4:junit4]   2> 19655 T1708 oasc.ZkController.register We are http://127.0.0.1:51048/i/collection1/ and leader is http://127.0.0.1:45216/i/collection1/
[junit4:junit4]   2> 19656 T1708 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:51048/i
[junit4:junit4]   2> 19656 T1708 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 19656 T1708 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C291 name=collection1 org.apache.solr.core.SolrCore@558a72ee url=http://127.0.0.1:51048/i/collection1 node=127.0.0.1:51048_i C291_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:51048_i, base_url=http://127.0.0.1:51048/i}
[junit4:junit4]   2> 19656 T1710 C291 P51048 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 19657 T1710 C291 P51048 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 19657 T1708 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 19658 T1710 C291 P51048 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4]   2> 19658 T1710 C291 P51048 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 19659 T1616 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 19659 T1616 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 19660 T1616 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 19660 T1710 C291 P51048 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 19680 T1616 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 19681 T1616 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 19691 T1649 oasc.CoreContainer.create Creating SolrCore 'onenodecollectioncore' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363281319300/onenodecollectioncore
[junit4:junit4]   2> 19692 T1649 oasc.ZkController.createCollectionZkNode Check for collection zkNode:onenodecollection
[junit4:junit4]   2> 19693 T1649 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:onenodecollection
[junit4:junit4]   2> 19693 T1649 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4]   2> 19694 T1649 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
[junit4:junit4]   2> 19695 T1649 oascc.SolrZkClient.makePath makePath: /collections/onenodecollection
[junit4:junit4]   2> 19697 T1649 oasc.ZkController.readConfigName Load collection config from:/collections/onenodecollection
[junit4:junit4]   2> 19699 T1649 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363281319300/onenodecollectioncore/'
[junit4:junit4]   2> 19760 T1649 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 19815 T1649 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 19816 T1649 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 19822 T1649 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 20414 T1649 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 20423 T1649 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 20426 T1649 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 20437 T1649 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 20441 T1649 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 20444 T1649 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 20445 T1649 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 20445 T1649 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 20445 T1649 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 20446 T1649 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 20446 T1649 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 20446 T1649 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 20447 T1649 oasc.SolrCore.<init> [onenodecollectioncore] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363281319300/onenodecollectioncore/, dataDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1363281315308/onenodecollection/
[junit4:junit4]   2> 20447 T1649 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2e097617
[junit4:junit4]   2> 20448 T1649 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 20448 T1649 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1363281315308/onenodecollection forceNew: false
[junit4:junit4]   2> 20448 T1649 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1363281315308/onenodecollection/index/
[junit4:junit4]   2> 20449 T1649 oasc.SolrCore.initIndex WARNING [onenodecollectioncore] Solr index directory '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1363281315308/onenodecollection/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 20450 T1649 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1363281315308/onenodecollection/index forceNew: false
[junit4:junit4]   2> 20454 T1649 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.SimpleFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/index4947649492tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@1e21cab7; maxCacheMB=0.599609375 maxMergeSizeMB=0.6162109375)),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 20454 T1649 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 20456 T1649 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 20457 T1649 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 20457 T1649 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 20457 T1649 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 20458 T1649 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 20458 T1649 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 20458 T1649 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 20459 T1649 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 20459 T1649 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 20464 T1649 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 20468 T1649 oass.SolrIndexSearcher.<init> Opening Searcher@647e3129 main
[junit4:junit4]   2> 20469 T1649 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1363281315308/onenodecollection/tlog
[junit4:junit4]   2> 20469 T1649 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 20470 T1649 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 20472 T1712 oasc.SolrCore.registerSearcher [onenodecollectioncore] Registered new searcher Searcher@647e3129 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 20472 T1649 oasc.ZkController.publish publishing core=onenodecollectioncore state=down
[junit4:junit4]   2>  C290_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:49442_i, base_url=http://127.0.0.1:49442/i}
[junit4:junit4]   2> 20533 T1693 C290 P49442 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:37767/i/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4]   2> 20533 T1693 C290 P49442 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:49442/i START replicas=[http://127.0.0.1:37767/i/collection1/] nUpdates=100
[junit4:junit4]   2> 20533 T1693 C290 P49442 oasu.PeerSync.sync WARNING no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 20534 T1693 C290 P49442 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4]   2> 20534 T1693 C290 P49442 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4]   2> 20534 T1693 C290 P49442 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4]   2> 20534 T1693 C290 P49442 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4]   2> 20534 T1693 C290 P49442 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:37767/i/collection1/. core=collection1
[junit4:junit4]   2> 20534 T1693 C290 P49442 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C292 name=collection1 org.apache.solr.core.SolrCore@4a7b9c3a url=http://127.0.0.1:37767/i/collection1 node=127.0.0.1:37767_i C292_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:37767_i, base_url=http://127.0.0.1:37767/i, leader=true}
[junit4:junit4]   2> 20538 T1650 C292 P37767 REQ /get {getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0 
[junit4:junit4]   2> 20549 T1651 C292 P37767 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 20551 T1651 C292 P37767 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.SimpleFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/index4947649488tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@2d955f79; maxCacheMB=0.599609375 maxMergeSizeMB=0.6162109375)),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 20552 T1651 C292 P37767 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 20558 T1651 C292 P37767 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.SimpleFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/index4947649488tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@2d955f79; maxCacheMB=0.599609375 maxMergeSizeMB=0.6162109375)),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.SimpleFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/index4947649488tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@2d955f79; maxCacheMB=0.599609375 maxMergeSizeMB=0.6162109375)),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 20559 T1651 C292 P37767 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[segments_2]
[junit4:junit4]   2> 20559 T1651 C292 P37767 oass.SolrIndexSearcher.<init> Opening Searcher@380cf50 realtime
[junit4:junit4]   2> 20560 T1651 C292 P37767 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 20560 T1651 C292 P37767 /update {waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 11
[junit4:junit4]   2> 20561 T1693 C290 P49442 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 20561 T1693 C290 P49442 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4]   2> 20563 T1652 C292 P37767 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 20563 T1652 C292 P37767 REQ /replication {command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 20564 T1693 C290 P49442 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4]   2> 20564 T1693 C290 P49442 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4]   2> 20564 T1693 C290 P49442 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4]   2> 20566 T1652 C292 P37767 REQ /replication {command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0 
[junit4:junit4]   2> 20566 T1693 C290 P49442 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4]   2> 20567 T1693 C290 P49442 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363281315308/jetty3/index.20130314101535870 forceNew: false
[junit4:junit4]   2> 20568 T1693 C290 P49442 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@56085162 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1381d8f4)) fullCopy=false
[junit4:junit4]   2> 20570 T1652 C292 P37767 REQ /replication {file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> 20571 T1693 C290 P49442 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4]   2> 20572 T1693 C290 P49442 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4]   2> 20573 T1693 C290 P49442 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4]   2> 20576 T1693 C290 P49442 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.SimpleFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/index4947649490tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@9f96e06; maxCacheMB=0.599609375 maxMergeSizeMB=0.6162109375)),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.SimpleFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/index4947649490tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@9f96e06; maxCacheMB=0.599609375 maxMergeSizeMB=0.6162109375)),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 20576 T1693 C290 P49442 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[segments_2]
[junit4:junit4]   2> 20577 T1693 C290 P49442 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4]   2> 20577 T1693 C290 P49442 oass.SolrIndexSearcher.<init> Opening Searcher@24d6e941 main
[junit4:junit4]   2> 20578 T1692 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@24d6e941 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 20578 T1693 C290 P49442 oash.SnapPuller.fetchLatestIndex removing temporary index download directory files MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@56085162 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1381d8f4))
[junit4:junit4]   2> 20578 T1693 C290 P49442 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4]   2> 20578 T1693 C290 P49442 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4]   2> 20579 T1693 C290 P49442 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 20579 T1693 C290 P49442 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 20592 T1693 C290 P49442 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4]   2> 20724 T1639 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 20725 T1639 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":"shard2",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:51048_i",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:51048/i"}
[junit4:junit4]   2> 20729 T1639 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":"none",
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"onenodecollectioncore",
[junit4:junit4]   2> 	  "collection":"onenodecollection",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:37767_i",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:37767/i"}
[junit4:junit4]   2> 20729 T1639 oasc.Overseer$ClusterStateUpdater.createCollection Create collection onenodecollection with numShards 1
[junit4:junit4]   2> 20729 T1639 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 20799 T1639 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:49442_i",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:49442/i"}
[junit4:junit4]   2> 20844 T1690 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 20844 T1707 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 20844 T1645 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 20844 T1674 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 20844 T1638 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 20844 T1658 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 21497 T1649 oasc.CoreContainer.registerCore registering core: onenodecollectioncore
[junit4:junit4]   2> 21497 T1649 oasc.ZkController.register Register replica - core:onenodecollectioncore address:http://127.0.0.1:37767/i collection:onenodecollection shard:shard1
[junit4:junit4]   2> 21498 T1649 oascc.SolrZkClient.makePath makePath: /collections/onenodecollection/leader_elect/shard1/election
[junit4:junit4]   2> 21520 T1621 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d69e605f10006 type:delete cxid:0xa8 zxid:0xc4 txntype:-1 reqpath:n/a Error Path:/solr/collections/onenodecollection/leaders Error:KeeperErrorCode = NoNode for /solr/collections/onenodecollection/leaders
[junit4:junit4]   2> 21521 T1649 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 21521 T1621 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d69e605f10006 type:create cxid:0xa9 zxid:0xc5 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 21524 T1649 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 21524 T1649 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 21524 T1649 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:37767/i/onenodecollectioncore/
[junit4:junit4]   2> 21525 T1649 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 21525 T1649 oasc.SyncStrategy.syncToMe http://127.0.0.1:37767/i/onenodecollectioncore/ has no replicas
[junit4:junit4]   2> 21525 T1649 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:37767/i/onenodecollectioncore/
[junit4:junit4]   2> 21526 T1649 oascc.SolrZkClient.makePath makePath: /collections/onenodecollection/leaders/shard1
[junit4:junit4]   2> 21530 T1621 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d69e605f10006 type:create cxid:0xb3 zxid:0xc9 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 21683 T1664 oass.SolrDispatchFilter.handleAdminRequest /admin/cores {coreNodeName=127.0.0.1:51048_i_collection1&state=recovering&nodeName=127.0.0.1:51048_i&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=2002 
[junit4:junit4]   2> 22359 T1639 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 22380 T1690 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 22380 T1674 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 22380 T1638 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 22380 T1658 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 22380 T1645 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 22380 T1707 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 22399 T1649 oasc.ZkController.register We are http://127.0.0.1:37767/i/onenodecollectioncore/ and leader is http://127.0.0.1:37767/i/onenodecollectioncore/
[junit4:junit4]   2> 22399 T1649 oasc.ZkController.register No LogReplay needed for core=onenodecollectioncore baseURL=http://127.0.0.1:37767/i
[junit4:junit4]   2> 22399 T1649 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 22400 T1649 oasc.ZkController.publish publishing core=onenodecollectioncore state=active
[junit4:junit4]   2> 22402 T1649 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 22403 T1649 oass.SolrDispatchFilter.handleAdminRequest /admin/cores {numShards=1&roles=none&name=onenodecollectioncore&action=CREATE&collection=onenodecollection&dataDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1363281315308/onenodecollection&wt=javabin&version=2} status=0 QTime=2711 
[junit4:junit4]   2> 22403 T1616 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: onenodecollection failOnTimeout:true timeout (sec):230
[junit4:junit4]   2> 22404 T1616 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 23406 T1616 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2>  C291_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:51048_i, base_url=http://127.0.0.1:51048/i}
[junit4:junit4]   2> 23684 T1710 C291 P51048 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:45216/i/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4]   2> 23684 T1710 C291 P51048 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:51048/i START replicas=[http://127.0.0.1:45216/i/collection1/] nUpdates=100
[junit4:junit4]   2> 23685 T1710 C291 P51048 oasu.PeerSync.sync WARNING no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 23685 T1710 C291 P51048 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4]   2> 23685 T1710 C291 P51048 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4]   2> 23685 T1710 C291 P51048 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4]   2> 23685 T1710 C291 P51048 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4]   2> 23685 T1710 C291 P51048 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:45216/i/collection1/. core=collection1
[junit4:junit4]   2> 23686 T1710 C291 P51048 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C293 name=collection1 org.apache.solr.core.SolrCore@5b7c0e2b url=http://127.0.0.1:45216/i/collection1 node=127.0.0.1:45216_i C293_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:45216_i, base_url=http://127.0.0.1:45216/i, leader=true}
[junit4:junit4]   2> 23699 T1665 C293 P45216 REQ /get {getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0 
[junit4:junit4]   2> 23705 T1666 C293 P45216 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 23709 T1666 C293 P45216 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.SimpleFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/index4947649489tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@5b07c310; maxCacheMB=0.599609375 maxMergeSizeMB=0.6162109375)),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 23710 T1666 C293 P45216 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 23748 T1666 C293 P45216 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.SimpleFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/index4947649489tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@5b07c310; maxCacheMB=0.599609375 maxMergeSizeMB=0.6162109375)),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.SimpleFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/index4947649489tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@5b07c310; maxCacheMB=0.599609375 maxMergeSizeMB=0.6162109375)),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 23748 T1666 C293 P45216 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[segments_2]
[junit4:junit4]   2> 23749 T1666 C293 P45216 oass.SolrIndexSearcher.<init> Opening Searcher@722fc6da realtime
[junit4:junit4]   2> 23749 T1666 C293 P45216 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 23750 T1666 C293 P45216 /update {waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 45
[junit4:junit4]   2> 23750 T1710 C291 P51048 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 23751 T1710 C291 P51048 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4]   2> 23752 T1667 C293 P45216 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 23753 T1667 C293 P45216 REQ /replication {command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 23753 T1710 C291 P51048 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4]   2> 23753 T1710 C291 P51048 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4]   2> 23753 T1710 C291 P51048 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4]   2> 23756 T1667 C293 P45216 REQ /replication {command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0 
[junit4:junit4]   2> 23757 T1710 C291 P51048 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4]   2> 23758 T1710 C291 P51048 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363281315308/jetty4/index.20130314101539060 forceNew: false
[junit4:junit4]   2> 23758 T1710 C291 P51048 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@63eb624a lockFactory=org.apache.lucene.store.NativeFSLockFactory@3bdcf37e)) fullCopy=false
[junit4:junit4]   2> 23761 T1667 C293 P45216 REQ /replication {file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> 23762 T1710 C291 P51048 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4]   2> 23763 T1710 C291 P51048 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4]   2> 23764 T1710 C291 P51048 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4]   2> 23767 T1710 C291 P51048 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.SimpleFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/index4947649491tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@3a607bbb; maxCacheMB=0.599609375 maxMergeSizeMB=0.6162109375)),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.SimpleFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/index4947649491tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@3a607bbb; maxCacheMB=0.599609375 maxMergeSizeMB=0.6162109375)),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 23767 T1710 C291 P51048 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[segments_2]
[junit4:junit4]   2> 23767 T1710 C291 P51048 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4]   2> 23768 T1710 C291 P51048 oass.SolrIndexSearcher.<init> Opening Searcher@187885c2 main
[junit4:junit4]   2> 23768 T1709 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@187885c2 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 23769 T1710 C291 P51048 oash.SnapPuller.fetchLatestIndex removing temporary index download directory files MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@63eb624a lockFactory=org.apache.lucene.store.NativeFSLockFactory@3bdcf37e))
[junit4:junit4]   2> 23769 T1710 C291 P51048 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4]   2> 23769 T1710 C291 P51048 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4]   2> 23769 T1710 C291 P51048 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 23770 T1710 C291 P51048 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 23771 T1710 C291 P51048 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4]   2> 23892 T1639 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 23893 T1639 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":"none",
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"onenodecollectioncore",
[junit4:junit4]   2> 	  "collection":"onenodecollection",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:37767_i",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:37767/i"}
[junit4:junit4]   2> 23922 T1639 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":"shard2",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:51048_i",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:51048/i"}
[junit4:junit4]   2> 23933 T1690 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 23933 T1658 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 23933 T1638 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 23933 T1645 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 23933 T1674 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 23933 T1707 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 24408 T1616 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 24409 T1616 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: onenodecollection
[junit4:junit4]   2> 24410 T1616 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C294 name=onenodecollectioncore org.apache.solr.core.SolrCore@7433866e url=http://127.0.0.1:37767/i/onenodecollectioncore node=127.0.0.1:37767_i C294_STATE=coll:onenodecollection core:onenodecollectioncore props:{shard=shard1, roles=none, state=active, core=onenodecollectioncore, collection=onenodecollection, node_name=127.0.0.1:37767_i, base_url=http://127.0.0.1:37767/i, leader=true}
[junit4:junit4]   2> 24438 T1653 C294 P37767 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.SimpleFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/index4947649492tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@1e21cab7; maxCacheMB=0.599609375 maxMergeSizeMB=0.6162109375)),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 24439 T1653 C294 P37767 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 24441 T1653 C294 P37767 /update {wt=javabin&version=2} {add=[1 (1429504094097113088)]} 0 7
[junit4:junit4]   2> 24448 T1653 C294 P37767 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 24478 T1653 C294 P37767 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.SimpleFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/index4947649492tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@1e21cab7; maxCacheMB=0.599609375 maxMergeSizeMB=0.6162109375)),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.SimpleFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/index4947649492tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@1e21cab7; maxCacheMB=0.599609375 maxMergeSizeMB=0.6162109375)),segFN=segments_2,generation=2,filenames=[_0.cbnd, _0.fnm, _0.dvdm, _0.cbnm, _0.pos, _0.tim, segments_2, _0.fdx, _0.si, _0.tip, _0.doc, _0.dvdd, _0.fdt]
[junit4:junit4]   2> 24478 T1653 C294 P37767 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[_0.cbnd, _0.fnm, _0.dvdm, _0.cbnm, _0.pos, _0.tim, segments_2, _0.fdx, _0.si, _0.tip, _0.doc, _0.dvdd, _0.fdt]
[junit4:junit4]   2> 24487 T1653 C294 P37767 oass.SolrIndexSearcher.<init> Opening Searcher@347d95b main
[junit4:junit4]   2> 24487 T1653 C294 P37767 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 24488 T1712 oasc.SolrCore.registerSearcher [onenodecollectioncore] Registered new searcher Searcher@347d95b main{StandardDirectoryReader(segments_2:3 _0(5.0):C1)}
[junit4:junit4]   2> 24489 T1653 C294 P37767 /update {waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 41
[junit4:junit4]   2> 24492 T1653 C294 P37767 REQ /select {q=*:*&wt=javabin&version=2} hits=1 status=0 QTime=1 
[junit4:junit4]   2> 24493 T1616 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 24508 T1653 C294 P37767 REQ /select {q=*:*&wt=javabin&version=2} hits=1 status=0 QTime=0 
[junit4:junit4]   2> ASYNC  NEW_CORE C295 name=collection1 org.apache.solr.core.SolrCore@2d7f7271 url=http://127.0.0.1:35834/i/collection1 node=127.0.0.1:35834_i C295_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:35834_i, base_url=http://127.0.0.1:35834/i, leader=true}
[junit4:junit4]   2> 24522 T1628 C295 P35834 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.SimpleFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/index4947649487tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@13718f92; maxCacheMB=0.599609375 maxMergeSizeMB=0.6162109375)),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 24522 T1628 C295 P35834 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 24526 T1628 C295 P35834 /update {wt=javabin&version=2} {add=[1 (1429504094185193472)]} 0 8
[junit4:junit4]   2>  C290_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:49442_i, base_url=http://127.0.0.1:49442/i}
[junit4:junit4]   2> 24551 T1680 C290 P49442 /update {distrib.from=http://127.0.0.1:37767/i/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1429504094203019264)]} 0 3
[junit4:junit4]   2> 24552 T1654 C292 P37767 /update {wt=javabin&version=2} {add=[1 (1429504094203019264)]} 0 17
[junit4:junit4]   2> 24555 T1628 C295 P35834 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 24588 T1628 C295 P35834 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.SimpleFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/index4947649487tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@13718f92; maxCacheMB=0.599609375 maxMergeSizeMB=0.6162109375)),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.SimpleFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/index4947649487tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@13718f92; maxCacheMB=0.599609375 maxMergeSizeMB=0.6162109375)),segFN=segments_2,generation=2,filenames=[_0.cbnd, _0.fnm, _0.dvdm, _0.cbnm, _0.pos, _0.tim, segments_2, _0.fdx, _0.si, _0.tip, _0.doc, _0.dvdd, _0.fdt]
[junit4:junit4]   2> 24589 T1628 C295 P35834 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[_0.cbnd, _0.fnm, _0.dvdm, _0.cbnm, _0.pos, _0.tim, segments_2, _0.fdx, _0.si, _0.tip, _0.doc, _0.dvdd, _0.fdt]
[junit4:junit4]   2> 24598 T1628 C295 P35834 oass.SolrIndexSearcher.<init> Opening Searcher@6eb9d8c main
[junit4:junit4]   2> 24599 T1628 C295 P35834 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 24600 T1642 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6eb9d8c main{StandardDirectoryReader(segments_2:3 _0(5.0):C1)}
[junit4:junit4]   2> 24600 T1628 C295 P35834 /update {waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 45
[junit4:junit4]   2> 24602 T1670 C293 P45216 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 24605 T1670 C293 P45216 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.SimpleFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/index4947649489tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@5b07c310; maxCacheMB=0.599609375 maxMergeSizeMB=0.6162109375)),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.SimpleFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/index4947649489tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@5b07c310; maxCacheMB=0.599609375 maxMergeSizeMB=0.6162109375)),segFN=segments_3,generation=3,filenames=[segments_3]
[junit4:junit4]   2> 24606 T1670 C293 P45216 oasc.SolrDeletionPolicy.updateCommits newest commit = 3[segments_3]
[junit4:junit4]   2> 24606 T1670 C293 P45216 oass.SolrIndexSearcher.<init> Opening Searcher@f0836b6 main
[junit4:junit4]   2> 24607 T1670 C293 P45216 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 24607 T1676 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@f0836b6 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 24608 T1670 C293 P45216 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:37767/i/collection1/, StdNode: http://127.0.0.1:49442/i/collection1/, StdNode: http://127.0.0.1:51048/i/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4]   2> 24611 T1648 C292 P37767 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 24611 T1681 C290 P49442 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2>  C291_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:51048_i, base_url=http://127.0.0.1:51048/i}
[junit4:junit4]   2> 24639 T1697 C291 P51048 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 24642 T1648 C292 P37767 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.SimpleFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/index4947649488tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@2d955f79; maxCacheMB=0.599609375 maxMergeSizeMB=0.6162109375)),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.SimpleFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/index4947649488tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@2d955f79; maxCacheMB=0.599609375 maxMergeSizeMB=0.6162109375)),segFN=segments_3,generation=3,filenames=[_0.cbnd, _0.fnm, _0.dvdm, _0.cbnm, _0.pos, _0.tim, _0.fdx, _0.si, _0.tip, _0.doc, segments_3, _0.dvdd, _0.fdt]
[junit4:junit4]   2> 24643 T1648 C292 P37767 oasc.SolrDeletionPolicy.updateCommits newest commit = 3[_0.cbnd, _0.fnm, _0.dvdm, _0.cbnm, _0.pos, _0.tim, _0.fdx, _0.si, _0.tip, _0.doc, segments_3, _0.dvdd, _0.fdt]
[junit4:junit4]   2> 24648 T1648 C292 P37767 oass.SolrIndexSearcher.<init> Opening Searcher@b057679 main
[junit4:junit4]   2> 24649 T1648 C292 P37767 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 24649 T1660 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@b057679 main{StandardDirectoryReader(segments_3:3:nrt _0(5.0):C1)}
[junit4:junit4]   2> 24650 T1648 C292 P37767 /update {waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 39
[junit4:junit4]   2> 24654 T1697 C291 P51048 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.SimpleFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/index4947649491tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@3a607bbb; maxCacheMB=0.599609375 maxMergeSizeMB=0.6162109375)),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.SimpleFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/index4947649491tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@3a607bbb; maxCacheMB=0.599609375 maxMergeSizeMB=0.6162109375)),segFN=segments_3,generation=3,filenames=[segments_3]
[junit4:junit4]   2> 24654 T1697 C291 P51048 oasc.SolrDeletionPolicy.updateCommits newest commit = 3[segments_3]
[junit4:junit4]   2> 24655 T1697 C291 P51048 oass.SolrIndexSearcher.<init> Opening Searcher@379512ed main
[junit4:junit4]   2> 24656 T1697 C291 P51048 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 24656 T1709 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@379512ed main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 24657 T1697 C291 P51048 /update {waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 18
[junit4:junit4]   2> 24667 T1681 C290 P49442 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.SimpleFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/index4947649490tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@9f96e06; maxCacheMB=0.599609375 maxMergeSizeMB=0.6162109375)),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.SimpleFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/index4947649490tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@9f96e06; maxCacheMB=0.599609375 maxMergeSizeMB=0.6162109375)),segFN=segments_3,generation=3,filenames=[_0.cbnd, _0.fnm, _0.dvdm, _0.cbnm, _0.pos, _0.tim, _0.fdx, _0.si, _0.tip, _0.doc, segments_3, _0.dvdd, _0.fdt]
[junit4:junit4]   2> 24667 T1681 C290 P49442 oasc.SolrDeletionPolicy.updateCommits newest commit = 3[_0.cbnd, _0.fnm, _0.dvdm, _0.cbnm, _0.pos, _0.tim, _0.fdx, _0.si, _0.tip, _0.doc, segments_3, _0.dvdd, _0.fdt]
[junit4:junit4]   2> 24673 T1681 C290 P49442 oass.SolrIndexSearcher.<init> Opening Searcher@46ec355c main
[junit4:junit4]   2> 24673 T1681 C290 P49442 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 24674 T1692 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@46ec355c main{StandardDirectoryReader(segments_3:3:nrt _0(5.0):C1)}
[junit4:junit4]   2> 24674 T1681 C290 P49442 /update {waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 63
[junit4:junit4]   2> 24675 T1670 C293 P45216 /update {waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 73
[junit4:junit4]   2> 24675 T1616 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):230
[junit4:junit4]   2> 24676 T1616 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 24677 T1616 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4]   2> 24680 T1628 C295 P35834 REQ /select {q=*:*&wt=javabin&version=2} hits=1 status=0 QTime=0 
[junit4:junit4]   2> 24681 T1616 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:41586/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@3da34f64
[junit4:junit4]   2> 24682 T1616 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 24682 T1722 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:41586. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 24683 T1722 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:41586, initiating session
[junit4:junit4]   2> 24683 T1618 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:32152
[junit4:junit4]   2> 24684 T1618 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:32152
[junit4:junit4]   2> 24685 T1620 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d69e605f1000d with negotiated timeout 10000 for client /140.211.11.196:32152
[junit4:junit4]   2> 24685 T1722 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:41586, sessionid = 0x13d69e605f1000d, negotiated timeout = 10000
[junit4:junit4]   2> 24685 T1723 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3da34f64 name:ZooKeeperConnection Watcher:127.0.0.1:41586/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 24685 T1616 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 24687 T1616 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 24689 T1621 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d69e605f1000d
[junit4:junit4]   2> 24690 T1723 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 24690 T1616 oaz.ZooKeeper.close Session: 0x13d69e605f1000d closed
[junit4:junit4]   2> 24690 T1618 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:32152 which had sessioni

[...truncated too long message...]

nit4]   1>    DATA:
[junit4:junit4]   1>        {"id":"89344004280352771-127.0.0.1:35834_i-n_0000000000"}
[junit4:junit4]   1>  /zookeeper (1)
[junit4:junit4]   1>  DATA:
[junit4:junit4]   1>      
[junit4:junit4]   1> 
[junit4:junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=BasicDistributedZk2Test -Dtests.method=testDistribSearch -Dtests.seed=77ED1EE1D70B422A -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=mt_MT -Dtests.timezone=PNT -Dtests.file.encoding=US-ASCII
[junit4:junit4] ERROR    137s J1 | BasicDistributedZk2Test.testDistribSearch <<<
[junit4:junit4]    > Throwable #1: org.apache.solr.common.SolrException: No registered leader was found, collection:collection1 slice:shard1
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([77ED1EE1D70B422A:F60B90F9A0542216]:0)
[junit4:junit4]    > 	at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:484)
[junit4:junit4]    > 	at org.apache.solr.cloud.BasicDistributedZk2Test.brindDownShardIndexSomeDocsAndRecover(BasicDistributedZk2Test.java:291)
[junit4:junit4]    > 	at org.apache.solr.cloud.BasicDistributedZk2Test.doTest(BasicDistributedZk2Test.java:116)
[junit4:junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:806)
[junit4:junit4]    > 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit4:junit4]    > 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
[junit4:junit4]    > 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit4:junit4]    > 	at java.lang.reflect.Method.invoke(Method.java:616)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
[junit4:junit4]    > 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
[junit4:junit4]    > 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
[junit4:junit4]    > 	at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 137484 T1616 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4]   2> 137491 T1615 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
[junit4:junit4]   2> 137515 T1707 oascc.ZkStateReader$3.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 137515 T1707 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 137515 T1707 oascc.ZkStateReader$2.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 137515 T1707 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> NOTE: test params are: codec=CheapBastard, sim=RandomSimilarityProvider(queryNorm=false,coord=crazy): {}, locale=mt_MT, timezone=PNT
[junit4:junit4]   2> NOTE: FreeBSD 9.0-RELEASE amd64/Sun Microsystems Inc. 1.6.0_32 (64-bit)/cpus=16,threads=1,free=188825104,total=436273152
[junit4:junit4]   2> NOTE: All tests run in this JVM: [TestComponentsName, TestNumberUtils, TestFastOutputStream, ZkNodePropsTest, TestIndexingPerformance, TestLuceneMatchVersion, TestSolrDiscoveryProperties, TestDistributedSearch, CopyFieldTest, TestPropInject, DOMUtilTest, URLClassifyProcessorTest, CurrencyFieldXmlFileTest, SpellPossibilityIteratorTest, TestCodecSupport, TestSuggestSpellingConverter, SortByFunctionTest, BasicZkTest, TestFieldResource, TestWordDelimiterFilterFactory, TestPostingsSolrHighlighter, TestFiltering, DirectUpdateHandlerTest, StatsComponentTest, TestBinaryResponseWriter, SolrCoreCheckLockOnStartupTest, SuggesterTSTTest, SolrCmdDistributorTest, TestFieldTypeResource, SpellCheckCollatorTest, TestRecovery, OpenExchangeRatesOrgProviderTest, TestStressLucene, TestArbitraryIndexDir, HighlighterTest, NoCacheHeaderTest, SampleTest, SpellCheckComponentTest, TestElisionMultitermQuery, CoreContainerCoreInitFailuresTest, StandardRequestHandlerTest, ChaosMonkeySafeLeaderTest, DateMathParserTest, TestIndexSearcher, SignatureUpdateProcessorFactoryTest, PingRequestHandlerTest, TestShardHandlerFactory, TestOmitPositions, HighlighterConfigTest, TestJmxIntegration, OutputWriterTest, DocumentAnalysisRequestHandlerTest, OpenCloseCoreStressTest, TestAtomicUpdateErrorCases, TestQueryTypes, MultiTermTest, EchoParamsTest, NotRequiredUniqueKeyTest, XmlUpdateRequestHandlerTest, TestRTGBase, TestUpdate, IndexSchemaRuntimeFieldTest, TestStressReorder, ClusterStateTest, BasicDistributedZk2Test]
[junit4:junit4] Completed on J1 in 137.76s, 1 test, 1 error <<< FAILURES!

[...truncated 421 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/build.xml:381: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/build.xml:361: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build.xml:183: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/common-build.xml:449: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/lucene/common-build.xml:1213: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/lucene/common-build.xml:877: There were test failures: 272 suites, 1144 tests, 1 error, 27 ignored (8 assumptions)

Total time: 55 minutes 47 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-Tests-trunk-Java6 - Build # 16002 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-trunk-Java6/16002/

1 tests failed.
REGRESSION:  org.apache.solr.cloud.SyncSliceTest.testDistribSearch

Error Message:
Test Setup Failure: shard1 should have just been set up to be inconsistent - but it's still consistent. Leader:http://127.0.0.1:32313/collection1skip list:[CloudJettyRunner [url=http://127.0.0.1:13969/collection1], CloudJettyRunner [url=http://127.0.0.1:41673/collection1]]

Stack Trace:
java.lang.AssertionError: Test Setup Failure: shard1 should have just been set up to be inconsistent - but it's still consistent. Leader:http://127.0.0.1:32313/collection1skip list:[CloudJettyRunner [url=http://127.0.0.1:13969/collection1], CloudJettyRunner [url=http://127.0.0.1:41673/collection1]]
	at __randomizedtesting.SeedInfo.seed([D2139F8170B5DA86:53F5119907EABABA]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNotNull(Assert.java:526)
	at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:212)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:806)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:679)




Build Log:
[...truncated 8377 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.SyncSliceTest
[junit4:junit4]   2> 0 T846 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4:junit4]   2> 5 T846 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4]   2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./solrtest-SyncSliceTest-1363253958515
[junit4:junit4]   2> 7 T846 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 7 T847 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4]   2> 10 T847 oazs.ZooKeeperServer.setTickTime tickTime set to 1000
[junit4:junit4]   2> 10 T847 oazs.ZooKeeperServer.setMinSessionTimeout minSessionTimeout set to -1
[junit4:junit4]   2> 10 T847 oazs.ZooKeeperServer.setMaxSessionTimeout maxSessionTimeout set to -1
[junit4:junit4]   2> 11 T847 oazs.NIOServerCnxnFactory.configure binding to port 0.0.0.0/0.0.0.0:0
[junit4:junit4]   2> 12 T847 oazsp.FileTxnSnapLog.save Snapshotting: 0x0 to /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-1363253958515/zookeeper/server1/data/version-2/snapshot.0
[junit4:junit4]   2> 108 T846 oasc.ZkTestServer.run start zk server on port:65403
[junit4:junit4]   2> 108 T846 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:65403 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@69683ca4
[junit4:junit4]   2> 109 T846 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 110 T852 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:65403. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 110 T852 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:65403, initiating session
[junit4:junit4]   2> 111 T848 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:65404
[junit4:junit4]   2> 111 T848 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:65404
[junit4:junit4]   2> 111 T850 oazsp.FileTxnLog.append Creating new log file: log.1
[junit4:junit4]   2> 114 T850 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6844977a0000 with negotiated timeout 10000 for client /140.211.11.196:65404
[junit4:junit4]   2> 114 T852 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:65403, sessionid = 0x13d6844977a0000, negotiated timeout = 10000
[junit4:junit4]   2> 115 T853 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@69683ca4 name:ZooKeeperConnection Watcher:127.0.0.1:65403 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 115 T846 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 116 T846 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4]   2> 119 T851 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d6844977a0000
[junit4:junit4]   2> 120 T853 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 121 T848 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:65404 which had sessionid 0x13d6844977a0000
[junit4:junit4]   2> 120 T846 oaz.ZooKeeper.close Session: 0x13d6844977a0000 closed
[junit4:junit4]   2> 122 T846 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:65403/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@585bd6a6
[junit4:junit4]   2> 123 T846 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 123 T854 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:65403. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 124 T854 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:65403, initiating session
[junit4:junit4]   2> 124 T848 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:65405
[junit4:junit4]   2> 124 T848 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:65405
[junit4:junit4]   2> 126 T850 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6844977a0001 with negotiated timeout 10000 for client /140.211.11.196:65405
[junit4:junit4]   2> 126 T854 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:65403, sessionid = 0x13d6844977a0001, negotiated timeout = 10000
[junit4:junit4]   2> 126 T855 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@585bd6a6 name:ZooKeeperConnection Watcher:127.0.0.1:65403/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 126 T846 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 127 T846 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4]   2> 138 T846 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4]   2> 142 T846 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4]   2> 144 T846 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4]   2> 152 T846 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 153 T846 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 168 T846 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4]   2> 170 T846 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4]   2> 273 T846 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4]   2> 274 T846 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4]   2> 277 T846 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4]   2> 278 T846 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4]   2> 290 T846 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4]   2> 291 T846 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4]   2> 294 T846 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 294 T846 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 297 T846 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 298 T846 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 301 T846 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 301 T846 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 304 T846 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4]   2> 305 T846 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4]   2> 312 T851 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d6844977a0001
[junit4:junit4]   2> 313 T855 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 313 T848 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:65405 which had sessionid 0x13d6844977a0001
[junit4:junit4]   2> 313 T846 oaz.ZooKeeper.close Session: 0x13d6844977a0001 closed
[junit4:junit4]   2> 502 T846 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 520 T846 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:60654
[junit4:junit4]   2> 521 T846 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 521 T846 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 522 T846 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1363253958823
[junit4:junit4]   2> 522 T846 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1363253958823/solr.xml
[junit4:junit4]   2> 523 T846 oasc.CoreContainer.<init> New CoreContainer 1239450605
[junit4:junit4]   2> 523 T846 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1363253958823/'
[junit4:junit4]   2> 524 T846 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1363253958823/'
[junit4:junit4]   2> 574 T846 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 575 T846 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 576 T846 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 576 T846 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 577 T846 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 577 T846 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 577 T846 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 578 T846 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 578 T846 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 579 T846 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 601 T846 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 622 T846 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:65403/solr
[junit4:junit4]   2> 622 T846 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 623 T846 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:65403 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@2e78c714
[junit4:junit4]   2> 624 T846 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 624 T865 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:65403. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 625 T865 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:65403, initiating session
[junit4:junit4]   2> 625 T848 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:24448
[junit4:junit4]   2> 626 T848 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:24448
[junit4:junit4]   2> 631 T850 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6844977a0002 with negotiated timeout 20000 for client /140.211.11.196:24448
[junit4:junit4]   2> 631 T865 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:65403, sessionid = 0x13d6844977a0002, negotiated timeout = 20000
[junit4:junit4]   2> 631 T866 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2e78c714 name:ZooKeeperConnection Watcher:127.0.0.1:65403 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 632 T846 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 633 T851 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d6844977a0002
[junit4:junit4]   2> 643 T866 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 643 T848 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:24448 which had sessionid 0x13d6844977a0002
[junit4:junit4]   2> 643 T846 oaz.ZooKeeper.close Session: 0x13d6844977a0002 closed
[junit4:junit4]   2> 644 T846 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 656 T846 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:65403/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@25b63670
[junit4:junit4]   2> 657 T846 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 658 T867 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:65403. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 658 T867 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:65403, initiating session
[junit4:junit4]   2> 658 T848 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:20058
[junit4:junit4]   2> 659 T848 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:20058
[junit4:junit4]   2> 660 T850 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6844977a0003 with negotiated timeout 20000 for client /140.211.11.196:20058
[junit4:junit4]   2> 660 T867 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:65403, sessionid = 0x13d6844977a0003, negotiated timeout = 20000
[junit4:junit4]   2> 661 T868 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@25b63670 name:ZooKeeperConnection Watcher:127.0.0.1:65403/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 661 T846 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 667 T851 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6844977a0003 type:create cxid:0x2 zxid:0x1a txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 669 T846 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4]   2> 671 T846 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60654_
[junit4:junit4]   2> 672 T851 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6844977a0003 type:delete cxid:0x8 zxid:0x1c txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:60654_ Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:60654_
[junit4:junit4]   2> 678 T846 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:60654_
[junit4:junit4]   2> 681 T846 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4]   2> 692 T851 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6844977a0003 type:delete cxid:0x16 zxid:0x21 txntype:-1 reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4]   2> 698 T846 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 700 T846 oasc.Overseer.start Overseer (id=89342211425697795-127.0.0.1:60654_-n_0000000000) starting
[junit4:junit4]   2> 701 T851 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6844977a0003 type:create cxid:0x1b zxid:0x23 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 703 T851 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6844977a0003 type:create cxid:0x1c zxid:0x24 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 711 T851 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6844977a0003 type:create cxid:0x1d zxid:0x25 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 718 T851 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6844977a0003 type:create cxid:0x1e zxid:0x26 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 783 T870 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4]   2> 784 T846 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4]   2> 810 T846 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4]   2> 812 T846 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 813 T869 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 816 T871 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1363253958823/collection1
[junit4:junit4]   2> 816 T871 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4]   2> 817 T871 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 817 T871 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4]   2> 818 T871 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1363253958823/collection1/'
[junit4:junit4]   2> 820 T871 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-controljetty-1363253958823/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 821 T871 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-controljetty-1363253958823/collection1/lib/README' to classloader
[junit4:junit4]   2> 897 T871 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 990 T871 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 1091 T871 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 1103 T871 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 1762 T871 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 1771 T871 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 1773 T871 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 1784 T871 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 1787 T871 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 1790 T871 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 1791 T871 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 1792 T871 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 1792 T871 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 1792 T871 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 1793 T871 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 1793 T871 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 1793 T871 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1363253958823/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1363253958515/control/data/
[junit4:junit4]   2> 1793 T871 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@25cbfa42
[junit4:junit4]   2> 1794 T871 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 1794 T871 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363253958515/control/data forceNew: false
[junit4:junit4]   2> 1794 T871 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1363253958515/control/data/index/
[junit4:junit4]   2> 1794 T871 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1363253958515/control/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 1795 T871 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363253958515/control/data/index forceNew: false
[junit4:junit4]   2> 1797 T871 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4e57449e lockFactory=org.apache.lucene.store.NativeFSLockFactory@4c156b4b),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 1797 T871 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 1799 T871 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 1799 T871 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 1800 T871 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 1800 T871 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 1800 T871 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 1800 T871 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 1801 T871 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 1801 T871 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 1801 T871 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 1806 T871 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 1810 T871 oass.SolrIndexSearcher.<init> Opening Searcher@51e26ce0 main
[junit4:junit4]   2> 1810 T871 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.SyncSliceTest-1363253958515/control/data/tlog
[junit4:junit4]   2> 1811 T871 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 1811 T871 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 1813 T872 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@51e26ce0 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 1814 T871 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 1814 T871 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 1814 T851 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6844977a0003 type:create cxid:0x55 zxid:0x2a txntype:-1 reqpath:n/a Error Path:/solr/overseer/queue Error:KeeperErrorCode = NoNode for /solr/overseer/queue
[junit4:junit4]   2> 2316 T869 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2317 T869 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"1",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"control_collection",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:60654_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:60654"}
[junit4:junit4]   2> 2317 T869 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4]   2> 2317 T869 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 2317 T851 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6844977a0003 type:create cxid:0x5d zxid:0x2d txntype:-1 reqpath:n/a Error Path:/solr/overseer/queue-work Error:KeeperErrorCode = NoNode for /solr/overseer/queue-work
[junit4:junit4]   2> 2331 T868 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 2830 T871 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 2830 T871 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:60654 collection:control_collection shard:shard1
[junit4:junit4]   2> 2831 T871 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4]   2> 2847 T851 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6844977a0003 type:delete cxid:0x77 zxid:0x37 txntype:-1 reqpath:n/a Error Path:/solr/collections/control_collection/leaders Error:KeeperErrorCode = NoNode for /solr/collections/control_collection/leaders
[junit4:junit4]   2> 2848 T871 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 2848 T851 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6844977a0003 type:create cxid:0x78 zxid:0x38 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 2856 T871 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 2856 T871 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 2857 T871 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:60654/collection1/
[junit4:junit4]   2> 2857 T871 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 2857 T871 oasc.SyncStrategy.syncToMe http://127.0.0.1:60654/collection1/ has no replicas
[junit4:junit4]   2> 2857 T871 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:60654/collection1/
[junit4:junit4]   2> 2857 T871 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4]   2> 2867 T851 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6844977a0003 type:create cxid:0x82 zxid:0x3c txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 3841 T869 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 3859 T868 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 3889 T871 oasc.ZkController.register We are http://127.0.0.1:60654/collection1/ and leader is http://127.0.0.1:60654/collection1/
[junit4:junit4]   2> 3889 T871 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:60654
[junit4:junit4]   2> 3889 T871 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 3889 T871 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 3890 T871 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 3891 T871 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 3892 T846 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0
[junit4:junit4]   2> 3893 T846 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 3893 T846 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 3902 T846 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4]   2> 3903 T846 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:65403/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@69bcc736
[junit4:junit4]   2> 3904 T846 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 3904 T874 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:65403. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 3905 T874 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:65403, initiating session
[junit4:junit4]   2> 3905 T848 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:50472
[junit4:junit4]   2> 3905 T848 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:50472
[junit4:junit4]   2> 3906 T850 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6844977a0004 with negotiated timeout 10000 for client /140.211.11.196:50472
[junit4:junit4]   2> 3906 T874 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:65403, sessionid = 0x13d6844977a0004, negotiated timeout = 10000
[junit4:junit4]   2> 3907 T875 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@69bcc736 name:ZooKeeperConnection Watcher:127.0.0.1:65403/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 3907 T846 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 3908 T846 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 3909 T846 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4]   2> 4034 T846 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 4036 T846 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:55618
[junit4:junit4]   2> 4037 T846 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 4037 T846 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 4037 T846 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1363253962419
[junit4:junit4]   2> 4038 T846 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1363253962419/solr.xml
[junit4:junit4]   2> 4038 T846 oasc.CoreContainer.<init> New CoreContainer 2026732578
[junit4:junit4]   2> 4038 T846 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1363253962419/'
[junit4:junit4]   2> 4039 T846 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1363253962419/'
[junit4:junit4]   2> 4071 T846 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 4072 T846 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 4072 T846 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 4072 T846 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 4072 T846 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 4073 T846 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 4073 T846 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 4073 T846 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 4074 T846 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 4074 T846 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 4087 T846 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 4100 T846 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:65403/solr
[junit4:junit4]   2> 4101 T846 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 4101 T846 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:65403 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@3be16eb2
[junit4:junit4]   2> 4102 T846 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 4102 T885 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:65403. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 4103 T885 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:65403, initiating session
[junit4:junit4]   2> 4103 T848 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:12544
[junit4:junit4]   2> 4103 T848 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:12544
[junit4:junit4]   2> 4105 T850 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6844977a0005 with negotiated timeout 20000 for client /140.211.11.196:12544
[junit4:junit4]   2> 4105 T885 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:65403, sessionid = 0x13d6844977a0005, negotiated timeout = 20000
[junit4:junit4]   2> 4106 T886 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3be16eb2 name:ZooKeeperConnection Watcher:127.0.0.1:65403 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 4106 T846 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 4107 T851 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d6844977a0005
[junit4:junit4]   2> 4118 T886 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 4118 T846 oaz.ZooKeeper.close Session: 0x13d6844977a0005 closed
[junit4:junit4]   2> 4118 T848 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:12544 which had sessionid 0x13d6844977a0005
[junit4:junit4]   2> 4118 T846 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 4125 T846 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:65403/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@16f5e10a
[junit4:junit4]   2> 4126 T846 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 4126 T887 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:65403. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 4126 T887 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:65403, initiating session
[junit4:junit4]   2> 4127 T848 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:40801
[junit4:junit4]   2> 4127 T848 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:40801
[junit4:junit4]   2> 4128 T850 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6844977a0006 with negotiated timeout 20000 for client /140.211.11.196:40801
[junit4:junit4]   2> 4128 T887 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:65403, sessionid = 0x13d6844977a0006, negotiated timeout = 20000
[junit4:junit4]   2> 4128 T888 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@16f5e10a name:ZooKeeperConnection Watcher:127.0.0.1:65403/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 4129 T846 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 4129 T851 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6844977a0006 type:create cxid:0x1 zxid:0x4a txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 4131 T851 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6844977a0006 type:create cxid:0x2 zxid:0x4b txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 4138 T846 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 5140 T846 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55618_
[junit4:junit4]   2> 5141 T851 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6844977a0006 type:delete cxid:0xb zxid:0x4c txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:55618_ Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:55618_
[junit4:junit4]   2> 5171 T846 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:55618_
[junit4:junit4]   2> 5173 T868 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 5174 T875 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 5174 T888 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 5175 T868 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 5192 T889 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1363253962419/collection1
[junit4:junit4]   2> 5192 T889 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 5192 T889 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 5192 T889 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 5193 T889 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1363253962419/collection1/'
[junit4:junit4]   2> 5194 T889 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty1-1363253962419/collection1/lib/README' to classloader
[junit4:junit4]   2> 5195 T889 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty1-1363253962419/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 5228 T889 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 5284 T889 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 5371 T869 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 5372 T869 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"1",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"control_collection",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:60654_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:60654"}
[junit4:junit4]   2> 5384 T875 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 5384 T868 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 5385 T889 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 5385 T888 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 5393 T889 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 5866 T889 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 5875 T889 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 5878 T889 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 5890 T889 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 5894 T889 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 5898 T889 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 5899 T889 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 5899 T889 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 5900 T889 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 5900 T889 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 5901 T889 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 5901 T889 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 5901 T889 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1363253962419/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1363253958515/jetty1/
[junit4:junit4]   2> 5901 T889 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@25cbfa42
[junit4:junit4]   2> 5902 T889 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 5902 T889 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363253958515/jetty1 forceNew: false
[junit4:junit4]   2> 5902 T889 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1363253958515/jetty1/index/
[junit4:junit4]   2> 5902 T889 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1363253958515/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 5903 T889 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363253958515/jetty1/index forceNew: false
[junit4:junit4]   2> 5905 T889 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2d6d4204 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7a21bd99),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 5905 T889 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 5907 T889 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 5907 T889 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 5907 T889 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 5908 T889 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 5908 T889 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 5909 T889 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 5909 T889 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 5909 T889 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 5909 T889 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 5914 T889 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 5918 T889 oass.SolrIndexSearcher.<init> Opening Searcher@2ea0ca7d main
[junit4:junit4]   2> 5919 T889 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.SyncSliceTest-1363253958515/jetty1/tlog
[junit4:junit4]   2> 5919 T889 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 5920 T889 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 5922 T890 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2ea0ca7d main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 5924 T889 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 5924 T889 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 6889 T869 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 6890 T869 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"2",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:55618_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:55618"}
[junit4:junit4]   2> 6890 T869 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 1
[junit4:junit4]   2> 6890 T869 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 6931 T868 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 6931 T875 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 6932 T888 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 7928 T889 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 7928 T889 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:55618 collection:collection1 shard:shard1
[junit4:junit4]   2> 7929 T889 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 7975 T851 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6844977a0006 type:delete cxid:0x48 zxid:0x5c txntype:-1 reqpath:n/a Error Path:/solr/collections/collection1/leaders Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders
[junit4:junit4]   2> 7975 T889 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 7976 T851 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6844977a0006 type:create cxid:0x49 zxid:0x5d txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 7985 T889 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 7985 T889 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 7985 T889 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:55618/collection1/
[junit4:junit4]   2> 7985 T889 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 7985 T889 oasc.SyncStrategy.syncToMe http://127.0.0.1:55618/collection1/ has no replicas
[junit4:junit4]   2> 7986 T889 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:55618/collection1/
[junit4:junit4]   2> 7986 T889 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4]   2> 7989 T851 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6844977a0006 type:create cxid:0x53 zxid:0x61 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 8435 T869 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 8452 T868 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 8452 T875 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 8452 T888 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 8510 T889 oasc.ZkController.register We are http://127.0.0.1:55618/collection1/ and leader is http://127.0.0.1:55618/collection1/
[junit4:junit4]   2> 8510 T889 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:55618
[junit4:junit4]   2> 8510 T889 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 8510 T889 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 8511 T889 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 8513 T889 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 8514 T846 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0
[junit4:junit4]   2> 8514 T846 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 8515 T846 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 8651 T846 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 8653 T846 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:32313
[junit4:junit4]   2> 8654 T846 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 8654 T846 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 8654 T846 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1363253967033
[junit4:junit4]   2> 8655 T846 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1363253967033/solr.xml
[junit4:junit4]   2> 8655 T846 oasc.CoreContainer.<init> New CoreContainer 1907774383
[junit4:junit4]   2> 8655 T846 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1363253967033/'
[junit4:junit4]   2> 8656 T846 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1363253967033/'
[junit4:junit4]   2> 8689 T846 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 8689 T846 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 8690 T846 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 8690 T846 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 8690 T846 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 8691 T846 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 8691 T846 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 8691 T846 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 8691 T846 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 8692 T846 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 8705 T846 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 8718 T846 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:65403/solr
[junit4:junit4]   2> 8718 T846 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 8718 T846 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:65403 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@4ed98965
[junit4:junit4]   2> 8719 T846 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 8720 T901 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:65403. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 8720 T901 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:65403, initiating session
[junit4:junit4]   2> 8721 T848 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:39842
[junit4:junit4]   2> 8721 T848 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:39842
[junit4:junit4]   2> 8723 T850 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6844977a0007 with negotiated timeout 20000 for client /140.211.11.196:39842
[junit4:junit4]   2> 8723 T901 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:65403, sessionid = 0x13d6844977a0007, negotiated timeout = 20000
[junit4:junit4]   2> 8723 T902 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4ed98965 name:ZooKeeperConnection Watcher:127.0.0.1:65403 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 8723 T846 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 8724 T851 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d6844977a0007
[junit4:junit4]   2> 8734 T902 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 8734 T848 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:39842 which had sessionid 0x13d6844977a0007
[junit4:junit4]   2> 8734 T846 oaz.ZooKeeper.close Session: 0x13d6844977a0007 closed
[junit4:junit4]   2> 8734 T846 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 8741 T846 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:65403/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@219776d7
[junit4:junit4]   2> 8742 T846 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 8742 T903 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:65403. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 8742 T903 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:65403, initiating session
[junit4:junit4]   2> 8743 T848 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:20105
[junit4:junit4]   2> 8743 T848 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:20105
[junit4:junit4]   2> 8744 T850 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6844977a0008 with negotiated timeout 20000 for client /140.211.11.196:20105
[junit4:junit4]   2> 8744 T903 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:65403, sessionid = 0x13d6844977a0008, negotiated timeout = 20000
[junit4:junit4]   2> 8744 T904 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@219776d7 name:ZooKeeperConnection Watcher:127.0.0.1:65403/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 8745 T846 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 8745 T851 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6844977a0008 type:create cxid:0x1 zxid:0x6e txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 8751 T851 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6844977a0008 type:create cxid:0x2 zxid:0x6f txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 8753 T846 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 9755 T846 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:32313_
[junit4:junit4]   2> 9756 T851 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6844977a0008 type:delete cxid:0xb zxid:0x70 txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:32313_ Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:32313_
[junit4:junit4]   2> 9758 T846 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:32313_
[junit4:junit4]   2> 9760 T888 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 9760 T868 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 9760 T875 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 9760 T904 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 9761 T875 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 9769 T888 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 9769 T868 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 9775 T905 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1363253967033/collection1
[junit4:junit4]   2> 9775 T905 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 9776 T905 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 9776 T905 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 9777 T905 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1363253967033/collection1/'
[junit4:junit4]   2> 9779 T905 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty2-1363253967033/collection1/lib/README' to classloader
[junit4:junit4]   2> 9779 T905 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty2-1363253967033/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 9826 T905 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 9882 T905 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 9965 T869 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 9966 T869 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"2",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:55618_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:55618"}
[junit4:junit4]   2> 9979 T868 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 9979 T888 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 9979 T875 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 9983 T904 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 9983 T905 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 9991 T905 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 10636 T905 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 10649 T905 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 10653 T905 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 10669 T905 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 10674 T905 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 10679 T905 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 10680 T905 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 10681 T905 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 10681 T905 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 10682 T905 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 10682 T905 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 10683 T905 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 10683 T905 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1363253967033/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1363253958515/jetty2/
[junit4:junit4]   2> 10683 T905 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@25cbfa42
[junit4:junit4]   2> 10684 T905 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 10684 T905 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363253958515/jetty2 forceNew: false
[junit4:junit4]   2> 10685 T905 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1363253958515/jetty2/index/
[junit4:junit4]   2> 10685 T905 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1363253958515/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 10685 T905 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363253958515/jetty2/index forceNew: false
[junit4:junit4]   2> 10688 T905 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2909423b lockFactory=org.apache.lucene.store.NativeFSLockFactory@4d708e32),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 10688 T905 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 10691 T905 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 10691 T905 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 10692 T905 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 10693 T905 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 10693 T905 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 10693 T905 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 10694 T905 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 10694 T905 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 10695 T905 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 10702 T905 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 10708 T905 oass.SolrIndexSearcher.<init> Opening Searcher@7796af5d main
[junit4:junit4]   2> 10708 T905 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.SyncSliceTest-1363253958515/jetty2/tlog
[junit4:junit4]   2> 10709 T905 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 10710 T905 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 10717 T905 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 10717 T906 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7796af5d main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 10718 T905 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 11483 T869 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 11484 T869 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"3",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:32313_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:32313"}
[junit4:junit4]   2> 11484 T869 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4]   2> 11485 T869 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 11497 T868 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 11497 T888 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 11497 T904 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 11497 T875 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 11754 T905 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 11754 T905 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:32313 collection:collection1 shard:shard1
[junit4:junit4]   2> 11757 T905 oasc.ZkController.register We are http://127.0.0.1:32313/collection1/ and leader is http://127.0.0.1:55618/collection1/
[junit4:junit4]   2> 11757 T905 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:32313
[junit4:junit4]   2> 11758 T905 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 11758 T905 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C28 name=collection1 org.apache.solr.core.SolrCore@449ff1d3 url=http://127.0.0.1:32313/collection1 node=127.0.0.1:32313_ C28_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:32313_, base_url=http://127.0.0.1:32313}
[junit4:junit4]   2> 11758 T907 C28 P32313 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 11759 T905 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 11759 T907 C28 P32313 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 11760 T907 C28 P32313 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4]   2> 11760 T907 C28 P32313 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 11760 T846 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0
[junit4:junit4]   2> 11761 T846 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 11761 T846 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 11781 T907 C28 P32313 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 11973 T846 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 11976 T846 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:41673
[junit4:junit4]   2> 11976 T846 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 11977 T846 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 11977 T846 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1363253970287
[junit4:junit4]   2> 11978 T846 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1363253970287/solr.xml
[junit4:junit4]   2> 11978 T846 oasc.CoreContainer.<init> New CoreContainer 1564839734
[junit4:junit4]   2> 11979 T846 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1363253970287/'
[junit4:junit4]   2> 11979 T846 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1363253970287/'
[junit4:junit4]   2> 12029 T846 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 12030 T846 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 12030 T846 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 12030 T846 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 12031 T846 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 12031 T846 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 12032 T846 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 12032 T846 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 12033 T846 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 12033 T846 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 12055 T846 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 12074 T846 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:65403/solr
[junit4:junit4]   2> 12074 T846 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 12075 T846 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:65403 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@3456dafb
[junit4:junit4]   2> 12076 T846 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 12076 T918 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:65403. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 12077 T918 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:65403, initiating session
[junit4:junit4]   2> 12077 T848 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:16545
[junit4:junit4]   2> 12078 T848 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:16545
[junit4:junit4]   2> 12080 T850 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6844977a0009 with negotiated timeout 20000 for client /140.211.11.196:16545
[junit4:junit4]   2> 12080 T918 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:65403, sessionid = 0x13d6844977a0009, negotiated timeout = 20000
[junit4:junit4]   2> 12080 T919 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3456dafb name:ZooKeeperConnection Watcher:127.0.0.1:65403 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 12080 T846 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 12081 T851 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d6844977a0009
[junit4:junit4]   2> 12090 T919 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 12090 T848 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:16545 which had sessionid 0x13d6844977a0009
[junit4:junit4]   2> 12090 T846 oaz.ZooKeeper.close Session: 0x13d6844977a0009 closed
[junit4:junit4]   2> 12091 T846 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 12103 T846 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:65403/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@76be3008
[junit4:junit4]   2> 12104 T846 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 12104 T920 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:65403. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 12105 T920 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:65403, initiating session
[junit4:junit4]   2> 12105 T848 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:10340
[junit4:junit4]   2> 12106 T848 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:10340
[junit4:junit4]   2> 12107 T850 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6844977a000a with negotiated timeout 20000 for client /140.211.11.196:10340
[junit4:junit4]   2> 12107 T920 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:65403, sessionid = 0x13d6844977a000a, negotiated timeout = 20000
[junit4:junit4]   2> 12107 T921 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@76be3008 name:ZooKeeperConnection Watcher:127.0.0.1:65403/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 12107 T846 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 12108 T851 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6844977a000a type:create cxid:0x1 zxid:0x81 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 12110 T851 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6844977a000a type:create cxid:0x2 zxid:0x82 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 12117 T846 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 13001 T869 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 13002 T869 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"3",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:32313_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:32313"}
[junit4:junit4]   2> 13038 T888 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 13038 T904 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 13038 T868 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 13038 T921 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 13038 T875 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 13119 T846 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41673_
[junit4:junit4]   2> 13120 T851 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6844977a000a type:delete cxid:0xd zxid:0x87 txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:41673_ Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:41673_
[junit4:junit4]   2> 13122 T846 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:41673_
[junit4:junit4]   2> 13123 T888 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 13123 T904 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 13123 T868 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 13123 T921 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 13123 T875 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 13124 T921 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 13124 T875 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 13125 T888 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 13133 T868 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 13133 T904 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 13138 T922 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1363253970287/collection1
[junit4:junit4]   2> 13138 T922 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 13138 T922 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 13139 T922 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 13139 T922 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1363253970287/collection1/'
[junit4:junit4]   2> 13140 T922 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty3-1363253970287/collection1/lib/README' to classloader
[junit4:junit4]   2> 13141 T922 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty3-1363253970287/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 13174 T922 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 13216 T922 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 13317 T922 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 13322 T922 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 13742 T922 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 13750 T922 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 13753 T922 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 13766 T922 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 13769 T922 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 13772 T922 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 13773 T922 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 13773 T922 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 13773 T922 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 13774 T922 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 13774 T922 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 13774 T922 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 13774 T922 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1363253970287/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1363253958515/jetty3/
[junit4:junit4]   2> 13775 T922 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@25cbfa42
[junit4:junit4]   2> 13775 T922 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 13775 T922 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363253958515/jetty3 forceNew: false
[junit4:junit4]   2> 13776 T922 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1363253958515/jetty3/index/
[junit4:junit4]   2> 13776 T922 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1363253958515/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 13776 T922 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363253958515/jetty3/index forceNew: false
[junit4:junit4]   2> 13779 T922 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2ad3c1ec lockFactory=org.apache.lucene.store.NativeFSLockFactory@2c332d2a),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 13780 T922 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 13782 T922 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 13782 T922 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 13782 T922 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 13783 T922 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 13783 T922 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 13783 T922 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 13784 T922 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 13784 T922 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 13784 T922 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 13789 T922 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 13794 T922 oass.SolrIndexSearcher.<init> Opening Searcher@35d249e4 main
[junit4:junit4]   2> 13795 T922 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.SyncSliceTest-1363253958515/jetty3/tlog
[junit4:junit4]   2> 13795 T922 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 13795 T922 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 13797 T878 oass.SolrDispatchFilter.handleAdminRequest /admin/cores {coreNodeName=3&state=recovering&nodeName=127.0.0.1:32313_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=2002 
[junit4:junit4]   2> 13798 T923 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@35d249e4 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 13800 T922 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 13800 T922 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 14547 T869 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 14548 T869 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"4",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:41673_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:41673"}
[junit4:junit4]   2> 14548 T869 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4]   2> 14548 T869 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 14581 T921 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 14581 T904 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 14581 T875 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 14581 T888 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 14581 T868 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 14806 T922 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 14806 T922 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:41673 collection:collection1 shard:shard1
[junit4:junit4]   2> 14810 T922 oasc.ZkController.register We are http://127.0.0.1:41673/collection1/ and leader is http://127.0.0.1:55618/collection1/
[junit4:junit4]   2> 14810 T922 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:41673
[junit4:junit4]   2> 14810 T922 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 14810 T922 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C29 name=collection1 org.apache.solr.core.SolrCore@72ced13f url=http://127.0.0.1:41673/collection1 node=127.0.0.1:41673_ C29_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:41673_, base_url=http://127.0.0.1:41673}
[junit4:junit4]   2> 14811 T924 C29 P41673 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 14811 T922 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 14812 T924 C29 P41673 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 14812 T924 C29 P41673 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4]   2> 14813 T924 C29 P41673 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 14813 T846 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0
[junit4:junit4]   2> 14813 T846 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 14814 T846 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 14814 T924 C29 P41673 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 14980 T846 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 14996 T846 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:13969
[junit4:junit4]   2> 14997 T846 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 14997 T846 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 14997 T846 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1363253973335
[junit4:junit4]   2> 14997 T846 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1363253973335/solr.xml
[junit4:junit4]   2> 14998 T846 oasc.CoreContainer.<init> New CoreContainer 415225662
[junit4:junit4]   2> 14998 T846 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1363253973335/'
[junit4:junit4]   2> 14998 T846 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1363253973335/'
[junit4:junit4]   2> 15032 T846 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 15032 T846 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 15033 T846 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 15033 T846 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 15033 T846 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 15034 T846 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 15034 T846 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 15034 T846 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 15034 T846 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 15035 T846 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 15049 T846 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 15063 T846 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:65403/solr
[junit4:junit4]   2> 15064 T846 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 15064 T846 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:65403 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@14731467
[junit4:junit4]   2> 15065 T846 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 15066 T935 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:65403. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 15067 T935 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:65403, initiating session
[junit4:junit4]   2> 15067 T848 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:42607
[junit4:junit4]   2> 15067 T848 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:42607
[junit4:junit4]   2> 15069 T850 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6844977a000b with negotiated timeout 20000 for client /140.211.11.196:42607
[junit4:junit4]   2> 15069 T935 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:65403, sessionid = 0x13d6844977a000b, negotiated timeout = 20000
[junit4:junit4]   2> 15069 T936 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@14731467 name:ZooKeeperConnection Watcher:127.0.0.1:65403 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 15069 T846 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 15070 T851 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d6844977a000b
[junit4:junit4]   2> 15071 T936 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 15071 T846 oaz.ZooKeeper.close Session: 0x13d6844977a000b closed
[junit4:junit4]   2> 15071 T848 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:42607 which had sessionid 0x13d6844977a000b
[junit4:junit4]   2> 15072 T846 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 15080 T846 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:65403/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@866bb0
[junit4:junit4]   2> 15080 T846 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 15081 T937 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:65403. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 15081 T937 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:65403, initiating session
[junit4:junit4]   2> 15082 T848 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:46964
[junit4:junit4]   2> 15082 T848 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:46964
[junit4:junit4]   2> 15083 T850 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6844977a000c with negotiated timeout 20000 for client /140.211.11.196:46964
[junit4:junit4]   2> 15083 T937 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:65403, sessionid = 0x13d6844977a000c, negotiated timeout = 20000
[junit4:junit4]   2> 15084 T938 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@866bb0 name:ZooKeeperConnection Watcher:127.0.0.1:65403/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 15084 T846 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 15084 T851 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6844977a000c type:create cxid:0x1 zxid:0x94 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 15086 T851 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6844977a000c type:create cxid:0x2 zxid:0x95 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 15087 T846 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2>  C28_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:32313_, base_url=http://127.0.0.1:32313}
[junit4:junit4]   2> 15798 T907 C28 P32313 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:55618/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4]   2> 15798 T907 C28 P32313 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:32313 START replicas=[http://127.0.0.1:55618/collection1/] nUpdates=100
[junit4:junit4]   2> 15799 T907 C28 P32313 oasu.PeerSync.sync WARNING no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 15801 T907 C28 P32313 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4]   2> 15801 T907 C28 P32313 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4]   2> 15801 T907 C28 P32313 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4]   2> 15801 T907 C28 P32313 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4]   2> 15802 T907 C28 P32313 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:55618/collection1/. core=collection1
[junit4:junit4]   2> 15802 T907 C28 P32313 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C30 name=collection1 org.apache.solr.core.SolrCore@f1917cf url=http://127.0.0.1:55618/collection1 node=127.0.0.1:55618_ C30_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:55618_, base_url=http://127.0.0.1:55618, leader=true}
[junit4:junit4]   2> 15818 T880 C30 P55618 REQ /get {getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=9 
[junit4:junit4]   2> 15822 T881 C30 P55618 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 15825 T881 C30 P55618 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2d6d4204 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7a21bd99),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 15825 T881 C30 P55618 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 15826 T881 C30 P55618 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2d6d4204 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7a21bd99),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2d6d4204 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7a21bd99),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 15826 T881 C30 P55618 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[segments_2]
[junit4:junit4]   2> 15826 T881 C30 P55618 oass.SolrIndexSearcher.<init> Opening Searcher@26c0e0f3 realtime
[junit4:junit4]   2> 15827 T881 C30 P55618 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 15827 T881 C30 P55618 /update {waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 5
[junit4:junit4]   2> 15828 T907 C28 P32313 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 15828 T907 C28 P32313 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4]   2> 15836 T882 C30 P55618 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 15836 T882 C30 P55618 REQ /replication {command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=7 
[junit4:junit4]   2> 15838 T907 C28 P32313 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4]   2> 15838 T907 C28 P32313 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4]   2> 15838 T907 C28 P32313 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4]   2> 15841 T882 C30 P55618 REQ /replication {command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=1 
[junit4:junit4]   2> 15842 T907 C28 P32313 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4]   2> 15853 T907 C28 P32313 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363253958515/jetty2/index.20130314113934352 forceNew: false
[junit4:junit4]   2> 15853 T907 C28 P32313 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@30f5ab56 lockFactory=org.apache.lucene.store.NativeFSLockFactory@606c6ea6) fullCopy=false
[junit4:junit4]   2> 15857 T882 C30 P55618 REQ /replication {file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=1 
[junit4:junit4]   2> 15858 T907 C28 P32313 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4]   2> 15861 T907 C28 P32313 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4]   2> 15861 T907 C28 P32313 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4]   2> 15868 T907 C28 P32313 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2909423b lockFactory=org.apache.lucene.store.NativeFSLockFactory@4d708e32),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2909423b lockFactory=org.apache.lucene.store.NativeFSLockFactory@4d708e32),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 15869 T907 C28 P32313 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[segments_2]
[junit4:junit4]   2> 15869 T907 C28 P32313 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4]   2> 15869 T907 C28 P32313 oass.SolrIndexSearcher.<init> Opening Searcher@7387d7ee main
[junit4:junit4]   2> 15870 T906 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7387d7ee main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 15870 T907 C28 P32313 oash.SnapPuller.fetchLatestIndex removing temporary index download directory files MockDirWrapper(org.apache.lucene.store.RAMDirectory@30f5ab56 lockFactory=org.apache.lucene.store.NativeFSLockFactory@606c6ea6)
[junit4:junit4]   2> 15870 T907 C28 P32313 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4]   2> 15870 T907 C28 P32313 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4]   2> 15871 T907 C28 P32313 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 15871 T907 C28 P32313 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 15878 T907 C28 P32313 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4]   2> 16090 T846 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:13969_
[junit4:junit4]   2> 16091 T851 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6844977a000c type:delete cxid:0xb zxid:0x97 txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:13969_ Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:13969_
[junit4:junit4]   2> 16112 T846 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:13969_
[junit4:junit4]   2> 16114 T869 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 16115 T904 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 16115 T888 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 16116 T921 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 16116 T938 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 16115 T868 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 16116 T921 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 16116 T875 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 16117 T904 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 16116 T869 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"4",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:41673_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:41673"}
[junit4:junit4]   2> 16117 T875 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 16118 T888 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 16135 T869 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"3",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:32313_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:32313"}
[junit4:junit4]   2> 16136 T941 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1363253973335/collection1
[junit4:junit4]   2> 16136 T941 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 16143 T941 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 16143 T941 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 16145 T941 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1363253973335/collection1/'
[junit4:junit4]   2> 16146 T941 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty4-1363253973335/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 16146 T941 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty4-1363253973335/collection1/lib/README' to classloader
[junit4:junit4]   2> 16151 T904 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 16151 T938 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 16151 T875 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 16151 T921 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 16151 T888 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 16156 T868 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 16156 T868 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 16195 T941 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 16253 T941 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 16354 T941 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 16361 T941 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 16830 T879 oass.SolrDispatchFilter.handleAdminRequest /admin/cores {coreNodeName=4&state=recovering&nodeName=127.0.0.1:41673_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=2002 
[junit4:junit4]   2> 16997 T941 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 17010 T941 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 17014 T941 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 17029 T941 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 17033 T941 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 17038 T941 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 17039 T941 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 17040 T941 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 17040 T941 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 17041 T941 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 17041 T941 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 17041 T941 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 17042 T941 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1363253973335/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1363253958515/jetty4/
[junit4:junit4]   2> 17042 T941 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@25cbfa42
[junit4:junit4]   2> 17042 T941 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 17043 T941 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363253958515/jetty4 forceNew: false
[junit4:junit4]   2> 17043 T941 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1363253958515/jetty4/index/
[junit4:junit4]   2> 17043 T941 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1363253958515/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 17044 T941 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363253958515/jetty4/index forceNew: false
[junit4:junit4]   2> 17047 T941 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@492d49e0 lockFactory=org.apache.lucene.store.NativeFSLockFactory@713f6e85),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 17047 T941 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 17050 T941 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 17050 T941 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 17050 T941 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 17051 T941 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 17052 T941 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 17052 T941 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 17052 T941 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 17053 T941 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 17053 T941 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 17060 T941 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 17066 T941 oass.SolrIndexSearcher.<init> Opening Searcher@2ca158e8 main
[junit4:junit4]   2> 17067 T941 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.SyncSliceTest-1363253958515/jetty4/tlog
[junit4:junit4]   2> 17068 T941 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 17068 T941 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 17072 T942 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2ca158e8 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 17074 T941 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 17075 T941 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 17657 T869 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 17658 T869 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"5",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:13969_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:13969"}
[junit4:junit4]   2> 17658 T869 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4]   2> 17658 T869 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 17671 T921 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 17671 T904 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 17672 T938 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 17671 T875 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 17671 T868 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 17671 T888 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 18077 T941 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 18077 T941 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:13969 collection:collection1 shard:shard1
[junit4:junit4]   2> 18082 T941 oasc.ZkController.register We are http://127.0.0.1:13969/collection1/ and leader is http://127.0.0.1:55618/collection1/
[junit4:junit4]   2> 18082 T941 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:13969
[junit4:junit4]   2> 18083 T941 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 18083 T941 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C31 name=collection1 org.apache.solr.core.SolrCore@7ba75fd0 url=http://127.0.0.1:13969/collection1 node=127.0.0.1:13969_ C31_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:13969_, base_url=http://127.0.0.1:13969}
[junit4:junit4]   2> 18083 T943 C31 P13969 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 18084 T943 C31 P13969 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 18084 T943 C31 P13969 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4]   2> 18084 T941 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 18085 T943 C31 P13969 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 18085 T846 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J0
[junit4:junit4]   2> 18086 T846 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 18086 T943 C31 P13969 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 18086 T846 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 18101 T846 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 18102 T846 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 15 for each attempt
[junit4:junit4]   2> 18102 T846 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):15
[junit4:junit4]   2> 18103 T846 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2>  C29_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:41673_, base_url=http://127.0.0.1:41673}
[junit4:junit4]   2> 18832 T924 C29 P41673 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:55618/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4]   2> 18832 T924 C29 P41673 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:41673 START replicas=[http://127.0.0.1:55618/collection1/] nUpdates=100
[junit4:junit4]   2> 18833 T924 C29 P41673 oasu.PeerSync.sync WARNING no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 18833 T924 C29 P41673 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4]   2> 18833 T924 C29 P41673 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4]   2> 18833 T924 C29 P41673 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4]   2> 18833 T924 C29 P41673 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4]   2> 18834 T924 C29 P41673 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:55618/collection1/. core=collection1
[junit4:junit4]   2> 18834 T924 C29 P41673 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 18836 T880 C30 P55618 REQ /get {getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0 
[junit4:junit4]   2> 18848 T884 C30 P55618 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 18849 T884 C30 P55618 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2d6d4204 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7a21bd99),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2d6d4204 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7a21bd99),segFN=segments_3,generation=3,filenames=[segments_3]
[junit4:junit4]   2> 18850 T884 C30 P55618 oasc.SolrDeletionPolicy.updateCommits newest commit = 3[segments_3]
[junit4:junit4]   2> 18850 T884 C30 P55618 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 18851 T884 C30 P55618 /update {waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 3
[junit4:junit4]   2> 18852 T924 C29 P41673 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 18852 T924 C29 P41673 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4]   2> 18854 T882 C30 P55618 REQ /replication {command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 18855 T924 C29 P41673 oash.SnapPuller.fetchLatestIndex Master's generation: 3
[junit4:junit4]   2> 18855 T924 C29 P41673 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4]   2> 18856 T924 C29 P41673 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4]   2> 18859 T882 C30 P55618 REQ /replication {command=filelist&qt=/replication&wt=javabin&generation=3&version=2} status=0 QTime=1 
[junit4:junit4]   2> 18859 T924 C29 P41673 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4]   2> 18861 T924 C29 P41673 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363253958515/jetty3/index.20130314113937369 forceNew: false
[junit4:junit4]   2> 18861 T924 C29 P41673 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@1a2434e7 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7c9c6aa5) fullCopy=false
[junit4:junit4]   2> 18864 T882 C30 P55618 REQ /replication {file=segments_3&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=3} status=0 QTime=0 
[junit4:junit4]   2> 18865 T924 C29 P41673 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4]   2> 18866 T924 C29 P41673 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4]   2> 18866 T924 C29 P41673 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4]   2> 18868 T924 C29 P41673 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2ad3c1ec lockFactory=org.apache.lucene.store.NativeFSLockFactory@2c332d2a),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2ad3c1ec lockFactory=org.apache.lucene.store.NativeFSLockFactory@2c332d2a),segFN=segments_3,generation=3,filenames=[segments_3]
[junit4:junit4]   2> 18868 T924 C29 P41673 oasc.SolrDeletionPolicy.updateCommits newest commit = 3[segments_3]
[junit4:junit4]   2> 18868 T924 C29 P41673 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4]   2> 18869 T924 C29 P41673 oass.SolrIndexSearcher.<init> Opening Searcher@78d88d3e main
[junit4:junit4]   2> 18870 T923 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@78d88d3e main{StandardDirectoryReader(segments_3:1:nrt)}
[junit4:junit4]   2> 18870 T924 C29 P41673 oash.SnapPuller.fetchLatestIndex removing temporary index download directory files MockDirWrapper(org.apache.lucene.store.RAMDirectory@1a2434e7 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7c9c6aa5)
[junit4:junit4]   2> 18870 T924 C29 P41673 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4]   2> 18870 T924 C29 P41673 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4]   2> 18870 T924 C29 P41673 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 18871 T924 C29 P41673 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 18873 T924 C29 P41673 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4]   2> 19105 T846 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 19178 T869 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 19179 T869 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"5",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:13969_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:13969"}
[junit4:junit4]   2> 19191 T869 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"4",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:41673_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:41673"}
[junit4:junit4]   2> 19203 T868 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 19203 T921 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 19203 T938 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 19203 T888 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 19203 T875 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 19203 T904 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 20105 T883 oass.SolrDispatchFilter.handleAdminRequest /admin/cores {coreNodeName=5&state=recovering&nodeName=127.0.0.1:13969_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=2002 
[junit4:junit4]   2> 20107 T846 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 21109 T846 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2>  C31_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:13969_, base_url=http://127.0.0.1:13969}
[junit4:junit4]   2> 22107 T943 C31 P13969 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:55618/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4]   2> 22107 T943 C31 P13969 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:13969 START replicas=[http://127.0.0.1:55618/collection1/] nUpdates=100
[junit4:junit4]   2> 22108 T943 C31 P13969 oasu.PeerSync.sync WARNING no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 22108 T943 C31 P13969 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4]   2> 22108 T943 C31 P13969 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4]   2> 22108 T943 C31 P13969 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4]   2> 22108 T943 C31 P13969 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4]   2> 22108 T943 C31 P13969 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:55618/collection1/. core=collection1
[junit4:junit4]   2> 22108 T943 C31 P13969 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 22110 T880 C30 P55618 REQ /get {getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0 
[junit4:junit4]   2> 22111 T846 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 22117 T878 C30 P55618 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 22118 T878 C30 P55618 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=3
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2d6d4204 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7a21bd99),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2d6d4204 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7a21bd99),segFN=segments_3,generation=3,filenames=[segments_3]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2d6d4204 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7a21bd99),segFN=segments_4,generation=4,filenames=[segments_4]
[junit4:junit4]   2> 22118 T878 C30 P55618 oasc.SolrDeletionPolicy.updateCommits newest commit = 4[segments_4]
[junit4:junit4]   2> 22118 T878 C30 P55618 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 22119 T878 C30 P55618 /update {waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 2
[junit4:junit4]   2> 22119 T943 C31 P13969 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 22120 T943 C31 P13969 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4]   2> 22122 T882 C30 P55618 REQ /replication {command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 22123 T943 C31 P13969 oash.SnapPuller.fetchLatestIndex Master's generation: 4
[junit4:junit4]   2> 22123 T943 C31 P13969 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4]   2> 22123 T943 C31 P13969 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4]   2> 22126 T882 C30 P55618 REQ /replication {command=filelist&qt=/replication&wt=javabin&generation=4&version=2} status=0 QTime=1 
[junit4:junit4]   2> 22126 T943 C31 P13969 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4]   2> 22127 T943 C31 P13969 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363253958515/jetty4/index.20130314113940636 forceNew: false
[junit4:junit4]   2> 22127 T943 C31 P13969 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@5ce6129 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2564575) fullCopy=false
[junit4:junit4]   2> 22130 T882 C30 P55618 REQ /replication {file=segments_4&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=4} status=0 QTime=0 
[junit4:junit4]   2> 22131 T943 C31 P13969 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4]   2> 22132 T943 C31 P13969 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4]   2> 22132 T943 C31 P13969 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4]   2> 22134 T943 C31 P13969 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@492d49e0 lockFactory=org.apache.lucene.store.NativeFSLockFactory@713f6e85),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@492d49e0 lockFactory=org.apache.lucene.store.NativeFSLockFactory@713f6e85),segFN=segments_4,generation=4,filenames=[segments_4]
[junit4:junit4]   2> 22134 T943 C31 P13969 oasc.SolrDeletionPolicy.updateCommits newest commit = 4[segments_4]
[junit4:junit4]   2> 22134 T943 C31 P13969 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4]   2> 22135 T943 C31 P13969 oass.SolrIndexSearcher.<init> Opening Searcher@37d0d20b main
[junit4:junit4]   2> 22135 T942 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@37d0d20b main{StandardDirectoryReader(segments_4:1:nrt)}
[junit4:junit4]   2> 22136 T943 C31 P13969 oash.SnapPuller.fetchLatestIndex removing temporary index download directory files MockDirWrapper(org.apache.lucene.store.RAMDirectory@5ce6129 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2564575)
[junit4:junit4]   2> 22136 T943 C31 P13969 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4]   2> 22136 T943 C31 P13969 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4]   2> 22136 T943 C31 P13969 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 22136 T943 C31 P13969 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 22163 T943 C31 P13969 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4]   2> 22215 T869 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 22215 T869 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"5",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:13969_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:13969"}
[junit4:junit4]   2> 22230 T904 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 22230 T938 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 22230 T875 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 22230 T921 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 22230 T888 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 22230 T868 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 23113 T846 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 23114 T846 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4]   2> ASYNC  NEW_CORE C32 name=collection1 org.apache.solr.core.SolrCore@17694e84 url=http://127.0.0.1:60654/collection1 node=127.0.0.1:60654_ C32_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:60654_, base_url=http://127.0.0.1:60654, leader=true}
[junit4:junit4]   2> 23120 T858 C32 P60654 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 23124 T858 C32 P60654 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4e57449e lockFactory=org.apache.lucene.store.NativeFSLockFactory@4c156b4b),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 23124 T858 C32 P60654 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 23125 T858 C32 P60654 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4e57449e lockFactory=org.apache.lucene.store.NativeFSLockFactory@4c156b4b),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4e57449e lockFactory=org.apache.lucene.store.NativeFSLockFactory@4c156b4b),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 23125 T858 C32 P60654 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[segments_2]
[junit4:junit4]   2> 23126 T858 C32 P60654 oass.SolrIndexSearcher.<init> Opening Searcher@7d74bd37 main
[junit4:junit4]   2> 23126 T858 C32 P60654 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 23127 T872 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7d74bd37 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 23127 T858 C32 P60654 /update {waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 7
[junit4:junit4]   2> 23137 T881 C30 P55618 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 23137 T881 C30 P55618 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=4
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2d6d4204 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7a21bd99),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2d6d4204 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7a21bd99),segFN=segments_3,generation=3,filenames=[segments_3]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2d6d4204 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7a21bd99),segFN=segments_4,generation=4,filenames=[segments_4]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2d6d4204 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7a21bd99),segFN=segments_5,generation=5,filenames=[segments_5]
[junit4:junit4]   2> 23138 T881 C30 P55618 oasc.SolrDeletionPolicy.updateCommits newest commit = 5[segments_5]
[junit4:junit4]   2> 23138 T881 C30 P55618 oass.SolrIndexSearcher.<init> Opening Searcher@1d6e13d7 main
[junit4:junit4]   2> 23138 T881 C30 P55618 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 23139 T890 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1d6e13d7 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 23139 T881 C30 P55618 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:32313/collection1/, StdNode: http://127.0.0.1:41673/collection1/, StdNode: http://127.0.0.1:13969/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4]   2>  C31_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:13969_, base_url=http://127.0.0.1:13969}
[junit4:junit4]   2> 23146 T928 C31 P13969 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2>  C29_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:41673_, base_url=http://127.0.0.1:41673}
[junit4:junit4]   2> 23146 T911 C29 P41673 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2>  C28_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:32313_, base_url=http://127.0.0.1:32313}
[junit4:junit4]   2> 23146 T894 C28 P32313 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 23147 T911 C29 P41673 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2ad3c1ec lockFactory=org.apache.lucene.store.NativeFSLockFactory@2c332d2a),segFN=segments_3,generation=3,filenames=[segments_3]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2ad3c1ec lockFactory=org.apache.lucene.store.NativeFSLockFactory@2c332d2a),segFN=segments_4,generation=4,filenames=[segments_4]
[junit4:junit4]   2> 23147 T928 C31 P13969 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@492d49e0 lockFactory=org.apache.lucene.store.NativeFSLockFactory@713f6e85),segFN=segments_4,generation=4,filenames=[segments_4]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@492d49e0 lockFactory=org.apache.lucene.store.NativeFSLockFactory@713f6e85),segFN=segments_5,generation=5,filenames=[segments_5]
[junit4:junit4]   2> 23147 T894 C28 P32313 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2909423b lockFactory=org.apache.lucene.store.NativeFSLockFactory@4d708e32),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2909423b lockFactory=org.apache.lucene.store.NativeFSLockFactory@4d708e32),segFN=segments_3,generation=3,filenames=[segments_3]
[junit4:junit4]   2> 23148 T911 C29 P41673 oasc.SolrDeletionPolicy.updateCommits newest commit = 4[segments_4]
[junit4:junit4]   2> 23148 T928 C31 P13969 oasc.SolrDeletionPolicy.updateCommits newest commit = 5[segments_5]
[junit4:junit4]   2> 23148 T894 C28 P32313 oasc.SolrDeletionPolicy.updateCommits newest commit = 3[segments_3]
[junit4:junit4]   2> 23148 T911 C29 P41673 oass.SolrIndexSearcher.<init> Opening Searcher@e89c98 main
[junit4:junit4]   2> 23148 T928 C31 P13969 oass.SolrIndexSearcher.<init> Opening Searcher@75a473fb main
[junit4:junit4]   2> 23148 T894 C28 P32313 oass.SolrIndexSearcher.<init> Opening Searcher@356380e8 main
[junit4:junit4]   2> 23149 T928 C31 P13969 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 23149 T894 C28 P32313 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 23149 T923 oasc.JmxMonitoredMap.put WARNING Failed to register info bean: searcher javax.management.InstanceAlreadyExistsException: solr/collection1:type=searcher,id=org.apache.solr.search.SolrIndexSearcher
[junit4:junit4]   2> 	at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:467)
[junit4:junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.internal_addObject(DefaultMBeanServerInterceptor.java:1520)
[junit4:junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:986)
[junit4:junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:938)
[junit4:junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:330)
[junit4:junit4]   2> 	at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:517)
[junit4:junit4]   2> 	at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:140)
[junit4:junit4]   2> 	at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:51)
[junit4:junit4]   2> 	at org.apache.solr.search.SolrIndexSearcher.register(SolrIndexSearcher.java:239)
[junit4:junit4]   2> 	at org.apache.solr.core.SolrCore.registerSearcher(SolrCore.java:1763)
[junit4:junit4]   2> 	at org.apache.solr.core.SolrCore.access$000(SolrCore.java:129)
[junit4:junit4]   2> 	at org.apache.solr.core.SolrCore$7.call(SolrCore.java:1640)
[junit4:junit4]   2> 	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
[junit4:junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
[junit4:junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
[junit4:junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4:junit4]   2> 	at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 
[junit4:junit4]   2> 23149 T942 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@75a473fb main{StandardDirectoryReader(segments_4:1:nrt)}
[junit4:junit4]   2> 23149 T911 C29 P41673 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 23149 T906 oasc.JmxMonitoredMap.put WARNING Failed to register info bean: searcher javax.management.InstanceAlreadyExistsException: solr/collection1:type=searcher,id=org.apache.solr.search.SolrIndexSearcher
[junit4:junit4]   2> 	at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:467)
[junit4:junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.internal_addObject(DefaultMBeanServerInterceptor.java:1520)
[junit4:junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:986)
[junit4:junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:938)
[junit4:junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:330)
[junit4:junit4]   2> 	at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:517)
[junit4:junit4]   2> 	at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:140)
[junit4:junit4]   2> 	at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:51)
[junit4:junit4]   2> 	at org.apache.solr.search.SolrIndexSearcher.register(SolrIndexSearcher.java:239)
[junit4:junit4]   2> 	at org.apache.solr.core.SolrCore.registerSearcher(SolrCore.java:1763)
[junit4:junit4]   2> 	at org.apache.solr.core.SolrCore.access$000(SolrCore.java:129)
[junit4:junit4]   2> 	at org.apache.solr.core.SolrCore$7.call(SolrCore.java:1640)
[junit4:junit4]   2> 	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
[junit4:junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
[junit4:junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
[junit4:junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4:junit4]   2> 	at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 
[junit4:junit4]   2> 23150 T928 C31 P13969 /update {waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 4
[junit4:junit4]   2> 23150 T923 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@e89c98 main{StandardDirectoryReader(segments_3:1:nrt)}
[junit4:junit4]   2> 23150 T906 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@356380e8 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 23151 T911 C29 P41673 /update {waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 5
[junit4:junit4]   2> 23151 T894 C28 P32313 /update {waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 5
[junit4:junit4]   2> 23151 T881 C30 P55618 /update {waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 15
[junit4:junit4]   2> 23152 T846 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 23155 T879 C30 P55618 REQ /select {tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 23157 T895 C28 P32313 REQ /select {tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 23158 T912 C29 P41673 REQ /select {tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0 
[junit4:junit4]   2> 23160 T929 C31 P13969 REQ /select {tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0 
[junit4:junit4]   2> 25166 T858 C32 P60654 oass.SolrIndexSearcher.<init> Opening Searcher@5807cdd8 realtime
[junit4:junit4]   2> 25166 T858 C32 P60654 /update {wt=javabin&version=2} {deleteByQuery=*:* (-1429475409183899648)} 0 2
[junit4:junit4]   2> 25175 T928 C31 P13969 /update {update.distrib=FROMLEADER&_version_=-1429475409190191104&update.from=http://127.0.0.1:55618/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1429475409190191104)} 0 2
[junit4:junit4]   2> 25175 T894 C28 P32313 /update {update.distrib=FROMLEADER&_version_=-1429475409190191104&update.from=http://127.0.0.1:55618/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1429475409190191104)} 0 2
[junit4:junit4]   2> 25175 T911 C29 P41673 /update {update.distrib=FROMLEADER&_version_=-1429475409190191104&update.from=http://127.0.0.1:55618/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1429475409190191104)} 0 2
[junit4:junit4]   2> 25176 T881 C30 P55618 /update {wt=javabin&version=2} {deleteByQuery=*:* (-1429475409190191104)} 0 7
[junit4:junit4]   2> 25183 T858 C32 P60654 /update {wt=javabin&version=2} {add=[0 (1429475409200676864)]} 0 4
[junit4:junit4]   2> 25196 T894 C28 P32313 /update {distrib.from=http://127.0.0.1:55618/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1429475409208016896)]} 0 4
[junit4:junit4]   2> 25196 T928 C31 P13969 /update {distrib.from=http://127.0.0.1:55618/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1429475409208016896)]} 0 4
[junit4:junit4]   2> 25196 T911 C29 P41673 /update {distrib.from=http://127.0.0.1:55618/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1429475409208016896)]} 0 3
[junit4:junit4]   2> 25197 T881 C30 P55618 /update {wt=javabin&version=2} {add=[0 (1429475409208016896)]} 0 11
[junit4:junit4]   2> 25203 T858 C32 P60654 /update {wt=javabin&version=2} {add=[1 (1429475409222696960)]} 0 3
[junit4:junit4]   2> 25214 T894 C28 P32313 /update {distrib.from=http://127.0.0.1:55618/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1429475409228988416)]} 0 3
[junit4:junit4]   2> 25214 T928 C31 P13969 /update {distrib.from=http://127.0.0.1:55618/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1429475409228988416)]} 0 2
[junit4:junit4]   2> 25214 T911 C29 P41673 /update {distrib.from=http://127.0.0.1:55618/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1429475409228988416)]} 0 3
[junit4:junit4]   2> 25221 T881 C30 P55618 /update {wt=javabin&version=2} {add=[1 (1429475409228988416)]} 0 15
[junit4:junit4]   2> 25227 T858 C32 P60654 /update {wt=javabin&version=2} {add=[2 (1429475409247862784)]} 0 3
[junit4:junit4]   2> 25238 T928 C31 P13969 /update {distrib.from=http://127.0.0.1:55618/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[2 (1429475409254154240)]} 0 3
[junit4:junit4]   2> 25238 T911 C29 P41673 /update {distrib.from=http://127.0.0.1:55618/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[2 (1429475409254154240)]} 0 3
[junit4:junit4]   2> 25239 T881 C30 P55618 /update {test.distrib.skip.servers=http://127.0.0.1:32313/collection1/&wt=javabin&version=2} {add=[2 (1429475409254154240)]} 0 9
[junit4:junit4]   2> 25245 T858 C32 P60654 /update {wt=javabin&version=2} {add=[3 (1429475409266737152)]} 0 3
[junit4:junit4]   2> 25256 T928 C31 P13969 /update {distrib.from=http://127.0.0.1:55618/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[3 (1429475409273028608)]} 0 3
[junit4:junit4]   2> 25257 T881 C30 P55618 /update {test.distrib.skip.servers=http://127.0.0.1:32313/collection1/&test.distrib.skip.servers=http://127.0.0.1:41673/collection1/&wt=javabin&version=2} {add=[3 (1429475409273028608)]} 0 9
[junit4:junit4]   2> 25259 T858 C32 P60654 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 25268 T858 C32 P60654 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4e57449e lockFactory=org.apache.lucene.store.NativeFSLockFactory@4c156b4b),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4e57449e lockFactory=org.apache.lucene.store.NativeFSLockFactory@4c156b4b),segFN=segments_3,generation=3,filenames=[_0_NestedPulsing_0.doc, _0_Lucene41WithOrds_0.tib, _0.fnm, _0_Lucene41WithOrds_0.doc, _0_Lucene41WithOrds_0.tii, _0_Lucene41WithOrds_0.pos, _0.nvm, _0_NestedPulsing_0.tip, _0_SimpleText_0.dat, _0_NestedPulsing_0.tim, _0_NestedPulsing_0.pos, _0_Direct_0.doc, _0_Pulsing41_0.pos, _0_Pulsing41_0.doc, _0_Pulsing41_0.tim, _0_Pulsing41_0.tip, _0.nvd, _0_Direct_0.pos, _0_Direct_0.tip, _0.fdx, _0.si, _0_Direct_0.tim, _0.fdt, segments_3]
[junit4:junit4]   2> 25269 T858 C32 P60654 oasc.SolrDeletionPolicy.updateCommits newest commit = 3[_0_NestedPulsing_0.doc, _0_Lucene41WithOrds_0.tib, _0.fnm, _0_Lucene41WithOrds_0.doc, _0_Lucene41WithOrds_0.tii, _0_Lucene41WithOrds_0.pos, _0.nvm, _0_NestedPulsing_0.tip, _0_SimpleText_0.dat, _0_NestedPulsing_0.tim, _0_NestedPulsing_0.pos, _0_Direct_0.doc, _0_Pulsing41_0.pos, _0_Pulsing41_0.doc, _0_Pulsing41_0.tim, _0_Pulsing41_0.tip, _0.nvd, _0_Direct_0.pos, _0_Direct_0.tip, _0.fdx, _0.si, _0_Direct_0.tim, _0.fdt, segments_3]
[junit4:junit4]   2> 25300 T858 C32 P60654 oass.SolrIndexSearcher.<init> Opening Searcher@1f74e74b main
[junit4:junit4]   2> 25301 T858 C32 P60654 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 25302 T872 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1f74e74b main{StandardDirectoryReader(segments_3:3:nrt _0(5.0):C4)}
[junit4:junit4]   2> 25303 T858 C32 P60654 /update {waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 44
[junit4:junit4]   2> 25307 T881 C30 P55618 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 25318 T881 C30 P55618 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=5
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2d6d4204 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7a21bd99),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2d6d4204 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7a21bd99),segFN=segments_3,generation=3,filenames=[segments_3]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2d6d4204 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7a21bd99),segFN=segments_4,generation=4,filenames=[segments_4]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2d6d4204 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7a21bd99),segFN=segments_5,generation=5,filenames=[segments_5]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2d6d4204 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7a21bd99),segFN=segments_6,generation=6,filenames=[_0_NestedPulsing_0.doc, _0_Lucene41WithOrds_0.tib, _0.fnm, _0_Lucene41WithOrds_0.doc, _0_Lucene41WithOrds_0.tii, _0_Lucene41WithOrds_0.pos, _0.nvm, _0_NestedPulsing_0.tip, _0_SimpleText_0.dat, _0_NestedPulsing_0.tim, _0_NestedPulsing_0.pos, _0_Direct_0.doc, _0_Pulsing41_0.pos, _0_Pulsing41_0.doc, _0_Pulsing41_0.tim, _0_Pulsing41_0.tip, _0.nvd, _0_Direct_0.pos, _0_Direct_0.tip, _0.fdx, _0.si, _0_Direct_0.tim, segments_6, _0.fdt]
[junit4:junit4]   2> 25319 T881 C30 P55618 oasc.SolrDeletionPolicy.updateCommits newest commit = 6[_0_NestedPulsing_0.doc, _0_Lucene41WithOrds_0.tib, _0.fnm, _0_Lucene41WithOrds_0.doc, _0_Lucene41WithOrds_0.tii, _0_Lucene41WithOrds_0.pos, _0.nvm, _0_NestedPulsing_0.tip, _0_SimpleText_0.dat, _0_NestedPulsing_0.tim, _0_NestedPulsing_0.pos, _0_Direct_0.doc, _0_Pulsing41_0.pos, _0_Pulsing41_0.doc, _0_Pulsing41_0.tim, _0_Pulsing41_0.tip, _0.nvd, _0_Direct_0.pos, _0_Direct_0.tip, _0.fdx, _0.si, _0_Direct_0.tim, segments_6, _0.fdt]
[junit4:junit4]   2> 25324 T881 C30 P55618 oass.SolrIndexSearcher.<init> Opening Searcher@3fec7107 main
[junit4:junit4]   2> 25325 T881 C30 P55618 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 25326 T890 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3fec7107 main{StandardDirectoryReader(segments_6:3:nrt _0(5.0):C4)}
[junit4:junit4]   2> 25326 T881 C30 P55618 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:32313/collection1/, StdNode: http://127.0.0.1:41673/collection1/, StdNode: http://127.0.0.1:13969/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4]   2> 25330 T894 C28 P32313 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 25330 T911 C29 P41673 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 25337 T928 C31 P13969 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 25341 T894 C28 P32313 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2909423b lockFactory=org.apache.lucene.store.NativeFSLockFactory@4d708e32),segFN=segments_3,generation=3,filenames=[segments_3]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2909423b lockFactory=org.apache.lucene.store.NativeFSLockFactory@4d708e32),segFN=segments_4,generation=4,filenames=[_0_NestedPulsing_0.doc, _0_Lucene41WithOrds_0.tib, _0.fnm, _0_Lucene41WithOrds_0.doc, _0_Lucene41WithOrds_0.tii, _0_Lucene41WithOrds_0.pos, _0.nvm, _0_NestedPulsing_0.tip, _0_SimpleText_0.dat, _0_NestedPulsing_0.tim, _0_NestedPulsing_0.pos, _0_Direct_0.doc, _0_Pulsing41_0.pos, _0_Pulsing41_0.doc, _0_Pulsing41_0.tim, _0_Pulsing41_0.tip, _0.nvd, _0_Direct_0.pos, _0_Direct_0.tip, _0.fdx, _0.si, _0_Direct_0.tim, _0.fdt, segments_4]
[junit4:junit4]   2> 25342 T894 C28 P32313 oasc.SolrDeletionPolicy.updateCommits newest commit = 4[_0_NestedPulsing_0.doc, _0_Lucene41WithOrds_0.tib, _0.fnm, _0_Lucene41WithOrds_0.doc, _0_Lucene41WithOrds_0.tii, _0_Lucene41WithOrds_0.pos, _0.nvm, _0_NestedPulsing_0.tip, _0_SimpleText_0.dat, _0_NestedPulsing_0.tim, _0_NestedPulsing_0.pos, _0_Direct_0.doc, _0_Pulsing41_0.pos, _0_Pulsing41_0.doc, _0_Pulsing41_0.tim, _0_Pulsing41_0.tip, _0.nvd, _0_Direct_0.pos, _0_Direct_0.tip, _0.fdx, _0.si, _0_Direct_0.tim, _0.fdt, segments_4]
[junit4:junit4]   2> 25342 T911 C29 P41673 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2ad3c1ec lockFactory=org.apache.lucene.store.NativeFSLockFactory@2c332d2a),segFN=segments_4,generation=4,filenames=[segments_4]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2ad3c1ec lockFactory=org.apache.lucene.store.NativeFSLockFactory@2c332d2a),segFN=segments_5,generation=5,filenames=[_0_NestedPulsing_0.doc, _0_Lucene41WithOrds_0.tib, _0.fnm, _0_Lucene41WithOrds_0.doc, _0_Lucene41WithOrds_0.tii, _0_Lucene41WithOrds_0.pos, _0.nvm, _0_NestedPulsing_0.tip, _0_SimpleText_0.dat, _0_NestedPulsing_0.tim, _0_NestedPulsing_0.pos, _0_Direct_0.doc, _0_Pulsing41_0.pos, _0_Pulsing41_0.doc, _0_Pulsing41_0.tim, _0_Pulsing41_0.tip, _0.nvd, _0_Direct_0.pos, _0_Direct_0.tip, _0.fdx, _0.si, segments_5, _0_Direct_0.tim, _0.fdt]
[junit4:junit4]   2> 25343 T911 C29 P41673 oasc.SolrDeletionPolicy.updateCommits newest commit = 5[_0_NestedPulsing_0.doc, _0_Lucene41WithOrds_0.tib, _0.fnm, _0_Lucene41WithOrds_0.doc, _0_Lucene41WithOrds_0.tii, _0_Lucene41WithOrds_0.pos, _0.nvm, _0_NestedPulsing_0.tip, _0_SimpleText_0.dat, _0_NestedPulsing_0.tim, _0_NestedPulsing_0.pos, _0_Direct_0.doc, _0_Pulsing41_0.pos, _0_Pulsing41_0.doc, _0_Pulsing41_0.tim, _0_Pulsing41_0.tip, _0.nvd, _0_Direct_0.pos, _0_Direct_0.tip, _0.fdx, _0.si, segments_5, _0_Direct_0.tim, _0.fdt]
[junit4:junit4]   2> 25346 T894 C28 P32313 oass.SolrIndexSearcher.<init> Opening Searcher@18102ffa main
[junit4:junit4]   2> 25347 T894 C28 P32313 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 25347 T928 C31 P13969 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@492d49e0 lockFactory=org.apache.lucene.store.NativeFSLockFactory@713f6e85),segFN=segments_5,generation=5,filenames=[segments_5]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@492d49e0 lockFactory=org.apache.lucene.store.NativeFSLockFactory@713f6e85),segFN=segments_6,generation=6,filenames=[_0_NestedPulsing_0.doc, _0_Lucene41WithOrds_0.tib, _0.fnm, _0_Lucene41WithOrds_0.doc, _0_Lucene41WithOrds_0.tii, _0_Lucene41WithOrds_0.pos, _0.nvm, _0_NestedPulsing_0.tip, _0_SimpleText_0.dat, _0_NestedPulsing_0.tim, _0_NestedPulsing_0.pos, _0_Direct_0.doc, _0_Pulsing41_0.pos, _0_Pulsing41_0.doc, _0_Pulsing41_0.tim, _0_Pulsing41_0.tip, _0.nvd, _0_Direct_0.pos, _0_Direct_0.tip, _0.fdx, _0.si, _0_Direct_0.tim, segments_6, _0.fdt]
[junit4:junit4]   2> 25347 T906 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@18102ffa main{StandardDirectoryReader(segments_4:3:nrt _0(5.0):C2)}
[junit4:junit4]   2> 25347 T928 C31 P13969 oasc.SolrDeletionPolicy.updateCommits newest commit = 6[_0_NestedPulsing_0.doc, _0_Lucene41WithOrds_0.tib, _0.fnm, _0_Lucene41WithOrds_0.doc, _0_Lucene41WithOrds_0.tii, _0_Lucene41WithOrds_0.pos, _0.nvm, _0_NestedPulsing_0.tip, _0_SimpleText_0.dat, _0_NestedPulsing_0.tim, _0_NestedPulsing_0.pos, _0_Direct_0.doc, _0_Pulsing41_0.pos, _0_Pulsing41_0.doc, _0_Pulsing41_0.tim, _0_Pulsing41_0.tip, _0.nvd, _0_Direct_0.pos, _0_Direct_0.tip, _0.fdx, _0.si, _0_Direct_0.tim, segments_6, _0.fdt]
[junit4:junit4]   2> 25348 T894 C28 P32313 /update {waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 18
[junit4:junit4]   2> 25349 T911 C29 P41673 oass.SolrIndexSearcher.<init> Opening Searcher@529d8124 main
[junit4:junit4]   2> 25350 T911 C29 P41673 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 25350 T923 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@529d8124 main{StandardDirectoryReader(segments_5:3:nrt _0(5.0):C3)}
[junit4:junit4]   2> 25351 T911 C29 P41673 /update {waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 21
[junit4:junit4]   2> 25351 T928 C31 P13969 oass.SolrIndexSearcher.<init> Opening Searcher@6bc137c7 main
[junit4:junit4]   2> 25352 T928 C31 P13969 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 25352 T942 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6bc137c7 main{StandardDirectoryReader(segments_6:3:nrt _0(5.0):C4)}
[junit4:junit4]   2> 25353 T928 C31 P13969 /update {waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 16
[junit4:junit4]   2> 25353 T881 C30 P55618 /update {waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 47
[junit4:junit4]   2> 25354 T846 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):230
[junit4:junit4]   2> 25355 T846 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 25356 T846 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4]   2> 25360 T879 C30 P55618 REQ /select {tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=1 
[junit4:junit4]   2> 25363 T895 C28 P32313 REQ /select {tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=2 status=0 QTime=0 
[junit4:junit4]   2> 25366 T912 C29 P41673 REQ /select {tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=3 status=0 QTime=0 
[junit4:junit4]   2> 25369 T929 C31 P13969 REQ /select {tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0 
[junit4:junit4]   2> 25375 T846 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 25394 T913 oasha.CollectionsHandler.handleSyncShardAction Syncing shard : shard=shard1&action=SYNCSHARD&collection=collection1&wt=javabin&version=2
[junit4:junit4]   2> 25395 T913 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 25408 T884 oasha.CoreAdminHandler.handleRequestSyncAction I have been requested to sync up my shard
[junit4:junit4]   2> 25408 T884 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:55618/collection1/
[junit4:junit4]   2> 25409 T884 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:55618 START replicas=[http://127.0.0.1:32313/collection1/, http://127.0.0.1:41673/collection1/, http://127.0.0.1:13969/collection1/] nUpda

[...truncated too long message...]


[junit4:junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=SyncSliceTest -Dtests.method=testDistribSearch -Dtests.seed=D2139F8170B5DA86 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ja_JP -Dtests.timezone=Africa/Blantyre -Dtests.file.encoding=ISO8859-1
[junit4:junit4] FAILURE  140s J0 | SyncSliceTest.testDistribSearch <<<
[junit4:junit4]    > Throwable #1: java.lang.AssertionError: Test Setup Failure: shard1 should have just been set up to be inconsistent - but it's still consistent. Leader:http://127.0.0.1:32313/collection1skip list:[CloudJettyRunner [url=http://127.0.0.1:13969/collection1], CloudJettyRunner [url=http://127.0.0.1:41673/collection1]]
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([D2139F8170B5DA86:53F5119907EABABA]:0)
[junit4:junit4]    > 	at org.junit.Assert.fail(Assert.java:93)
[junit4:junit4]    > 	at org.junit.Assert.assertTrue(Assert.java:43)
[junit4:junit4]    > 	at org.junit.Assert.assertNotNull(Assert.java:526)
[junit4:junit4]    > 	at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:212)
[junit4:junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:806)
[junit4:junit4]    > 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit4:junit4]    > 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
[junit4:junit4]    > 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit4:junit4]    > 	at java.lang.reflect.Method.invoke(Method.java:616)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
[junit4:junit4]    > 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
[junit4:junit4]    > 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
[junit4:junit4]    > 	at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 139781 T846 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4]   2> 139785 T845 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 2 leaked thread(s).
[junit4:junit4]   2> 139822 T921 oascc.ZkStateReader$3.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 139822 T921 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 139822 T921 oascc.ZkStateReader$2.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 139822 T921 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 139904 T277 oasc.OverseerCollectionProcessor.run WARNING Overseer cannot talk to ZK
[junit4:junit4]   2> 140492 T849 oazs.SessionTrackerImpl.run SessionTrackerImpl exited loop!
[junit4:junit4]   2> NOTE: test params are: codec=Lucene42: {range_facet_l=Pulsing41(freqCutoff=1 minBlockSize=85 maxBlockSize=177), text=PostingsFormat(name=NestedPulsing), _version_=Pulsing41(freqCutoff=1 minBlockSize=85 maxBlockSize=177), rnd_b=PostingsFormat(name=Lucene41WithOrds), intDefault=Pulsing41(freqCutoff=1 minBlockSize=85 maxBlockSize=177), id=PostingsFormat(name=Lucene41WithOrds), timestamp=Pulsing41(freqCutoff=1 minBlockSize=85 maxBlockSize=177), a_t=Pulsing41(freqCutoff=1 minBlockSize=85 maxBlockSize=177), range_facet_sl=PostingsFormat(name=NestedPulsing), range_facet_si=PostingsFormat(name=Direct), other_tl1=Pulsing41(freqCutoff=1 minBlockSize=85 maxBlockSize=177), multiDefault=PostingsFormat(name=Direct), a_si=PostingsFormat(name=Direct)}, docValues:{timestamp=DocValuesFormat(name=SimpleText)}, sim=RandomSimilarityProvider(queryNorm=true,coord=yes): {}, locale=ja_JP, timezone=Africa/Blantyre
[junit4:junit4]   2> NOTE: FreeBSD 9.0-RELEASE amd64/Sun Microsystems Inc. 1.6.0_32 (64-bit)/cpus=16,threads=5,free=136286672,total=326172672
[junit4:junit4]   2> NOTE: All tests run in this JVM: [CSVRequestHandlerTest, JsonLoaderTest, FieldMutatingUpdateProcessorTest, TestSolrQueryParser, QueryElevationComponentTest, TestXIncludeConfig, CoreAdminHandlerTest, TermsComponentTest, FileBasedSpellCheckerTest, TestStressRecovery, BasicFunctionalityTest, SolrTestCaseJ4Test, LegacyHTMLStripCharFilterTest, StatelessScriptUpdateProcessorFactoryTest, OverseerTest, QueryParsingTest, SolrCoreTest, ChaosMonkeyNothingIsSafeTest, PreAnalyzedFieldTest, BadComponentTest, SuggesterFSTTest, ShardRoutingTest, TestFastWriter, UUIDFieldTest, TestDocSet, SoftAutoCommitTest, PrimUtilsTest, SyncSliceTest]
[junit4:junit4] Completed on J0 in 140.57s, 1 test, 1 failure <<< FAILURES!

[...truncated 639 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/build.xml:381: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/build.xml:361: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build.xml:183: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/common-build.xml:449: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/lucene/common-build.xml:1213: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/lucene/common-build.xml:877: There were test failures: 271 suites, 1142 tests, 1 failure, 27 ignored (8 assumptions)

Total time: 53 minutes 4 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-Tests-trunk-Java6 - Build # 16001 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-trunk-Java6/16001/

1 tests failed.
REGRESSION:  org.apache.solr.cloud.BasicDistributedZk2Test.testDistribSearch

Error Message:
No registered leader was found, collection:collection1 slice:shard1

Stack Trace:
org.apache.solr.common.SolrException: No registered leader was found, collection:collection1 slice:shard1
	at __randomizedtesting.SeedInfo.seed([B04836D63D1D8C3D:31AEB8CE4A42EC01]:0)
	at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:484)
	at org.apache.solr.cloud.BasicDistributedZk2Test.brindDownShardIndexSomeDocsAndRecover(BasicDistributedZk2Test.java:291)
	at org.apache.solr.cloud.BasicDistributedZk2Test.doTest(BasicDistributedZk2Test.java:116)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:806)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:679)




Build Log:
[...truncated 8610 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.BasicDistributedZk2Test
[junit4:junit4]   2> 0 T2117 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4:junit4]   2> 5 T2117 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4]   2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1363227524745
[junit4:junit4]   2> 6 T2117 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 7 T2118 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4]   2> 11 T2118 oazs.ZooKeeperServer.setTickTime tickTime set to 1000
[junit4:junit4]   2> 11 T2118 oazs.ZooKeeperServer.setMinSessionTimeout minSessionTimeout set to -1
[junit4:junit4]   2> 11 T2118 oazs.ZooKeeperServer.setMaxSessionTimeout maxSessionTimeout set to -1
[junit4:junit4]   2> 12 T2118 oazs.NIOServerCnxnFactory.configure binding to port 0.0.0.0/0.0.0.0:0
[junit4:junit4]   2> 14 T2118 oazsp.FileTxnSnapLog.save Snapshotting: 0x0 to /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-1363227524744/zookeeper/server1/data/version-2/snapshot.0
[junit4:junit4]   2> 108 T2117 oasc.ZkTestServer.run start zk server on port:16556
[junit4:junit4]   2> 108 T2117 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:16556 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@6f536152
[junit4:junit4]   2> 109 T2117 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 110 T2123 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:16556. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 110 T2123 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:16556, initiating session
[junit4:junit4]   2> 111 T2119 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:51111
[junit4:junit4]   2> 111 T2119 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:51111
[junit4:junit4]   2> 111 T2121 oazsp.FileTxnLog.append Creating new log file: log.1
[junit4:junit4]   2> 114 T2121 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d66b13e920000 with negotiated timeout 10000 for client /140.211.11.196:51111
[junit4:junit4]   2> 114 T2123 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:16556, sessionid = 0x13d66b13e920000, negotiated timeout = 10000
[junit4:junit4]   2> 115 T2124 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6f536152 name:ZooKeeperConnection Watcher:127.0.0.1:16556 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 115 T2117 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 116 T2117 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4]   2> 120 T2122 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d66b13e920000
[junit4:junit4]   2> 129 T2124 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 129 T2117 oaz.ZooKeeper.close Session: 0x13d66b13e920000 closed
[junit4:junit4]   2> 131 T2119 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:51111 which had sessionid 0x13d66b13e920000
[junit4:junit4]   2> 131 T2117 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:16556/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@4abff439
[junit4:junit4]   2> 132 T2117 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 132 T2125 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:16556. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 133 T2125 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:16556, initiating session
[junit4:junit4]   2> 133 T2119 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:34047
[junit4:junit4]   2> 133 T2119 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:34047
[junit4:junit4]   2> 135 T2121 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d66b13e920001 with negotiated timeout 10000 for client /140.211.11.196:34047
[junit4:junit4]   2> 135 T2125 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:16556, sessionid = 0x13d66b13e920001, negotiated timeout = 10000
[junit4:junit4]   2> 135 T2126 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4abff439 name:ZooKeeperConnection Watcher:127.0.0.1:16556/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 135 T2117 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 136 T2117 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4]   2> 146 T2117 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4]   2> 150 T2117 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4]   2> 153 T2117 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4]   2> 162 T2117 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 163 T2117 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 177 T2117 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4]   2> 178 T2117 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4]   2> 283 T2117 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4]   2> 284 T2117 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4]   2> 287 T2117 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4]   2> 288 T2117 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4]   2> 292 T2117 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4]   2> 292 T2117 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4]   2> 302 T2117 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 302 T2117 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 305 T2117 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 306 T2117 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 309 T2117 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 310 T2117 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 313 T2117 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4]   2> 314 T2117 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4]   2> 317 T2122 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d66b13e920001
[junit4:junit4]   2> 318 T2126 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 318 T2119 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:34047 which had sessionid 0x13d66b13e920001
[junit4:junit4]   2> 318 T2117 oaz.ZooKeeper.close Session: 0x13d66b13e920001 closed
[junit4:junit4]   2> 534 T2117 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 538 T2117 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:14029
[junit4:junit4]   2> 539 T2117 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 539 T2117 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 540 T2117 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1363227525058
[junit4:junit4]   2> 540 T2117 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1363227525058/solr.xml
[junit4:junit4]   2> 540 T2117 oasc.CoreContainer.<init> New CoreContainer 1644733137
[junit4:junit4]   2> 541 T2117 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1363227525058/'
[junit4:junit4]   2> 542 T2117 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1363227525058/'
[junit4:junit4]   2> 600 T2117 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 601 T2117 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 601 T2117 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 602 T2117 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 602 T2117 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 603 T2117 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 603 T2117 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 604 T2117 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 604 T2117 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 604 T2117 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 627 T2117 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 647 T2117 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:16556/solr
[junit4:junit4]   2> 647 T2117 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 648 T2117 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:16556 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@769fc8af
[junit4:junit4]   2> 649 T2117 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 649 T2136 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:16556. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 650 T2136 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:16556, initiating session
[junit4:junit4]   2> 651 T2119 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:49136
[junit4:junit4]   2> 651 T2119 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:49136
[junit4:junit4]   2> 653 T2121 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d66b13e920002 with negotiated timeout 20000 for client /140.211.11.196:49136
[junit4:junit4]   2> 653 T2136 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:16556, sessionid = 0x13d66b13e920002, negotiated timeout = 20000
[junit4:junit4]   2> 653 T2137 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@769fc8af name:ZooKeeperConnection Watcher:127.0.0.1:16556 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 653 T2117 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 655 T2122 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d66b13e920002
[junit4:junit4]   2> 663 T2137 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 663 T2119 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:49136 which had sessionid 0x13d66b13e920002
[junit4:junit4]   2> 663 T2117 oaz.ZooKeeper.close Session: 0x13d66b13e920002 closed
[junit4:junit4]   2> 664 T2117 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 676 T2117 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:16556/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@66782c49
[junit4:junit4]   2> 677 T2117 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 677 T2138 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:16556. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 678 T2138 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:16556, initiating session
[junit4:junit4]   2> 678 T2119 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:40252
[junit4:junit4]   2> 678 T2119 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:40252
[junit4:junit4]   2> 681 T2121 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d66b13e920003 with negotiated timeout 20000 for client /140.211.11.196:40252
[junit4:junit4]   2> 681 T2138 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:16556, sessionid = 0x13d66b13e920003, negotiated timeout = 20000
[junit4:junit4]   2> 681 T2139 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@66782c49 name:ZooKeeperConnection Watcher:127.0.0.1:16556/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 681 T2117 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 684 T2122 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d66b13e920003 type:create cxid:0x2 zxid:0x1a txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 686 T2117 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4]   2> 695 T2117 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:14029_
[junit4:junit4]   2> 696 T2122 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d66b13e920003 type:delete cxid:0x8 zxid:0x1c txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:14029_ Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:14029_
[junit4:junit4]   2> 697 T2117 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:14029_
[junit4:junit4]   2> 700 T2117 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4]   2> 713 T2122 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d66b13e920003 type:delete cxid:0x16 zxid:0x21 txntype:-1 reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4]   2> 719 T2117 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 722 T2117 oasc.Overseer.start Overseer (id=89340479062278147-127.0.0.1:14029_-n_0000000000) starting
[junit4:junit4]   2> 723 T2122 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d66b13e920003 type:create cxid:0x1b zxid:0x23 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 725 T2122 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d66b13e920003 type:create cxid:0x1c zxid:0x24 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 733 T2122 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d66b13e920003 type:create cxid:0x1d zxid:0x25 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 734 T2122 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d66b13e920003 type:create cxid:0x1e zxid:0x26 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 741 T2141 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4]   2> 742 T2117 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4]   2> 752 T2117 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4]   2> 755 T2117 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 758 T2140 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 765 T2142 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1363227525058/collection1
[junit4:junit4]   2> 765 T2142 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4]   2> 766 T2142 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 766 T2142 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4]   2> 768 T2142 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1363227525058/collection1/'
[junit4:junit4]   2> 770 T2142 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1363227525058/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 770 T2142 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1363227525058/collection1/lib/README' to classloader
[junit4:junit4]   2> 823 T2142 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 893 T2142 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 995 T2142 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 1001 T2142 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 1667 T2142 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 1679 T2142 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 1684 T2142 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 1710 T2142 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 1716 T2142 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 1722 T2142 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 1724 T2142 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 1724 T2142 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 1724 T2142 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 1726 T2142 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 1726 T2142 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 1726 T2142 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 1727 T2142 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1363227525058/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363227524744/control/data/
[junit4:junit4]   2> 1727 T2142 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5c09624
[junit4:junit4]   2> 1728 T2142 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 1728 T2142 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363227524744/control/data forceNew: false
[junit4:junit4]   2> 1729 T2142 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1363227524744/control/data/index/
[junit4:junit4]   2> 1729 T2142 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1363227524744/control/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 1729 T2142 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363227524744/control/data/index forceNew: false
[junit4:junit4]   2> 1733 T2142 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@fdd241a lockFactory=org.apache.lucene.store.NativeFSLockFactory@69a1c8d6; maxCacheMB=0.578125 maxMergeSizeMB=0.6806640625)),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 1733 T2142 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 1736 T2142 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 1736 T2142 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 1737 T2142 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 1737 T2142 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 1738 T2142 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 1738 T2142 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 1738 T2142 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 1739 T2142 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 1739 T2142 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 1747 T2142 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 1755 T2142 oass.SolrIndexSearcher.<init> Opening Searcher@bd05853 main
[junit4:junit4]   2> 1755 T2142 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363227524744/control/data/tlog
[junit4:junit4]   2> 1757 T2142 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 1757 T2142 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 1760 T2143 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@bd05853 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 1761 T2142 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 1761 T2142 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 1762 T2122 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d66b13e920003 type:create cxid:0x55 zxid:0x2a txntype:-1 reqpath:n/a Error Path:/solr/overseer/queue Error:KeeperErrorCode = NoNode for /solr/overseer/queue
[junit4:junit4]   2> 2263 T2140 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2264 T2140 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"1",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"control_collection",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:14029_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:14029"}
[junit4:junit4]   2> 2264 T2140 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4]   2> 2264 T2140 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 2265 T2122 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d66b13e920003 type:create cxid:0x5d zxid:0x2d txntype:-1 reqpath:n/a Error Path:/solr/overseer/queue-work Error:KeeperErrorCode = NoNode for /solr/overseer/queue-work
[junit4:junit4]   2> 2281 T2139 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 2798 T2142 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 2798 T2142 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:14029 collection:control_collection shard:shard1
[junit4:junit4]   2> 2799 T2142 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4]   2> 2817 T2122 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d66b13e920003 type:delete cxid:0x77 zxid:0x37 txntype:-1 reqpath:n/a Error Path:/solr/collections/control_collection/leaders Error:KeeperErrorCode = NoNode for /solr/collections/control_collection/leaders
[junit4:junit4]   2> 2818 T2142 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 2819 T2122 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d66b13e920003 type:create cxid:0x78 zxid:0x38 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 2827 T2142 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 2828 T2142 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 2828 T2142 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:14029/collection1/
[junit4:junit4]   2> 2828 T2142 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 2828 T2142 oasc.SyncStrategy.syncToMe http://127.0.0.1:14029/collection1/ has no replicas
[junit4:junit4]   2> 2829 T2142 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:14029/collection1/
[junit4:junit4]   2> 2829 T2142 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4]   2> 2833 T2122 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d66b13e920003 type:create cxid:0x82 zxid:0x3c txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 3795 T2140 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 3844 T2139 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 3906 T2142 oasc.ZkController.register We are http://127.0.0.1:14029/collection1/ and leader is http://127.0.0.1:14029/collection1/
[junit4:junit4]   2> 3906 T2142 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:14029
[junit4:junit4]   2> 3906 T2142 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 3906 T2142 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 3907 T2142 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 3909 T2142 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 3910 T2117 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 3910 T2117 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 3911 T2117 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 3925 T2117 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4]   2> 3926 T2117 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:16556/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@442b26b
[junit4:junit4]   2> 3927 T2117 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 3927 T2145 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:16556. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 3928 T2145 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:16556, initiating session
[junit4:junit4]   2> 3928 T2119 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:32861
[junit4:junit4]   2> 3929 T2119 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:32861
[junit4:junit4]   2> 3930 T2121 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d66b13e920004 with negotiated timeout 10000 for client /140.211.11.196:32861
[junit4:junit4]   2> 3930 T2145 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:16556, sessionid = 0x13d66b13e920004, negotiated timeout = 10000
[junit4:junit4]   2> 3930 T2146 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@442b26b name:ZooKeeperConnection Watcher:127.0.0.1:16556/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 3930 T2117 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 3932 T2117 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 3934 T2117 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4]   2> 4133 T2117 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 4136 T2117 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:64400
[junit4:junit4]   2> 4136 T2117 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 4137 T2117 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 4137 T2117 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363227528673
[junit4:junit4]   2> 4138 T2117 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363227528673/solr.xml
[junit4:junit4]   2> 4138 T2117 oasc.CoreContainer.<init> New CoreContainer 1233374211
[junit4:junit4]   2> 4139 T2117 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363227528673/'
[junit4:junit4]   2> 4139 T2117 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363227528673/'
[junit4:junit4]   2> 4195 T2117 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 4195 T2117 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 4196 T2117 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 4196 T2117 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 4197 T2117 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 4197 T2117 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 4198 T2117 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 4198 T2117 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 4199 T2117 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 4199 T2117 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 4219 T2117 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 4243 T2117 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:16556/solr
[junit4:junit4]   2> 4244 T2117 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 4245 T2117 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:16556 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@55794086
[junit4:junit4]   2> 4246 T2117 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 4247 T2156 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:16556. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 4248 T2156 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:16556, initiating session
[junit4:junit4]   2> 4248 T2119 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:43415
[junit4:junit4]   2> 4249 T2119 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:43415
[junit4:junit4]   2> 4251 T2121 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d66b13e920005 with negotiated timeout 20000 for client /140.211.11.196:43415
[junit4:junit4]   2> 4251 T2156 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:16556, sessionid = 0x13d66b13e920005, negotiated timeout = 20000
[junit4:junit4]   2> 4252 T2157 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@55794086 name:ZooKeeperConnection Watcher:127.0.0.1:16556 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 4252 T2117 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 4253 T2122 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d66b13e920005
[junit4:junit4]   2> 4267 T2119 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:43415 which had sessionid 0x13d66b13e920005
[junit4:junit4]   2> 4267 T2157 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 4267 T2117 oaz.ZooKeeper.close Session: 0x13d66b13e920005 closed
[junit4:junit4]   2> 4268 T2117 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 4278 T2117 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:16556/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@45af4f83
[junit4:junit4]   2> 4279 T2117 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 4279 T2158 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:16556. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 4280 T2158 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:16556, initiating session
[junit4:junit4]   2> 4280 T2119 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:51793
[junit4:junit4]   2> 4281 T2119 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:51793
[junit4:junit4]   2> 4282 T2121 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d66b13e920006 with negotiated timeout 20000 for client /140.211.11.196:51793
[junit4:junit4]   2> 4282 T2158 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:16556, sessionid = 0x13d66b13e920006, negotiated timeout = 20000
[junit4:junit4]   2> 4282 T2159 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@45af4f83 name:ZooKeeperConnection Watcher:127.0.0.1:16556/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 4282 T2117 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 4284 T2122 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d66b13e920006 type:create cxid:0x1 zxid:0x4a txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 4285 T2122 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d66b13e920006 type:create cxid:0x2 zxid:0x4b txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 4294 T2117 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 5299 T2117 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:64400_
[junit4:junit4]   2> 5300 T2122 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d66b13e920006 type:delete cxid:0xb zxid:0x4c txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:64400_ Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:64400_
[junit4:junit4]   2> 5306 T2117 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:64400_
[junit4:junit4]   2> 5310 T2139 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 5310 T2146 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 5311 T2159 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 5312 T2139 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 5318 T2160 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363227528673/collection1
[junit4:junit4]   2> 5318 T2160 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 5319 T2160 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 5319 T2160 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 5320 T2160 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363227528673/collection1/'
[junit4:junit4]   2> 5321 T2160 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363227528673/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 5322 T2160 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363227528673/collection1/lib/README' to classloader
[junit4:junit4]   2> 5358 T2140 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 5359 T2140 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"1",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"control_collection",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:14029_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:14029"}
[junit4:junit4]   2> 5374 T2139 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 5374 T2146 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 5374 T2159 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 5381 T2160 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 5441 T2160 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 5542 T2160 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 5548 T2160 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 6251 T2160 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 6262 T2160 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 6265 T2160 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 6275 T2160 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 6278 T2160 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 6281 T2160 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 6282 T2160 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 6282 T2160 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 6282 T2160 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 6283 T2160 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 6283 T2160 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 6284 T2160 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 6284 T2160 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363227528673/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363227524744/jetty1/
[junit4:junit4]   2> 6284 T2160 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5c09624
[junit4:junit4]   2> 6284 T2160 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 6285 T2160 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363227524744/jetty1 forceNew: false
[junit4:junit4]   2> 6285 T2160 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1363227524744/jetty1/index/
[junit4:junit4]   2> 6285 T2160 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1363227524744/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 6285 T2160 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363227524744/jetty1/index forceNew: false
[junit4:junit4]   2> 6287 T2160 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@6809635b lockFactory=org.apache.lucene.store.NativeFSLockFactory@f40f91d; maxCacheMB=0.578125 maxMergeSizeMB=0.6806640625)),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 6288 T2160 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 6289 T2160 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 6289 T2160 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 6290 T2160 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 6290 T2160 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 6291 T2160 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 6291 T2160 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 6291 T2160 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 6291 T2160 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 6292 T2160 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 6296 T2160 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 6300 T2160 oass.SolrIndexSearcher.<init> Opening Searcher@5ee5f76f main
[junit4:junit4]   2> 6300 T2160 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363227524744/jetty1/tlog
[junit4:junit4]   2> 6301 T2160 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 6301 T2160 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 6310 T2161 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5ee5f76f main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 6311 T2160 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 6311 T2160 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 6879 T2140 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 6880 T2140 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"2",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:64400_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:64400"}
[junit4:junit4]   2> 6880 T2140 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 2
[junit4:junit4]   2> 6881 T2140 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 6933 T2146 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 6933 T2139 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 6933 T2159 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 7314 T2160 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 7314 T2160 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:64400 collection:collection1 shard:shard1
[junit4:junit4]   2> 7315 T2160 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 7344 T2122 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d66b13e920006 type:delete cxid:0x48 zxid:0x5c txntype:-1 reqpath:n/a Error Path:/solr/collections/collection1/leaders Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders
[junit4:junit4]   2> 7345 T2160 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 7346 T2122 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d66b13e920006 type:create cxid:0x49 zxid:0x5d txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 7357 T2160 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 7357 T2160 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 7357 T2160 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:64400/collection1/
[junit4:junit4]   2> 7358 T2160 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 7358 T2160 oasc.SyncStrategy.syncToMe http://127.0.0.1:64400/collection1/ has no replicas
[junit4:junit4]   2> 7358 T2160 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:64400/collection1/
[junit4:junit4]   2> 7358 T2160 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4]   2> 7362 T2122 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d66b13e920006 type:create cxid:0x53 zxid:0x61 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 8440 T2140 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 8463 T2139 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 8463 T2146 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 8463 T2159 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 8500 T2160 oasc.ZkController.register We are http://127.0.0.1:64400/collection1/ and leader is http://127.0.0.1:64400/collection1/
[junit4:junit4]   2> 8500 T2160 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:64400
[junit4:junit4]   2> 8500 T2160 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 8500 T2160 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 8501 T2160 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 8503 T2160 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 8504 T2117 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 8504 T2117 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 8505 T2117 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 8717 T2117 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 8721 T2117 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:17755
[junit4:junit4]   2> 8722 T2117 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 8722 T2117 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 8723 T2117 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1363227533258
[junit4:junit4]   2> 8723 T2117 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1363227533258/solr.xml
[junit4:junit4]   2> 8724 T2117 oasc.CoreContainer.<init> New CoreContainer 711119566
[junit4:junit4]   2> 8724 T2117 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1363227533258/'
[junit4:junit4]   2> 8725 T2117 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1363227533258/'
[junit4:junit4]   2> 8779 T2117 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 8780 T2117 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 8780 T2117 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 8781 T2117 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 8781 T2117 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 8782 T2117 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 8782 T2117 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 8783 T2117 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 8783 T2117 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 8783 T2117 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 8804 T2117 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 8823 T2117 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:16556/solr
[junit4:junit4]   2> 8824 T2117 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 8824 T2117 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:16556 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@64b0093e
[junit4:junit4]   2> 8825 T2117 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 8826 T2172 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:16556. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 8827 T2172 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:16556, initiating session
[junit4:junit4]   2> 8827 T2119 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:30880
[junit4:junit4]   2> 8827 T2119 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:30880
[junit4:junit4]   2> 8830 T2121 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d66b13e920007 with negotiated timeout 20000 for client /140.211.11.196:30880
[junit4:junit4]   2> 8830 T2172 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:16556, sessionid = 0x13d66b13e920007, negotiated timeout = 20000
[junit4:junit4]   2> 8830 T2173 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@64b0093e name:ZooKeeperConnection Watcher:127.0.0.1:16556 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 8830 T2117 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 8833 T2122 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d66b13e920007
[junit4:junit4]   2> 8845 T2173 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 8845 T2119 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:30880 which had sessionid 0x13d66b13e920007
[junit4:junit4]   2> 8845 T2117 oaz.ZooKeeper.close Session: 0x13d66b13e920007 closed
[junit4:junit4]   2> 8846 T2117 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 8855 T2117 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:16556/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@5af21858
[junit4:junit4]   2> 8856 T2117 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 8857 T2174 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:16556. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 8857 T2174 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:16556, initiating session
[junit4:junit4]   2> 8858 T2119 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:21474
[junit4:junit4]   2> 8858 T2119 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:21474
[junit4:junit4]   2> 8859 T2121 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d66b13e920008 with negotiated timeout 20000 for client /140.211.11.196:21474
[junit4:junit4]   2> 8860 T2174 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:16556, sessionid = 0x13d66b13e920008, negotiated timeout = 20000
[junit4:junit4]   2> 8860 T2175 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5af21858 name:ZooKeeperConnection Watcher:127.0.0.1:16556/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 8860 T2117 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 8862 T2122 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d66b13e920008 type:create cxid:0x1 zxid:0x6e txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 8864 T2122 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d66b13e920008 type:create cxid:0x2 zxid:0x6f txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 8868 T2117 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 9871 T2117 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:17755_
[junit4:junit4]   2> 9872 T2122 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d66b13e920008 type:delete cxid:0xb zxid:0x70 txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:17755_ Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:17755_
[junit4:junit4]   2> 9910 T2117 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:17755_
[junit4:junit4]   2> 9913 T2139 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 9914 T2146 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 9914 T2159 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 9914 T2175 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 9914 T2159 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 9914 T2146 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 9915 T2139 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 9921 T2176 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1363227533258/collection1
[junit4:junit4]   2> 9921 T2176 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 9922 T2176 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 9922 T2176 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 9924 T2176 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1363227533258/collection1/'
[junit4:junit4]   2> 9926 T2176 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1363227533258/collection1/lib/README' to classloader
[junit4:junit4]   2> 9926 T2176 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1363227533258/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 9976 T2140 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 9976 T2140 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"2",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:64400_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:64400"}
[junit4:junit4]   2> 9979 T2176 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 9990 T2146 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 9990 T2139 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 9990 T2159 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 9990 T2175 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 10043 T2176 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 10144 T2176 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 10153 T2176 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 10868 T2176 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 10877 T2176 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 10880 T2176 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 10893 T2176 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 10898 T2176 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 10902 T2176 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 10903 T2176 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 10903 T2176 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 10903 T2176 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 10904 T2176 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 10904 T2176 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 10905 T2176 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 10905 T2176 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1363227533258/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363227524744/jetty2/
[junit4:junit4]   2> 10905 T2176 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5c09624
[junit4:junit4]   2> 10905 T2176 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 10906 T2176 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363227524744/jetty2 forceNew: false
[junit4:junit4]   2> 10906 T2176 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1363227524744/jetty2/index/
[junit4:junit4]   2> 10906 T2176 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1363227524744/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 10906 T2176 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363227524744/jetty2/index forceNew: false
[junit4:junit4]   2> 10909 T2176 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@6f4bd2bb lockFactory=org.apache.lucene.store.NativeFSLockFactory@5f941df6; maxCacheMB=0.578125 maxMergeSizeMB=0.6806640625)),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 10909 T2176 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 10911 T2176 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 10911 T2176 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 10911 T2176 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 10912 T2176 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 10912 T2176 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 10912 T2176 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 10912 T2176 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 10913 T2176 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 10913 T2176 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 10918 T2176 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 10923 T2176 oass.SolrIndexSearcher.<init> Opening Searcher@7016c606 main
[junit4:junit4]   2> 10923 T2176 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363227524744/jetty2/tlog
[junit4:junit4]   2> 10924 T2176 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 10924 T2176 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 10927 T2177 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7016c606 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 10928 T2176 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 10929 T2176 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 11494 T2140 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 11495 T2140 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"3",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:17755_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:17755"}
[junit4:junit4]   2> 11495 T2140 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4]   2> 11495 T2140 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4:junit4]   2> 11510 T2146 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 11510 T2159 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 11510 T2175 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 11510 T2139 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 11931 T2176 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 11931 T2176 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:17755 collection:collection1 shard:shard2
[junit4:junit4]   2> 11932 T2176 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4:junit4]   2> 11985 T2122 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d66b13e920008 type:delete cxid:0x47 zxid:0x7f txntype:-1 reqpath:n/a Error Path:/solr/collections/collection1/leaders/shard2 Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders/shard2
[junit4:junit4]   2> 12021 T2176 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 12022 T2122 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d66b13e920008 type:create cxid:0x48 zxid:0x80 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 12046 T2176 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 12046 T2176 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 12046 T2176 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:17755/collection1/
[junit4:junit4]   2> 12046 T2176 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 12047 T2176 oasc.SyncStrategy.syncToMe http://127.0.0.1:17755/collection1/ has no replicas
[junit4:junit4]   2> 12047 T2176 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:17755/collection1/
[junit4:junit4]   2> 12047 T2176 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4]   2> 12050 T2122 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d66b13e920008 type:create cxid:0x51 zxid:0x83 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 13014 T2140 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 13036 T2146 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 13036 T2139 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 13036 T2175 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 13036 T2159 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 13059 T2176 oasc.ZkController.register We are http://127.0.0.1:17755/collection1/ and leader is http://127.0.0.1:17755/collection1/
[junit4:junit4]   2> 13059 T2176 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:17755
[junit4:junit4]   2> 13059 T2176 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 13059 T2176 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 13059 T2176 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 13061 T2176 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 13061 T2117 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 13062 T2117 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 13062 T2117 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 13214 T2117 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 13217 T2117 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:62522
[junit4:junit4]   2> 13218 T2117 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 13218 T2117 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 13218 T2117 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1363227537813
[junit4:junit4]   2> 13219 T2117 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1363227537813/solr.xml
[junit4:junit4]   2> 13219 T2117 oasc.CoreContainer.<init> New CoreContainer 610197557
[junit4:junit4]   2> 13219 T2117 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1363227537813/'
[junit4:junit4]   2> 13220 T2117 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1363227537813/'
[junit4:junit4]   2> 13256 T2117 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 13257 T2117 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 13257 T2117 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 13258 T2117 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 13258 T2117 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 13258 T2117 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 13258 T2117 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 13259 T2117 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 13259 T2117 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 13259 T2117 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 13275 T2117 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 13289 T2117 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:16556/solr
[junit4:junit4]   2> 13289 T2117 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 13289 T2117 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:16556 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@7d33cf49
[junit4:junit4]   2> 13290 T2117 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 13291 T2188 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:16556. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 13291 T2188 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:16556, initiating session
[junit4:junit4]   2> 13292 T2119 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:38971
[junit4:junit4]   2> 13292 T2119 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:38971
[junit4:junit4]   2> 13294 T2121 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d66b13e920009 with negotiated timeout 20000 for client /140.211.11.196:38971
[junit4:junit4]   2> 13294 T2188 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:16556, sessionid = 0x13d66b13e920009, negotiated timeout = 20000
[junit4:junit4]   2> 13294 T2189 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7d33cf49 name:ZooKeeperConnection Watcher:127.0.0.1:16556 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 13295 T2117 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 13296 T2122 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d66b13e920009
[junit4:junit4]   2> 13306 T2189 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 13306 T2119 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:38971 which had sessionid 0x13d66b13e920009
[junit4:junit4]   2> 13306 T2117 oaz.ZooKeeper.close Session: 0x13d66b13e920009 closed
[junit4:junit4]   2> 13307 T2117 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 13316 T2117 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:16556/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@312d2b53
[junit4:junit4]   2> 13317 T2117 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 13317 T2190 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:16556. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 13317 T2190 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:16556, initiating session
[junit4:junit4]   2> 13318 T2119 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:32757
[junit4:junit4]   2> 13318 T2119 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:32757
[junit4:junit4]   2> 13319 T2121 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d66b13e92000a with negotiated timeout 20000 for client /140.211.11.196:32757
[junit4:junit4]   2> 13319 T2190 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:16556, sessionid = 0x13d66b13e92000a, negotiated timeout = 20000
[junit4:junit4]   2> 13319 T2191 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@312d2b53 name:ZooKeeperConnection Watcher:127.0.0.1:16556/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 13319 T2117 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 13320 T2122 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d66b13e92000a type:create cxid:0x1 zxid:0x90 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 13326 T2122 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d66b13e92000a type:create cxid:0x2 zxid:0x91 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 13328 T2117 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 14330 T2117 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:62522_
[junit4:junit4]   2> 14331 T2122 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d66b13e92000a type:delete cxid:0xb zxid:0x92 txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:62522_ Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:62522_
[junit4:junit4]   2> 14333 T2117 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:62522_
[junit4:junit4]   2> 14346 T2139 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 14348 T2175 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 14348 T2146 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 14348 T2191 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 14348 T2159 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 14349 T2146 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 14349 T2159 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 14349 T2139 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 14349 T2175 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 14356 T2192 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1363227537813/collection1
[junit4:junit4]   2> 14356 T2192 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 14356 T2192 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 14357 T2192 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 14358 T2192 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1363227537813/collection1/'
[junit4:junit4]   2> 14359 T2192 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1363227537813/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 14360 T2192 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1363227537813/collection1/lib/README' to classloader
[junit4:junit4]   2> 14408 T2192 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 14464 T2192 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 14549 T2140 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 14550 T2140 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"3",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":"shard2",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:17755_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:17755"}
[junit4:junit4]   2> 14564 T2146 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 14564 T2175 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 14564 T2139 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 14564 T2159 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 14565 T2191 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 14565 T2192 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 14571 T2192 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 15198 T2192 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 15210 T2192 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 15214 T2192 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 15227 T2192 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 15232 T2192 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 15236 T2192 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 15237 T2192 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 15237 T2192 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 15238 T2192 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 15239 T2192 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 15239 T2192 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 15239 T2192 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 15239 T2192 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1363227537813/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363227524744/jetty3/
[junit4:junit4]   2> 15240 T2192 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5c09624
[junit4:junit4]   2> 15240 T2192 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 15241 T2192 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363227524744/jetty3 forceNew: false
[junit4:junit4]   2> 15241 T2192 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1363227524744/jetty3/index/
[junit4:junit4]   2> 15241 T2192 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1363227524744/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 15242 T2192 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363227524744/jetty3/index forceNew: false
[junit4:junit4]   2> 15245 T2192 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@7eab0d73 lockFactory=org.apache.lucene.store.NativeFSLockFactory@171944e1; maxCacheMB=0.578125 maxMergeSizeMB=0.6806640625)),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 15245 T2192 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 15248 T2192 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 15248 T2192 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 15249 T2192 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 15249 T2192 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 15250 T2192 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 15250 T2192 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 15250 T2192 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 15251 T2192 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 15251 T2192 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 15259 T2192 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 15264 T2192 oass.SolrIndexSearcher.<init> Opening Searcher@33e89f61 main
[junit4:junit4]   2> 15264 T2192 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363227524744/jetty3/tlog
[junit4:junit4]   2> 15265 T2192 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 15265 T2192 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 15268 T2193 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@33e89f61 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 15269 T2192 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 15269 T2192 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 16068 T2140 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 16069 T2140 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"4",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:62522_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:62522"}
[junit4:junit4]   2> 16069 T2140 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4]   2> 16069 T2140 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 16088 T2139 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 16088 T2146 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 16089 T2191 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 16089 T2175 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 16089 T2159 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 16272 T2192 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 16272 T2192 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:62522 collection:collection1 shard:shard1
[junit4:junit4]   2> 16275 T2192 oasc.ZkController.register We are http://127.0.0.1:62522/collection1/ and leader is http://127.0.0.1:64400/collection1/
[junit4:junit4]   2> 16275 T2192 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:62522
[junit4:junit4]   2> 16275 T2192 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 16276 T2192 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C160 name=collection1 org.apache.solr.core.SolrCore@55b4c83f url=http://127.0.0.1:62522/collection1 node=127.0.0.1:62522_ C160_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:62522_, base_url=http://127.0.0.1:62522}
[junit4:junit4]   2> 16277 T2194 C160 P62522 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 16277 T2194 C160 P62522 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 16277 T2192 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 16277 T2194 C160 P62522 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4]   2> 16277 T2194 C160 P62522 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 16278 T2117 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 16279 T2117 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 16280 T2117 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 16291 T2194 C160 P62522 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 16480 T2117 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 16483 T2117 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:47567
[junit4:junit4]   2> 16484 T2117 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 16484 T2117 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 16485 T2117 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1363227541034
[junit4:junit4]   2> 16485 T2117 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1363227541034/solr.xml
[junit4:junit4]   2> 16486 T2117 oasc.CoreContainer.<init> New CoreContainer 1931471321
[junit4:junit4]   2> 16487 T2117 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1363227541034/'
[junit4:junit4]   2> 16487 T2117 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1363227541034/'
[junit4:junit4]   2> 16540 T2117 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 16541 T2117 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 16541 T2117 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 16542 T2117 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 16542 T2117 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 16542 T2117 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 16543 T2117 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 16543 T2117 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 16544 T2117 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 16544 T2117 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 16563 T2117 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 16576 T2117 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:16556/solr
[junit4:junit4]   2> 16576 T2117 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 16577 T2117 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:16556 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@3690fae2
[junit4:junit4]   2> 16578 T2117 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 16578 T2205 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:16556. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 16580 T2205 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:16556, initiating session
[junit4:junit4]   2> 16580 T2119 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:24604
[junit4:junit4]   2> 16580 T2119 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:24604
[junit4:junit4]   2> 16582 T2121 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d66b13e92000b with negotiated timeout 20000 for client /140.211.11.196:24604
[junit4:junit4]   2> 16582 T2205 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:16556, sessionid = 0x13d66b13e92000b, negotiated timeout = 20000
[junit4:junit4]   2> 16583 T2206 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3690fae2 name:ZooKeeperConnection Watcher:127.0.0.1:16556 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 16583 T2117 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 16586 T2122 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d66b13e92000b
[junit4:junit4]   2> 16598 T2206 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 16598 T2119 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:24604 which had sessionid 0x13d66b13e92000b
[junit4:junit4]   2> 16598 T2117 oaz.ZooKeeper.close Session: 0x13d66b13e92000b closed
[junit4:junit4]   2> 16599 T2117 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 16607 T2117 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:16556/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@13d7c16b
[junit4:junit4]   2> 16608 T2117 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 16608 T2207 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:16556. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 16609 T2207 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:16556, initiating session
[junit4:junit4]   2> 16609 T2119 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:15828
[junit4:junit4]   2> 16609 T2119 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:15828
[junit4:junit4]   2> 16610 T2121 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d66b13e92000c with negotiated timeout 20000 for client /140.211.11.196:15828
[junit4:junit4]   2> 16610 T2207 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:16556, sessionid = 0x13d66b13e92000c, negotiated timeout = 20000
[junit4:junit4]   2> 16611 T2208 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@13d7c16b name:ZooKeeperConnection Watcher:127.0.0.1:16556/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 16611 T2117 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 16612 T2122 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d66b13e92000c type:create cxid:0x1 zxid:0xa3 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 16614 T2122 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d66b13e92000c type:create cxid:0x2 zxid:0xa4 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 16616 T2117 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 17593 T2140 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 17594 T2140 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"4",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:62522_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:62522"}
[junit4:junit4]   2> 17609 T2208 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 17609 T2159 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 17609 T2191 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 17609 T2146 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 17609 T2175 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 17609 T2139 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 17618 T2117 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:47567_
[junit4:junit4]   2> 17619 T2122 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d66b13e92000c type:delete cxid:0xd zxid:0xa9 txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:47567_ Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:47567_
[junit4:junit4]   2> 17620 T2117 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:47567_
[junit4:junit4]   2> 17621 T2139 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 17622 T2191 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 17622 T2175 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 17622 T2208 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 17622 T2146 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 17622 T2208 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 17622 T2146 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 17622 T2159 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 17622 T2159 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 17623 T2139 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 17623 T2175 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 17623 T2191 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 18305 T2149 oass.SolrDispatchFilter.handleAdminRequest /admin/cores {coreNodeName=4&state=recovering&nodeName=127.0.0.1:62522_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=2002 
[junit4:junit4]   2> 18337 T2209 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1363227541034/collection1
[junit4:junit4]   2> 18337 T2209 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 18338 T2209 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 18338 T2209 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 18339 T2209 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1363227541034/collection1/'
[junit4:junit4]   2> 18340 T2209 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1363227541034/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 18340 T2209 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1363227541034/collection1/lib/README' to classloader
[junit4:junit4]   2> 18376 T2209 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 18417 T2209 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 18517 T2209 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 18521 T2209 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 18952 T2209 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 18963 T2209 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 18966 T2209 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 18976 T2209 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 18980 T2209 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 18983 T2209 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 18984 T2209 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 18985 T2209 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 18985 T2209 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 18986 T2209 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 18986 T2209 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 18986 T2209 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 18986 T2209 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1363227541034/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363227524744/jetty4/
[junit4:junit4]   2> 18986 T2209 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5c09624
[junit4:junit4]   2> 18987 T2209 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 18987 T2209 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363227524744/jetty4 forceNew: false
[junit4:junit4]   2> 18987 T2209 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1363227524744/jetty4/index/
[junit4:junit4]   2> 18988 T2209 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1363227524744/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 18988 T2209 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363227524744/jetty4/index forceNew: false
[junit4:junit4]   2> 18990 T2209 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@7c67fb69 lockFactory=org.apache.lucene.store.NativeFSLockFactory@d244002; maxCacheMB=0.578125 maxMergeSizeMB=0.6806640625)),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 18990 T2209 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 18992 T2209 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 18992 T2209 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 18993 T2209 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 18993 T2209 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 18993 T2209 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 18994 T2209 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 18994 T2209 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 18994 T2209 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 18994 T2209 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 18999 T2209 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 19004 T2209 oass.SolrIndexSearcher.<init> Opening Searcher@3bc4520 main
[junit4:junit4]   2> 19004 T2209 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363227524744/jetty4/tlog
[junit4:junit4]   2> 19005 T2209 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 19005 T2209 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 19008 T2210 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3bc4520 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 19009 T2209 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 19009 T2209 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 19114 T2140 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 19115 T2140 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"5",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:47567_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:47567"}
[junit4:junit4]   2> 19115 T2140 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4]   2> 19115 T2140 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4:junit4]   2> 19128 T2208 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 19128 T2139 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 19128 T2159 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 19128 T2191 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 19128 T2175 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 19128 T2146 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 20012 T2209 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 20012 T2209 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:47567 collection:collection1 shard:shard2
[junit4:junit4]   2> 20015 T2209 oasc.ZkController.register We are http://127.0.0.1:47567/collection1/ and leader is http://127.0.0.1:17755/collection1/
[junit4:junit4]   2> 20015 T2209 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:47567
[junit4:junit4]   2> 20016 T2209 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 20016 T2209 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C161 name=collection1 org.apache.solr.core.SolrCore@462fec6c url=http://127.0.0.1:47567/collection1 node=127.0.0.1:47567_ C161_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:47567_, base_url=http://127.0.0.1:47567}
[junit4:junit4]   2> 20016 T2211 C161 P47567 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 20017 T2211 C161 P47567 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 20016 T2209 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 20017 T2211 C161 P47567 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4]   2> 20017 T2211 C161 P47567 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 20018 T2117 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 20018 T2211 C161 P47567 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 20018 T2117 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 20019 T2117 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 20033 T2117 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 20034 T2117 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 20047 T2150 oasc.CoreContainer.create Creating SolrCore 'onenodecollectioncore' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363227528673/onenodecollectioncore
[junit4:junit4]   2> 20048 T2150 oasc.ZkController.createCollectionZkNode Check for collection zkNode:onenodecollection
[junit4:junit4]   2> 20048 T2150 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:onenodecollection
[junit4:junit4]   2> 20049 T2150 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4]   2> 20050 T2150 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
[junit4:junit4]   2> 20050 T2150 oascc.SolrZkClient.makePath makePath: /collections/onenodecollection
[junit4:junit4]   2> 20052 T2150 oasc.ZkController.readConfigName Load collection config from:/collections/onenodecollection
[junit4:junit4]   2> 20054 T2150 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363227528673/onenodecollectioncore/'
[junit4:junit4]   2> 20128 T2150 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 20189 T2150 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 20290 T2150 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 20296 T2150 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2>  C160_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:62522_, base_url=http://127.0.0.1:62522}
[junit4:junit4]   2> 20307 T2194 C160 P62522 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:64400/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4]   2> 20307 T2194 C160 P62522 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:62522 START replicas=[http://127.0.0.1:64400/collection1/] nUpdates=100
[junit4:junit4]   2> 20307 T2194 C160 P62522 oasu.PeerSync.sync WARNING no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 20308 T2194 C160 P62522 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4]   2> 20308 T2194 C160 P62522 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4]   2> 20308 T2194 C160 P62522 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4]   2> 20309 T2194 C160 P62522 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4]   2> 20309 T2194 C160 P62522 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:64400/collection1/. core=collection1
[junit4:junit4]   2> 20309 T2194 C160 P62522 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C162 name=collection1 org.apache.solr.core.SolrCore@51373b2c url=http://127.0.0.1:64400/collection1 node=127.0.0.1:64400_ C162_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:64400_, base_url=http://127.0.0.1:64400, leader=true}
[junit4:junit4]   2> 20315 T2151 C162 P64400 REQ /get {getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0 
[junit4:junit4]   2> 20324 T2152 C162 P64400 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 20326 T2152 C162 P64400 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@6809635b lockFactory=org.apache.lucene.store.NativeFSLockFactory@f40f91d; maxCacheMB=0.578125 maxMergeSizeMB=0.6806640625)),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 20327 T2152 C162 P64400 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 20327 T2152 C162 P64400 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@6809635b lockFactory=org.apache.lucene.store.NativeFSLockFactory@f40f91d; maxCacheMB=0.578125 maxMergeSizeMB=0.6806640625)),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@6809635b lockFactory=org.apache.lucene.store.NativeFSLockFactory@f40f91d; maxCacheMB=0.578125 maxMergeSizeMB=0.6806640625)),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 20328 T2152 C162 P64400 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[segments_2]
[junit4:junit4]   2> 20328 T2152 C162 P64400 oass.SolrIndexSearcher.<init> Opening Searcher@4ed88b1e realtime
[junit4:junit4]   2> 20329 T2152 C162 P64400 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 20329 T2152 C162 P64400 /update {waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 6
[junit4:junit4]   2> 20330 T2194 C160 P62522 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 20330 T2194 C160 P62522 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4]   2> 20332 T2153 C162 P64400 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 20332 T2153 C162 P64400 REQ /replication {command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 20333 T2194 C160 P62522 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4]   2> 20333 T2194 C160 P62522 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4]   2> 20333 T2194 C160 P62522 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4]   2> 20335 T2153 C162 P64400 REQ /replication {command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0 
[junit4:junit4]   2> 20336 T2194 C160 P62522 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4]   2> 20336 T2194 C160 P62522 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363227524744/jetty3/index.20130313221905075 forceNew: false
[junit4:junit4]   2> 20337 T2194 C160 P62522 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@132b9848 lockFactory=org.apache.lucene.store.NativeFSLockFactory@3b403487)) fullCopy=false
[junit4:junit4]   2> 20339 T2153 C162 P64400 REQ /replication {file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> 20341 T2194 C160 P62522 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4]   2> 20342 T2194 C160 P62522 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4]   2> 20342 T2194 C160 P62522 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4]   2> 20344 T2194 C160 P62522 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@7eab0d73 lockFactory=org.apache.lucene.store.NativeFSLockFactory@171944e1; maxCacheMB=0.578125 maxMergeSizeMB=0.6806640625)),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@7eab0d73 lockFactory=org.apache.lucene.store.NativeFSLockFactory@171944e1; maxCacheMB=0.578125 maxMergeSizeMB=0.6806640625)),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 20345 T2194 C160 P62522 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[segments_2]
[junit4:junit4]   2> 20345 T2194 C160 P62522 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4]   2> 20345 T2194 C160 P62522 oass.SolrIndexSearcher.<init> Opening Searcher@71f2887c main
[junit4:junit4]   2> 20346 T2193 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@71f2887c main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 20347 T2194 C160 P62522 oash.SnapPuller.fetchLatestIndex removing temporary index download directory files MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@132b9848 lockFactory=org.apache.lucene.store.NativeFSLockFactory@3b403487))
[junit4:junit4]   2> 20347 T2194 C160 P62522 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4]   2> 20347 T2194 C160 P62522 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4]   2> 20347 T2194 C160 P62522 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 20347 T2194 C160 P62522 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 20349 T2194 C160 P62522 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4]   2> 20632 T2140 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 20633 T2140 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"5",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":"shard2",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:47567_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:47567"}
[junit4:junit4]   2> 20642 T2140 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"4",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:62522_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:62522"}
[junit4:junit4]   2> 20654 T2139 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 20654 T2191 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 20654 T2159 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 20654 T2208 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 20654 T2175 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 20654 T2146 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 20942 T2150 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 20953 T2150 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 20957 T2150 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 20969 T2150 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 20974 T2150 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 20978 T2150 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 20979 T2150 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 20980 T2150 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 20980 T2150 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 20981 T2150 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 20981 T2150 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 20982 T2150 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 20982 T2150 oasc.SolrCore.<init> [onenodecollectioncore] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363227528673/onenodecollectioncore/, dataDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1363227524745/onenodecollection/
[junit4:junit4]   2> 20983 T2150 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5c09624
[junit4:junit4]   2> 20983 T2150 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 20984 T2150 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1363227524745/onenodecollection forceNew: false
[junit4:junit4]   2> 20984 T2150 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1363227524745/onenodecollection/index/
[junit4:junit4]   2> 20985 T2150 oasc.SolrCore.initIndex WARNING [onenodecollectioncore] Solr index directory '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1363227524745/onenodecollection/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 20985 T2150 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1363227524745/onenodecollection/index forceNew: false
[junit4:junit4]   2> 20988 T2150 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@428c8e9d lockFactory=org.apache.lucene.store.NativeFSLockFactory@7459376; maxCacheMB=0.578125 maxMergeSizeMB=0.6806640625)),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 20989 T2150 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 20991 T2150 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 20991 T2150 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 20992 T2150 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 20993 T2150 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 20993 T2150 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 20994 T2150 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 20994 T2150 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 20995 T2150 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 20995 T2150 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 21003 T2150 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 21010 T2150 oass.SolrIndexSearcher.<init> Opening Searcher@5041595e main
[junit4:junit4]   2> 21010 T2150 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1363227524745/onenodecollection/tlog
[junit4:junit4]   2> 21011 T2150 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 21012 T2150 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 21014 T2215 oasc.SolrCore.registerSearcher [onenodecollectioncore] Registered new searcher Searcher@5041595e main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 21016 T2150 oasc.ZkController.publish publishing core=onenodecollectioncore state=down
[junit4:junit4]   2> 21034 T2165 oass.SolrDispatchFilter.handleAdminRequest /admin/cores {coreNodeName=5&state=recovering&nodeName=127.0.0.1:47567_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001 
[junit4:junit4]   2> 22159 T2140 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 22160 T2140 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":"none",
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"onenodecollectioncore",
[junit4:junit4]   2> 	  "collection":"onenodecollection",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:64400_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:64400"}
[junit4:junit4]   2> 22160 T2140 oasc.Overseer$ClusterStateUpdater.createCollection Create collection onenodecollection with numShards 1
[junit4:junit4]   2> 22160 T2140 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 22173 T2139 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 22173 T2208 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 22173 T2191 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 22173 T2146 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 22173 T2159 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 22173 T2175 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 23020 T2150 oasc.CoreContainer.registerCore registering core: onenodecollectioncore
[junit4:junit4]   2> 23020 T2150 oasc.ZkController.register Register replica - core:onenodecollectioncore address:http://127.0.0.1:64400 collection:onenodecollection shard:shard1
[junit4:junit4]   2> 23021 T2150 oascc.SolrZkClient.makePath makePath: /collections/onenodecollection/leader_elect/shard1/election
[junit4:junit4]   2>  C161_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:47567_, base_url=http://127.0.0.1:47567}
[junit4:junit4]   2> 23035 T2211 C161 P47567 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:17755/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4]   2> 23035 T2211 C161 P47567 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:47567 START replicas=[http://127.0.0.1:17755/collection1/] nUpdates=100
[junit4:junit4]   2> 23035 T2211 C161 P47567 oasu.PeerSync.sync WARNING no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 23035 T2211 C161 P47567 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4]   2> 23036 T2211 C161 P47567 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4]   2> 23036 T2211 C161 P47567 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4]   2> 23036 T2211 C161 P47567 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4]   2> 23036 T2211 C161 P47567 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:17755/collection1/. core=collection1
[junit4:junit4]   2> 23036 T2211 C161 P47567 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 23039 T2122 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d66b13e920006 type:delete cxid:0xaa zxid:0xc5 txntype:-1 reqpath:n/a Error Path:/solr/collections/onenodecollection/leaders Error:KeeperErrorCode = NoNode for /solr/collections/onenodecollection/leaders
[junit4:junit4]   2> 23040 T2150 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 23040 T2122 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d66b13e920006 type:create cxid:0xab zxid:0xc6 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 23043 T2150 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 23043 T2150 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> ASYNC  NEW_CORE C163 name=collection1 org.apache.solr.core.SolrCore@5f5ce228 url=http://127.0.0.1:17755/collection1 node=127.0.0.1:17755_ C163_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:17755_, base_url=http://127.0.0.1:17755, leader=true}
[junit4:junit4]   2> 23043 T2166 C163 P17755 REQ /get {getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0 
[junit4:junit4]   2> 23043 T2150 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:64400/onenodecollectioncore/
[junit4:junit4]   2> 23044 T2150 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 23044 T2150 oasc.SyncStrategy.syncToMe http://127.0.0.1:64400/onenodecollectioncore/ has no replicas
[junit4:junit4]   2> 23044 T2150 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:64400/onenodecollectioncore/
[junit4:junit4]   2> 23045 T2150 oascc.SolrZkClient.makePath makePath: /collections/onenodecollection/leaders/shard1
[junit4:junit4]   2> 23049 T2122 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d66b13e920006 type:create cxid:0xb5 zxid:0xca txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 23053 T2167 C163 P17755 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 23055 T2167 C163 P17755 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@6f4bd2bb lockFactory=org.apache.lucene.store.NativeFSLockFactory@5f941df6; maxCacheMB=0.578125 maxMergeSizeMB=0.6806640625)),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 23056 T2167 C163 P17755 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 23056 T2167 C163 P17755 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@6f4bd2bb lockFactory=org.apache.lucene.store.NativeFSLockFactory@5f941df6; maxCacheMB=0.578125 maxMergeSizeMB=0.6806640625)),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@6f4bd2bb lockFactory=org.apache.lucene.store.NativeFSLockFactory@5f941df6; maxCacheMB=0.578125 maxMergeSizeMB=0.6806640625)),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 23057 T2167 C163 P17755 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[segments_2]
[junit4:junit4]   2> 23057 T2167 C163 P17755 oass.SolrIndexSearcher.<init> Opening Searcher@684bea7e realtime
[junit4:junit4]   2> 23058 T2167 C163 P17755 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 23058 T2167 C163 P17755 /update {waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 5
[junit4:junit4]   2> 23059 T2211 C161 P47567 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 23059 T2211 C161 P47567 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4]   2> 23060 T2168 C163 P17755 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 23061 T2168 C163 P17755 REQ /replication {command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 23061 T2211 C161 P47567 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4]   2> 23061 T2211 C161 P47567 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4]   2> 23062 T2211 C161 P47567 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4]   2> 23064 T2168 C163 P17755 REQ /replication {command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0 
[junit4:junit4]   2> 23065 T2211 C161 P47567 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4]   2> 23066 T2211 C161 P47567 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363227524744/jetty4/index.20130313221907804 forceNew: false
[junit4:junit4]   2> 23066 T2211 C161 P47567 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@4cdbfd25 lockFactory=org.apache.lucene.store.NativeFSLockFactory@297dd4a9)) fullCopy=false
[junit4:junit4]   2> 23069 T2168 C163 P17755 REQ /replication {file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> 23070 T2211 C161 P47567 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4]   2> 23071 T2211 C161 P47567 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4]   2> 23071 T2211 C161 P47567 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4]   2> 23072 T2211 C161 P47567 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@7c67fb69 lockFactory=org.apache.lucene.store.NativeFSLockFactory@d244002; maxCacheMB=0.578125 maxMergeSizeMB=0.6806640625)),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@7c67fb69 lockFactory=org.apache.lucene.store.NativeFSLockFactory@d244002; maxCacheMB=0.578125 maxMergeSizeMB=0.6806640625)),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 23073 T2211 C161 P47567 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[segments_2]
[junit4:junit4]   2> 23073 T2211 C161 P47567 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4]   2> 23073 T2211 C161 P47567 oass.SolrIndexSearcher.<init> Opening Searcher@493e8270 main
[junit4:junit4]   2> 23074 T2210 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@493e8270 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 23075 T2211 C161 P47567 oash.SnapPuller.fetchLatestIndex removing temporary index download directory files MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@4cdbfd25 lockFactory=org.apache.lucene.store.NativeFSLockFactory@297dd4a9))
[junit4:junit4]   2> 23075 T2211 C161 P47567 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4]   2> 23075 T2211 C161 P47567 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4]   2> 23075 T2211 C161 P47567 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 23075 T2211 C161 P47567 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 23077 T2211 C161 P47567 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4]   2> 23677 T2140 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 23693 T2140 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"5",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":"shard2",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:47567_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:47567"}
[junit4:junit4]   2> 23705 T2139 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 23705 T2191 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 23705 T2146 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 23705 T2175 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 23705 T2208 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 23705 T2159 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 23714 T2150 oasc.ZkController.register We are http://127.0.0.1:64400/onenodecollectioncore/ and leader is http://127.0.0.1:64400/onenodecollectioncore/
[junit4:junit4]   2> 23714 T2150 oasc.ZkController.register No LogReplay needed for core=onenodecollectioncore baseURL=http://127.0.0.1:64400
[junit4:junit4]   2> 23714 T2150 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 23715 T2150 oasc.ZkController.publish publishing core=onenodecollectioncore state=active
[junit4:junit4]   2> 23719 T2150 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 23720 T2150 oass.SolrDispatchFilter.handleAdminRequest /admin/cores {numShards=1&roles=none&name=onenodecollectioncore&action=CREATE&collection=onenodecollection&dataDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1363227524745/onenodecollection&wt=javabin&version=2} status=0 QTime=3672 
[junit4:junit4]   2> 23720 T2117 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: onenodecollection failOnTimeout:true timeout (sec):230
[junit4:junit4]   2> 23721 T2117 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 24723 T2117 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 25220 T2140 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 25221 T2140 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":"none",
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"onenodecollectioncore",
[junit4:junit4]   2> 	  "collection":"onenodecollection",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:64400_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:64400"}
[junit4:junit4]   2> 25233 T2208 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 25233 T2146 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 25233 T2159 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 25233 T2191 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 25233 T2175 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 25233 T2139 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 25725 T2117 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 25727 T2117 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: onenodecollection
[junit4:junit4]   2> 25727 T2117 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C164 name=onenodecollectioncore org.apache.solr.core.SolrCore@483f6e20 url=http://127.0.0.1:64400/onenodecollectioncore node=127.0.0.1:64400_ C164_STATE=coll:onenodecollection core:onenodecollectioncore props:{shard=shard1, roles=none, state=active, core=onenodecollectioncore, collection=onenodecollection, node_name=127.0.0.1:64400_, base_url=http://127.0.0.1:64400, leader=true}
[junit4:junit4]   2> 25753 T2154 C164 P64400 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@428c8e9d lockFactory=org.apache.lucene.store.NativeFSLockFactory@7459376; maxCacheMB=0.578125 maxMergeSizeMB=0.6806640625)),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 25754 T2154 C164 P64400 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 25756 T2154 C164 P64400 /update {wt=javabin&version=2} {add=[1 (1429447691982602240)]} 0 6
[junit4:junit4]   2> 25763 T2154 C164 P64400 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 25766 T2154 C164 P64400 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@428c8e9d lockFactory=org.apache.lucene.store.NativeFSLockFactory@7459376; maxCacheMB=0.578125 maxMergeSizeMB=0.6806640625)),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@428c8e9d lockFactory=org.apache.lucene.store.NativeFSLockFactory@7459376; maxCacheMB=0.578125 maxMergeSizeMB=0.6806640625)),segFN=segments_2,generation=2,filenames=[_0.fnm, _0_Disk_0.dvdd, _0_MockSep_0.pos, _0_Memory_0.ram, _0.nvm, _0_Disk_0.dvdm, _0_MockSep_0.tib, _0_Pulsing41_0.doc, _0_Pulsing41_0.pos, _0_MockSep_0.skp, _0_Pulsing41_0.tim, _0_Pulsing41_0.tip, _0_SimpleText_0.pst, _0.nvd, _0_MockSep_0.frq, segments_2, _0.fdx, _0_MockSep_0.tii, _0.si, _0_MockSep_0.doc, _0_MockSep_0.pyl, _0.fdt]
[junit4:junit4]   2> 25767 T2154 C164 P64400 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[_0.fnm, _0_Disk_0.dvdd, _0_MockSep_0.pos, _0_Memory_0.ram, _0.nvm, _0_Disk_0.dvdm, _0_MockSep_0.tib, _0_Pulsing41_0.doc, _0_Pulsing41_0.pos, _0_MockSep_0.skp, _0_Pulsing41_0.tim, _0_Pulsing41_0.tip, _0_SimpleText_0.pst, _0.nvd, _0_MockSep_0.frq, segments_2, _0.fdx, _0_MockSep_0.tii, _0.si, _0_MockSep_0.doc, _0_MockSep_0.pyl, _0.fdt]
[junit4:junit4]   2> 25769 T2154 C164 P64400 oass.SolrIndexSearcher.<init> Opening Searcher@15d6296a main
[junit4:junit4]   2> 25769 T2154 C164 P64400 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 25770 T2215 oasc.SolrCore.registerSearcher [onenodecollectioncore] Registered new searcher Searcher@15d6296a main{StandardDirectoryReader(segments_2:3 _0(5.0):C1)}
[junit4:junit4]   2> 25770 T2154 C164 P64400 /update {waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 7
[junit4:junit4]   2> 25774 T2154 C164 P64400 REQ /select {q=*:*&wt=javabin&version=2} hits=1 status=0 QTime=1 
[junit4:junit4]   2> 25775 T2117 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 25784 T2154 C164 P64400 REQ /select {q=*:*&wt=javabin&version=2} hits=1 status=0 QTime=1 
[junit4:junit4]   2> ASYNC  NEW_CORE C165 name=collection1 org.apache.solr.core.SolrCore@19e99c0f url=http://127.0.0.1:14029/collection1 node=127.0.0.1:14029_ C165_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:14029_, base_url=http://127.0.0.1:14029, leader=true}
[junit4:junit4]   2> 25793 T2129 C165 P14029 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@fdd241a lockFactory=org.apache.lucene.store.NativeFSLockFactory@69a1c8d6; maxCacheMB=0.578125 maxMergeSizeMB=0.6806640625)),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 25794 T2129 C165 P14029 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 25797 T2129 C165 P14029 /update {wt=javabin&version=2} {add=[1 (1429447692024545280)]} 0 6
[junit4:junit4]   2>  C160_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:62522_, base_url=http://127.0.0.1:62522}
[junit4:junit4]   2> 25817 T2181 C160 P62522 /update {distrib.from=http://127.0.0.1:64400/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1429447692040273920)]} 0 2
[junit4:junit4]   2> 25818 T2155 C162 P64400 /update {distrib.from=http://127.0.0.1:17755/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[1 (1429447692040273920)]} 0 13
[junit4:junit4]   2> 25819 T2171 C163 P17755 /update {wt=javabin&version=2} {add=[1]} 0 16
[junit4:junit4]   2> 25821 T2129 C165 P14029 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 25827 T2129 C165 P14029 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@fdd241a lockFactory=org.apache.lucene.store.NativeFSLockFactory@69a1c8d6; maxCacheMB=0.578125 maxMergeSizeMB=0.6806640625)),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@fdd241a lockFactory=org.apache.lucene.store.NativeFSLockFactory@69a1c8d6; maxCacheMB=0.578125 maxMergeSizeMB=0.6806640625)),segFN=segments_2,generation=2,filenames=[_0.fnm, _0_Disk_0.dvdd, _0_MockSep_0.pos, _0_Memory_0.ram, _0.nvm, _0_Disk_0.dvdm, _0_MockSep_0.tib, _0_Pulsing41_0.pos, _0_Pulsing41_0.doc, _0_Pulsing41_0.tim, _0_MockSep_0.skp, _0_Pulsing41_0.tip, _0_SimpleText_0.pst, _0.nvd, _0_MockSep_0.frq, segments_2, _0.fdx, _0_MockSep_0.tii, _0.si, _0_MockSep_0.pyl, _0_MockSep_0.doc, _0.fdt]
[junit4:junit4]   2> 25828 T2129 C165 P14029 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[_0.fnm, _0_Disk_0.dvdd, _0_MockSep_0.pos, _0_Memory_0.ram, _0.nvm, _0_Disk_0.dvdm, _0_MockSep_0.tib, _0_Pulsing41_0.pos, _0_Pulsing41_0.doc, _0_Pulsing41_0.tim, _0_MockSep_0.skp, _0_Pulsing41_0.tip, _0_SimpleText_0.pst, _0.nvd, _0_MockSep_0.frq, segments_2, _0.fdx, _0_MockSep_0.tii, _0.si, _0_MockSep_0.pyl, _0_MockSep_0.doc, _0.fdt]
[junit4:junit4]   2> 25829 T2129 C165 P14029 oass.SolrIndexSearcher.<init> Opening Searcher@2aad2530 main
[junit4:junit4]   2> 25830 T2129 C165 P14029 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 25831 T2143 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2aad2530 main{StandardDirectoryReader(segments_2:3 _0(5.0):C1)}
[junit4:junit4]   2> 25831 T2129 C165 P14029 /update {waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 10
[junit4:junit4]   2> 25834 T2171 C163 P17755 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 25835 T2171 C163 P17755 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@6f4bd2bb lockFactory=org.apache.lucene.store.NativeFSLockFactory@5f941df6; maxCacheMB=0.578125 maxMergeSizeMB=0.6806640625)),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@6f4bd2bb lockFactory=org.apache.lucene.store.NativeFSLockFactory@5f941df6; maxCacheMB=0.578125 maxMergeSizeMB=0.6806640625)),segFN=segments_3,generation=3,filenames=[segments_3]
[junit4:junit4]   2> 25836 T2171 C163 P17755 oasc.SolrDeletionPolicy.updateCommits newest commit = 3[segments_3]
[junit4:junit4]   2> 25836 T2171 C163 P17755 oass.SolrIndexSearcher.<init> Opening Searcher@68a9f8a1 main
[junit4:junit4]   2> 25837 T2171 C163 P17755 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 25837 T2177 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@68a9f8a1 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 25837 T2171 C163 P17755 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:64400/collection1/, StdNode: http://127.0.0.1:62522/collection1/, StdNode: http://127.0.0.1:47567/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4]   2> 25839 T2182 C160 P62522 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 25840 T2155 C162 P64400 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 25845 T2155 C162 P64400 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@6809635b lockFactory=org.apache.lucene.store.NativeFSLockFactory@f40f91d; maxCacheMB=0.578125 maxMergeSizeMB=0.6806640625)),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@6809635b lockFactory=org.apache.lucene.store.NativeFSLockFactory@f40f91d; maxCacheMB=0.578125 maxMergeSizeMB=0.6806640625)),segFN=segments_3,generation=3,filenames=[_0.fnm, _0_Disk_0.dvdd, _0_MockSep_0.pos, _0_Memory_0.ram, _0.nvm, _0_Disk_0.dvdm, _0_MockSep_0.tib, _0_Pulsing41_0.pos, _0_Pulsing41_0.doc, _0_Pulsing41_0.tim, _0_MockSep_0.skp, _0_Pulsing41_0.tip, _0_SimpleText_0.pst, _0.nvd, _0_MockSep_0.frq, _0.fdx, _0_MockSep_0.tii, _0.si, _0_MockSep_0.pyl, _0_MockSep_0.doc, _0.fdt, segments_3]
[junit4:junit4]   2> 25846 T2155 C162 P64400 oasc.SolrDeletionPolicy.updateCommits newest commit = 3[_0.fnm, _0_Disk_0.dvdd, _0_MockSep_0.pos, _0_Memory_0.ram, _0.nvm, _0_Disk_0.dvdm, _0_MockSep_0.tib, _0_Pulsing41_0.pos, _0_Pulsing41_0.doc, _0_Pulsing41_0.tim, _0_MockSep_0.skp, _0_Pulsing41_0.tip, _0_SimpleText_0.pst, _0.nvd, _0_MockSep_0.frq, _0.fdx, _0_MockSep_0.tii, _0.si, _0_MockSep_0.pyl, _0_MockSep_0.doc, _0.fdt, segments_3]
[junit4:junit4]   2>  C161_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:47567_, base_url=http://127.0.0.1:47567}
[junit4:junit4]   2> 25846 T2198 C161 P47567 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 25846 T2182 C160 P62522 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@7eab0d73 lockFactory=org.apache.lucene.store.NativeFSLockFactory@171944e1; maxCacheMB=0.578125 maxMergeSizeMB=0.6806640625)),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@7eab0d73 lockFactory=org.apache.lucene.store.NativeFSLockFactory@171944e1; maxCacheMB=0.578125 maxMergeSizeMB=0.6806640625)),segFN=segments_3,generation=3,filenames=[_0.fnm, _0_Disk_0.dvdd, _0_MockSep_0.pos, _0_Memory_0.ram, _0.nvm, _0_Disk_0.dvdm, _0_MockSep_0.tib, _0_Pulsing41_0.pos, _0_Pulsing41_0.doc, _0_Pulsing41_0.tim, _0_MockSep_0.skp, _0_Pulsing41_0.tip, _0_SimpleText_0.pst, _0.nvd, _0_MockSep_0.frq, _0.fdx, _0_MockSep_0.tii, _0.si, _0_MockSep_0.pyl, _0_MockSep_0.doc, _0.fdt, segments_3]
[junit4:junit4]   2> 25847 T2182 C160 P62522 oasc.SolrDeletionPolicy.updateCommits newest commit = 3[_0.fnm, _0_Disk_0.dvdd, _0_MockSep_0.pos, _0_Memory_0.ram, _0.nvm, _0_Disk_0.dvdm, _0_MockSep_0.tib, _0_Pulsing41_0.pos, _0_Pulsing41_0.doc, _0_Pulsing41_0.tim, _0_MockSep_0.skp, _0_Pulsing41_0.tip, _0_SimpleText_0.pst, _0.nvd, _0_MockSep_0.frq, _0.fdx, _0_MockSep_0.tii, _0.si, _0_MockSep_0.pyl, _0_MockSep_0.doc, _0.fdt, segments_3]
[junit4:junit4]   2> 25847 T2155 C162 P64400 oass.SolrIndexSearcher.<init> Opening Searcher@740f07b7 main
[junit4:junit4]   2> 25847 T2198 C161 P47567 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@7c67fb69 lockFactory=org.apache.lucene.store.NativeFSLockFactory@d244002; maxCacheMB=0.578125 maxMergeSizeMB=0.6806640625)),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@7c67fb69 lockFactory=org.apache.lucene.store.NativeFSLockFactory@d244002; maxCacheMB=0.578125 maxMergeSizeMB=0.6806640625)),segFN=segments_3,generation=3,filenames=[segments_3]
[junit4:junit4]   2> 25848 T2155 C162 P64400 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 25848 T2198 C161 P47567 oasc.SolrDeletionPolicy.updateCommits newest commit = 3[segments_3]
[junit4:junit4]   2> 25848 T2161 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@740f07b7 main{StandardDirectoryReader(segments_3:3:nrt _0(5.0):C1)}
[junit4:junit4]   2> 25848 T2198 C161 P47567 oass.SolrIndexSearcher.<init> Opening Searcher@32bd7a8d main
[junit4:junit4]   2> 25849 T2155 C162 P64400 /update {waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 9
[junit4:junit4]   2> 25849 T2198 C161 P47567 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 25849 T2210 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@32bd7a8d main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 25850 T2198 C161 P47567 /update {waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 4
[junit4:junit4]   2> 25899 T2182 C160 P62522 oass.SolrIndexSearcher.<init> Opening Searcher@1de2ef31 main
[junit4:junit4]   2> 25900 T2182 C160 P62522 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 25900 T2193 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1de2ef31 main{StandardDirectoryReader(segments_3:3:nrt _0(5.0):C1)}
[junit4:junit4]   2> 25901 T2182 C160 P62522 /update {waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 62
[junit4:junit4]   2> 25901 T2171 C163 P17755 /update {waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 67
[junit4:junit4]   2> 25902 T2117 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):230
[junit4:junit4]   2> 25903 T2117 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 25904 T2117 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4]   2> 25906 T2129 C165 P14029 REQ /select {q=*:*&wt=javabin&version=2} hits=1 status=0 QTime=0 
[junit4:junit4]   2> 25907 T2117 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:16556/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@1dc5894b
[junit4:junit4]   2> 25908 T2117 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 25909 T2223 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:16556. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 25909 T2223 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:16556, initiating session
[junit4:junit4]   2> 25909 T2119 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:55208
[junit4:junit4]   2> 25910 T2119 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:55208
[junit4:junit4]   2> 25936 T2121 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d66b13e92000d with negotiated timeout 10000 for client /140.211.11.196:55208
[junit4:junit4]   2> 25936 T2223 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:16556, sessionid = 0x13d66b13e92000d, negotiated timeout = 10000
[junit4:junit4]   2> 25936 T2224 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1dc5894b name:ZooKeeperConnection Watcher:127.0.0.1:16556/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 25936 T2117 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 25938 T2117 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 25940 T2122 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d66b13e92000d
[junit4:junit4]   2> 25941 T2224 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 25941 T2119 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:55208 which had sessionid 0x13d66b13e92000d
[junit4:junit4]   2> 25941 T2117 oaz.ZooKeeper.close Session: 0x13d66b13e92000d closed
[junit4:junit4]   2> 25943 T2149 C162 P64400 REQ /select {q=*:*&distrib=false&wt=javabin&version=2} hits=1 status=0 QTime=0 
[junit4:junit4]   2> 25945 T2165 C163 P17755 REQ /select {q=*:*&distrib=false&wt=javabin&version=2} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 25949 T2152 C162 P64400 REQ /select {NOW=1363227550686&shard.url=127.0.0.1:64400/collection1/|127.0.0.1:62522/collection1/&fl=id,score&start=0&q=*:*&distrib=fals

[...truncated too long message...]

.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
[junit4:junit4]    > 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit4:junit4]    > 	at java.lang.reflect.Method.invoke(Method.java:616)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
[junit4:junit4]    > 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
[junit4:junit4]    > 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
[junit4:junit4]    > 	at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 139245 T2117 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4]   2> 139252 T2116 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
[junit4:junit4]   2> 139306 T2208 oascc.ZkStateReader$3.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 139306 T2208 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 139306 T2208 oascc.ZkStateReader$2.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 139306 T2208 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> NOTE: test params are: codec=Lucene42: {foo_b=PostingsFormat(name=MockSep), n_f1=PostingsFormat(name=Memory doPackFST= true), foo_d=Pulsing41(freqCutoff=17 minBlockSize=96 maxBlockSize=270), foo_f=PostingsFormat(name=MockSep), n_tl1=PostingsFormat(name=Memory doPackFST= true), n_d1=PostingsFormat(name=SimpleText), rnd_b=Pulsing41(freqCutoff=17 minBlockSize=96 maxBlockSize=270), intDefault=PostingsFormat(name=SimpleText), n_td1=PostingsFormat(name=Memory doPackFST= true), timestamp=PostingsFormat(name=SimpleText), id=Pulsing41(freqCutoff=17 minBlockSize=96 maxBlockSize=270), range_facet_sl=PostingsFormat(name=MockSep), range_facet_si=PostingsFormat(name=Memory doPackFST= true), oddField_s=PostingsFormat(name=MockSep), multiDefault=PostingsFormat(name=Memory doPackFST= true), n_tf1=PostingsFormat(name=SimpleText), n_dt1=PostingsFormat(name=Memory doPackFST= true), range_facet_l=PostingsFormat(name=SimpleText), n_ti1=Pulsing41(freqCutoff=17 minBlockSize=96 maxBlockSize=270), text=PostingsFormat(name=MockSep), _version_=PostingsFormat(name=SimpleText), SubjectTerms_mfacet=PostingsFormat(name=SimpleText), a_t=PostingsFormat(name=SimpleText), n_tdt1=PostingsFormat(name=SimpleText), other_tl1=PostingsFormat(name=SimpleText), n_l1=PostingsFormat(name=SimpleText), a_si=PostingsFormat(name=Memory doPackFST= true)}, docValues:{timestamp=DocValuesFormat(name=Disk)}, sim=DefaultSimilarity, locale=en_IE, timezone=America/Antigua
[junit4:junit4]   2> NOTE: FreeBSD 9.0-RELEASE amd64/Sun Microsystems Inc. 1.6.0_32 (64-bit)/cpus=16,threads=1,free=74297848,total=456196096
[junit4:junit4]   2> NOTE: All tests run in this JVM: [TestJmxIntegration, CollectionsAPIDistributedZkTest, TestRangeQuery, TestDistributedGrouping, PingRequestHandlerTest, TestBM25SimilarityFactory, CachingDirectoryFactoryTest, TestFieldCollectionResource, TestPropInject, TestLMJelinekMercerSimilarityFactory, XsltUpdateRequestHandlerTest, CurrencyFieldXmlFileTest, TestZkChroot, RequestHandlersTest, NoCacheHeaderTest, TestSuggestSpellingConverter, SpellCheckComponentTest, SolrRequestParserTest, TermVectorComponentDistributedTest, TestCodecSupport, ClusterStateTest, TestIndexingPerformance, TestMultiCoreConfBootstrap, TestWordDelimiterFilterFactory, TestDefaultSimilarityFactory, TestComponentsName, TestLazyCores, SortByFunctionTest, UniqFieldsUpdateProcessorFactoryTest, TestLuceneMatchVersion, XmlUpdateRequestHandlerTest, TestPartialUpdateDeduplication, TestFoldingMultitermQuery, TestCSVResponseWriter, SuggesterTSTTest, TestArbitraryIndexDir, EchoParamsTest, TestDistributedSearch, TestSystemIdResolver, TestUtils, UpdateParamsTest, TestPostingsSolrHighlighter, BadIndexSchemaTest, TestSolrJ, TestHashPartitioner, ShardRoutingCustomTest, ConvertedLegacyTest, TestSolr4Spatial, IndexReaderFactoryTest, DocValuesMultiTest, OpenExchangeRatesOrgProviderTest, TestFieldTypeCollectionResource, TestQueryTypes, IndexSchemaRuntimeFieldTest, URLClassifyProcessorTest, TestFiltering, TestSolrDiscoveryProperties, ZkNodePropsTest, TestStressLucene, TestBinaryResponseWriter, TestIndexSearcher, TestDFRSimilarityFactory, TestJmxMonitoredMap, BasicDistributedZkTest, ChaosMonkeySafeLeaderTest, BasicDistributedZk2Test]
[junit4:junit4] Completed on J1 in 139.54s, 1 test, 1 error <<< FAILURES!

[...truncated 424 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/build.xml:381: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/build.xml:361: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/build.xml:183: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/solr/common-build.xml:449: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/lucene/common-build.xml:1213: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java6/lucene/common-build.xml:877: There were test failures: 271 suites, 1142 tests, 1 error, 27 ignored (8 assumptions)

Total time: 51 minutes 31 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure