You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2013/03/09 18:33:10 UTC
[JENKINS] Lucene-Solr-trunk-Linux (32bit/ibm-j9-jdk7) - Build #
4639 - Failure!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/4639/
Java: 32bit/ibm-j9-jdk7
1 tests failed.
REGRESSION: org.apache.solr.cloud.BasicDistributedZk2Test.testDistribSearch
Error Message:
Server at http://127.0.0.1:50988/onenodecollectioncore returned non ok status:404, message:Can not find: /onenodecollectioncore/update
Stack Trace:
org.apache.solr.common.SolrException: Server at http://127.0.0.1:50988/onenodecollectioncore returned non ok status:404, message:Can not find: /onenodecollectioncore/update
at __randomizedtesting.SeedInfo.seed([FB5781DC2C96ADC:8E53F605B5960AE0]:0)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:373)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:181)
at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:117)
at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:116)
at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:102)
at org.apache.solr.cloud.BasicDistributedZk2Test.testNodeWithoutCollectionForwarding(BasicDistributedZk2Test.java:200)
at org.apache.solr.cloud.BasicDistributedZk2Test.doTest(BasicDistributedZk2Test.java:89)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:806)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:88)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:613)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:780)
Build Log:
[...truncated 9212 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.BasicDistributedZk2Test
[junit4:junit4] 2> 0 T2246 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4:junit4] 2> 4 T2246 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1362849930771
[junit4:junit4] 2> 5 T2246 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 6 T2247 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 7 T2247 oazs.ZooKeeperServer.setTickTime tickTime set to 1000
[junit4:junit4] 2> 7 T2247 oazs.ZooKeeperServer.setMinSessionTimeout minSessionTimeout set to -1
[junit4:junit4] 2> 7 T2247 oazs.ZooKeeperServer.setMaxSessionTimeout maxSessionTimeout set to -1
[junit4:junit4] 2> 8 T2247 oazs.NIOServerCnxnFactory.configure binding to port 0.0.0.0/0.0.0.0:0
[junit4:junit4] 2> 8 T2247 oazsp.FileTxnSnapLog.save Snapshotting: 0x0 to /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-1362849930771/zookeeper/server1/data/version-2/snapshot.0
[junit4:junit4] 2> 106 T2246 oasc.ZkTestServer.run start zk server on port:40583
[junit4:junit4] 2> 107 T2246 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:40583 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@e8525a8d
[junit4:junit4] 2> 110 T2252 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:40583. Will not attempt to authenticate using SASL (Access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 109 T2246 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 110 T2252 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:40583, initiating session
[junit4:junit4] 2> 111 T2248 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:45500
[junit4:junit4] 2> 112 T2248 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:45500
[junit4:junit4] 2> 112 T2250 oazsp.FileTxnLog.append Creating new log file: log.1
[junit4:junit4] 2> 561 T2250 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d502f9e170000 with negotiated timeout 10000 for client /127.0.0.1:45500
[junit4:junit4] 2> 561 T2252 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:40583, sessionid = 0x13d502f9e170000, negotiated timeout = 10000
[junit4:junit4] 2> 562 T2253 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e8525a8d name:ZooKeeperConnection Watcher:127.0.0.1:40583 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 562 T2246 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 563 T2246 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 564 T2251 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d502f9e170000
[junit4:junit4] 2> 565 T2253 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 565 T2248 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:45500 which had sessionid 0x13d502f9e170000
[junit4:junit4] 2> 565 T2246 oaz.ZooKeeper.close Session: 0x13d502f9e170000 closed
[junit4:junit4] 2> 565 T2246 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:40583/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@ca28fdf1
[junit4:junit4] 2> 567 T2254 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:40583. Will not attempt to authenticate using SASL (Access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 566 T2246 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 567 T2254 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:40583, initiating session
[junit4:junit4] 2> 567 T2248 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:45501
[junit4:junit4] 2> 568 T2248 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:45501
[junit4:junit4] 2> 568 T2250 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d502f9e170001 with negotiated timeout 10000 for client /127.0.0.1:45501
[junit4:junit4] 2> 568 T2254 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:40583, sessionid = 0x13d502f9e170001, negotiated timeout = 10000
[junit4:junit4] 2> 569 T2255 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ca28fdf1 name:ZooKeeperConnection Watcher:127.0.0.1:40583/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 569 T2246 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 569 T2246 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 571 T2246 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 573 T2246 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 575 T2246 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 577 T2246 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 578 T2246 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 583 T2246 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 584 T2246 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 586 T2246 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 587 T2246 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 590 T2246 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 590 T2246 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 592 T2246 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 593 T2246 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 595 T2246 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 596 T2246 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 598 T2246 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 598 T2246 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 600 T2246 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 601 T2246 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 603 T2246 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 604 T2246 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 606 T2251 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d502f9e170001
[junit4:junit4] 2> 606 T2255 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 606 T2248 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:45501 which had sessionid 0x13d502f9e170001
[junit4:junit4] 2> 606 T2246 oaz.ZooKeeper.close Session: 0x13d502f9e170001 closed
[junit4:junit4] 2> 666 T2246 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4] 2> 668 T2246 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:46733
[junit4:junit4] 2> 669 T2246 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 669 T2246 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 670 T2246 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1362849931373
[junit4:junit4] 2> 670 T2246 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1362849931373/solr.xml
[junit4:junit4] 2> 671 T2246 oasc.CoreContainer.<init> New CoreContainer 1047548841
[junit4:junit4] 2> 672 T2246 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1362849931373/'
[junit4:junit4] 2> 672 T2246 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1362849931373/'
[junit4:junit4] 2> 685 T2246 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 686 T2246 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 686 T2246 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 687 T2246 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 687 T2246 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 688 T2246 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 688 T2246 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 689 T2246 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 690 T2246 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 690 T2246 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 698 T2246 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4] 2> 708 T2246 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:40583/solr
[junit4:junit4] 2> 708 T2246 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 709 T2246 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:40583 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@9f91b05d
[junit4:junit4] 2> 710 T2265 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:40583. Will not attempt to authenticate using SASL (Access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 710 T2246 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 711 T2265 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:40583, initiating session
[junit4:junit4] 2> 711 T2248 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:45502
[junit4:junit4] 2> 712 T2248 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:45502
[junit4:junit4] 2> 712 T2250 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d502f9e170002 with negotiated timeout 20000 for client /127.0.0.1:45502
[junit4:junit4] 2> 712 T2265 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:40583, sessionid = 0x13d502f9e170002, negotiated timeout = 20000
[junit4:junit4] 2> 713 T2266 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@9f91b05d name:ZooKeeperConnection Watcher:127.0.0.1:40583 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 713 T2246 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 714 T2251 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d502f9e170002
[junit4:junit4] 2> 715 T2248 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:45502 which had sessionid 0x13d502f9e170002
[junit4:junit4] 2> 715 T2266 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 715 T2246 oaz.ZooKeeper.close Session: 0x13d502f9e170002 closed
[junit4:junit4] 2> 716 T2246 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 720 T2246 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:40583/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@3603fa98
[junit4:junit4] 2> 721 T2246 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 722 T2267 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:40583. Will not attempt to authenticate using SASL (Access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 722 T2267 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:40583, initiating session
[junit4:junit4] 2> 723 T2248 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:45503
[junit4:junit4] 2> 723 T2248 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:45503
[junit4:junit4] 2> 724 T2250 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d502f9e170003 with negotiated timeout 20000 for client /127.0.0.1:45503
[junit4:junit4] 2> 724 T2267 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:40583, sessionid = 0x13d502f9e170003, negotiated timeout = 20000
[junit4:junit4] 2> 725 T2268 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3603fa98 name:ZooKeeperConnection Watcher:127.0.0.1:40583/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 725 T2246 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 726 T2251 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d502f9e170003 type:create cxid:0x2 zxid:0x1a txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 727 T2246 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 729 T2246 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46733_
[junit4:junit4] 2> 729 T2251 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d502f9e170003 type:delete cxid:0x8 zxid:0x1c txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:46733_ Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:46733_
[junit4:junit4] 2> 730 T2246 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:46733_
[junit4:junit4] 2> 732 T2246 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 735 T2251 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d502f9e170003 type:delete cxid:0x16 zxid:0x21 txntype:-1 reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4] 2> 736 T2246 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 738 T2246 oasc.Overseer.start Overseer (id=89315733063270403-127.0.0.1:46733_-n_0000000000) starting
[junit4:junit4] 2> 739 T2251 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d502f9e170003 type:create cxid:0x1b zxid:0x23 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 740 T2251 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d502f9e170003 type:create cxid:0x1c zxid:0x24 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 741 T2251 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d502f9e170003 type:create cxid:0x1d zxid:0x25 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 741 T2251 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d502f9e170003 type:create cxid:0x1e zxid:0x26 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 742 T2270 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 743 T2246 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 745 T2246 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 746 T2246 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 748 T2269 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 750 T2271 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1362849931373/collection1
[junit4:junit4] 2> 751 T2271 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 751 T2271 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 751 T2271 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 752 T2271 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1362849931373/collection1/'
[junit4:junit4] 2> 753 T2271 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1362849931373/collection1/lib/README' to classloader
[junit4:junit4] 2> 753 T2271 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1362849931373/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 777 T2271 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 811 T2271 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 812 T2271 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4] 2> 816 T2271 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4] 2> 1143 T2271 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 1148 T2271 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 1151 T2271 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 1159 T2271 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 1163 T2271 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 1166 T2271 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 1167 T2271 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 1168 T2271 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 1168 T2271 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 1169 T2271 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 1169 T2271 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 1169 T2271 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1362849931373/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1362849930771/control/data/
[junit4:junit4] 2> 1170 T2271 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2059cd3d
[junit4:junit4] 2> 1170 T2271 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 1171 T2271 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-1362849930771/control/data forceNew: false
[junit4:junit4] 2> 1171 T2271 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1362849930771/control/data/index/
[junit4:junit4] 2> 1171 T2271 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1362849930771/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 1172 T2271 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-1362849930771/control/data/index forceNew: false
[junit4:junit4] 2> 1173 T2271 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4f641373 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1ab282b0),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> 1174 T2271 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4] 2> 1175 T2271 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 1175 T2271 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 1176 T2271 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 1176 T2271 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 1177 T2271 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 1177 T2271 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 1177 T2271 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 1177 T2271 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 1178 T2271 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 1179 T2271 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 1181 T2271 oass.SolrIndexSearcher.<init> Opening Searcher@725b1484 main
[junit4:junit4] 2> 1181 T2271 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1362849930771/control/data/tlog
[junit4:junit4] 2> 1182 T2271 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 1182 T2271 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 1185 T2272 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@725b1484 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 1185 T2271 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 1186 T2271 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 1186 T2251 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d502f9e170003 type:create cxid:0x55 zxid:0x2a txntype:-1 reqpath:n/a Error Path:/solr/overseer/queue Error:KeeperErrorCode = NoNode for /solr/overseer/queue
[junit4:junit4] 2> 2251 T2269 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 2251 T2269 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "base_url":"http://127.0.0.1:46733",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:46733_",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 2252 T2269 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 2252 T2269 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4] 2> 2252 T2251 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d502f9e170003 type:create cxid:0x5d zxid:0x2d txntype:-1 reqpath:n/a Error Path:/solr/overseer/queue-work Error:KeeperErrorCode = NoNode for /solr/overseer/queue-work
[junit4:junit4] 2> 2255 T2268 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 3189 T2271 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 3189 T2271 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:46733 collection:control_collection shard:shard1
[junit4:junit4] 2> 3191 T2271 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 3200 T2251 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d502f9e170003 type:delete cxid:0x77 zxid:0x37 txntype:-1 reqpath:n/a Error Path:/solr/collections/control_collection/leaders Error:KeeperErrorCode = NoNode for /solr/collections/control_collection/leaders
[junit4:junit4] 2> 3201 T2271 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 3202 T2251 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d502f9e170003 type:create cxid:0x78 zxid:0x38 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 3205 T2271 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 3206 T2271 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 3206 T2271 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:46733/collection1/
[junit4:junit4] 2> 3207 T2271 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 3207 T2271 oasc.SyncStrategy.syncToMe http://127.0.0.1:46733/collection1/ has no replicas
[junit4:junit4] 2> 3208 T2271 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:46733/collection1/
[junit4:junit4] 2> 3209 T2271 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 3214 T2251 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d502f9e170003 type:create cxid:0x82 zxid:0x3c txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 3758 T2269 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 3763 T2268 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 3768 T2271 oasc.ZkController.register We are http://127.0.0.1:46733/collection1/ and leader is http://127.0.0.1:46733/collection1/
[junit4:junit4] 2> 3768 T2271 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:46733
[junit4:junit4] 2> 3769 T2271 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 3769 T2271 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 3769 T2271 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 3770 T2271 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 3772 T2246 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 3772 T2246 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 3773 T2246 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 3777 T2246 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 3777 T2246 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:40583/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@94ecc167
[junit4:junit4] 2> 3778 T2274 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:40583. Will not attempt to authenticate using SASL (Access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 3778 T2246 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 3779 T2274 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:40583, initiating session
[junit4:junit4] 2> 3779 T2248 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:45509
[junit4:junit4] 2> 3779 T2248 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:45509
[junit4:junit4] 2> 3780 T2250 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d502f9e170004 with negotiated timeout 10000 for client /127.0.0.1:45509
[junit4:junit4] 2> 3780 T2274 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:40583, sessionid = 0x13d502f9e170004, negotiated timeout = 10000
[junit4:junit4] 2> 3781 T2275 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@94ecc167 name:ZooKeeperConnection Watcher:127.0.0.1:40583/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 3781 T2246 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 3783 T2246 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 3784 T2246 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 3840 T2246 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4] 2> 3841 T2246 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:40536
[junit4:junit4] 2> 3842 T2246 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 3843 T2246 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 3843 T2246 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1362849934551
[junit4:junit4] 2> 3844 T2246 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1362849934551/solr.xml
[junit4:junit4] 2> 3844 T2246 oasc.CoreContainer.<init> New CoreContainer 654594879
[junit4:junit4] 2> 3845 T2246 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1362849934551/'
[junit4:junit4] 2> 3845 T2246 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1362849934551/'
[junit4:junit4] 2> 3857 T2246 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 3858 T2246 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 3858 T2246 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 3859 T2246 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 3859 T2246 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 3860 T2246 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 3860 T2246 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 3861 T2246 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 3861 T2246 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 3862 T2246 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 3869 T2246 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4] 2> 3878 T2246 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:40583/solr
[junit4:junit4] 2> 3879 T2246 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 3879 T2246 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:40583 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@f3188000
[junit4:junit4] 2> 3881 T2285 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:40583. Will not attempt to authenticate using SASL (Access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 3880 T2246 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 3881 T2285 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:40583, initiating session
[junit4:junit4] 2> 3881 T2248 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:45510
[junit4:junit4] 2> 3882 T2248 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:45510
[junit4:junit4] 2> 3883 T2250 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d502f9e170005 with negotiated timeout 20000 for client /127.0.0.1:45510
[junit4:junit4] 2> 3883 T2285 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:40583, sessionid = 0x13d502f9e170005, negotiated timeout = 20000
[junit4:junit4] 2> 3883 T2286 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f3188000 name:ZooKeeperConnection Watcher:127.0.0.1:40583 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 3883 T2246 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 3884 T2251 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d502f9e170005
[junit4:junit4] 2> 3885 T2286 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 3885 T2248 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:45510 which had sessionid 0x13d502f9e170005
[junit4:junit4] 2> 3885 T2246 oaz.ZooKeeper.close Session: 0x13d502f9e170005 closed
[junit4:junit4] 2> 3885 T2246 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 3889 T2246 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:40583/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@1c96f8dd
[junit4:junit4] 2> 3891 T2287 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:40583. Will not attempt to authenticate using SASL (Access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 3890 T2246 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 3891 T2287 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:40583, initiating session
[junit4:junit4] 2> 3891 T2248 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:45511
[junit4:junit4] 2> 3892 T2248 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:45511
[junit4:junit4] 2> 3893 T2250 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d502f9e170006 with negotiated timeout 20000 for client /127.0.0.1:45511
[junit4:junit4] 2> 3893 T2287 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:40583, sessionid = 0x13d502f9e170006, negotiated timeout = 20000
[junit4:junit4] 2> 3894 T2288 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c96f8dd name:ZooKeeperConnection Watcher:127.0.0.1:40583/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 3894 T2246 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 3895 T2251 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d502f9e170006 type:create cxid:0x1 zxid:0x4a txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 3896 T2251 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d502f9e170006 type:create cxid:0x2 zxid:0x4b txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 3897 T2246 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 4899 T2246 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40536_
[junit4:junit4] 2> 4900 T2251 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d502f9e170006 type:delete cxid:0xb zxid:0x4c txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:40536_ Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:40536_
[junit4:junit4] 2> 4901 T2246 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:40536_
[junit4:junit4] 2> 4903 T2268 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 4903 T2275 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 4903 T2268 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 4903 T2288 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 4907 T2289 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1362849934551/collection1
[junit4:junit4] 2> 4907 T2289 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 4908 T2289 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 4908 T2289 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 4909 T2289 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1362849934551/collection1/'
[junit4:junit4] 2> 4910 T2289 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1362849934551/collection1/lib/README' to classloader
[junit4:junit4] 2> 4910 T2289 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1362849934551/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 4932 T2289 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 4960 T2289 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 4961 T2289 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4] 2> 4965 T2289 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4] 2> 5263 T2289 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 5267 T2269 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 5268 T2269 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "base_url":"http://127.0.0.1:46733",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:46733_",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 5268 T2289 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 5270 T2288 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 5270 T2275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 5270 T2268 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 5271 T2289 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 5279 T2289 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 5283 T2289 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 5286 T2289 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 5287 T2289 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 5287 T2289 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 5288 T2289 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 5289 T2289 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 5289 T2289 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 5289 T2289 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1362849934551/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1362849930771/jetty1/
[junit4:junit4] 2> 5290 T2289 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2059cd3d
[junit4:junit4] 2> 5290 T2289 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 5291 T2289 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-1362849930771/jetty1 forceNew: false
[junit4:junit4] 2> 5291 T2289 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1362849930771/jetty1/index/
[junit4:junit4] 2> 5291 T2289 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1362849930771/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 5292 T2289 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-1362849930771/jetty1/index forceNew: false
[junit4:junit4] 2> 5293 T2289 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@12f98e4b lockFactory=org.apache.lucene.store.NativeFSLockFactory@9cb4f53b),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> 5293 T2289 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4] 2> 5295 T2289 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 5295 T2289 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 5296 T2289 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 5296 T2289 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 5297 T2289 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 5297 T2289 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 5297 T2289 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 5298 T2289 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 5298 T2289 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 5299 T2289 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 5301 T2289 oass.SolrIndexSearcher.<init> Opening Searcher@5568e1b9 main
[junit4:junit4] 2> 5301 T2289 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1362849930771/jetty1/tlog
[junit4:junit4] 2> 5302 T2289 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 5302 T2289 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 5306 T2290 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5568e1b9 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 5307 T2289 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 5307 T2289 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 6774 T2269 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 6774 T2269 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "base_url":"http://127.0.0.1:40536",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:40536_",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 6774 T2269 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 2
[junit4:junit4] 2> 6775 T2269 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4] 2> 6778 T2275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 6778 T2288 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 6778 T2268 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 7309 T2289 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 7309 T2289 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:40536 collection:collection1 shard:shard1
[junit4:junit4] 2> 7310 T2289 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 7317 T2251 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d502f9e170006 type:delete cxid:0x48 zxid:0x5c txntype:-1 reqpath:n/a Error Path:/solr/collections/collection1/leaders Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders
[junit4:junit4] 2> 7318 T2289 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 7319 T2251 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d502f9e170006 type:create cxid:0x49 zxid:0x5d txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 7320 T2289 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 7321 T2289 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 7322 T2289 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:40536/collection1/
[junit4:junit4] 2> 7322 T2289 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 7323 T2289 oasc.SyncStrategy.syncToMe http://127.0.0.1:40536/collection1/ has no replicas
[junit4:junit4] 2> 7323 T2289 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:40536/collection1/
[junit4:junit4] 2> 7324 T2289 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 7327 T2251 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d502f9e170006 type:create cxid:0x53 zxid:0x61 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 8282 T2269 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 8289 T2268 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 8289 T2275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 8289 T2288 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 8331 T2289 oasc.ZkController.register We are http://127.0.0.1:40536/collection1/ and leader is http://127.0.0.1:40536/collection1/
[junit4:junit4] 2> 8331 T2289 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:40536
[junit4:junit4] 2> 8332 T2289 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 8332 T2289 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 8333 T2289 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 8334 T2289 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 8335 T2246 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 8336 T2246 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 8336 T2246 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 8405 T2246 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4] 2> 8407 T2246 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:50988
[junit4:junit4] 2> 8407 T2246 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 8408 T2246 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 8409 T2246 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1362849939106
[junit4:junit4] 2> 8409 T2246 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1362849939106/solr.xml
[junit4:junit4] 2> 8410 T2246 oasc.CoreContainer.<init> New CoreContainer -1605039329
[junit4:junit4] 2> 8410 T2246 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1362849939106/'
[junit4:junit4] 2> 8411 T2246 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1362849939106/'
[junit4:junit4] 2> 8423 T2246 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 8424 T2246 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 8424 T2246 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 8425 T2246 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 8425 T2246 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 8426 T2246 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 8426 T2246 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 8427 T2246 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 8428 T2246 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 8428 T2246 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 8435 T2246 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4] 2> 8446 T2246 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:40583/solr
[junit4:junit4] 2> 8447 T2246 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 8447 T2246 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:40583 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@69db4e84
[junit4:junit4] 2> 8449 T2301 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:40583. Will not attempt to authenticate using SASL (Access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 8448 T2246 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 8449 T2301 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:40583, initiating session
[junit4:junit4] 2> 8449 T2248 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:45515
[junit4:junit4] 2> 8450 T2248 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:45515
[junit4:junit4] 2> 8450 T2250 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d502f9e170007 with negotiated timeout 20000 for client /127.0.0.1:45515
[junit4:junit4] 2> 8450 T2301 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:40583, sessionid = 0x13d502f9e170007, negotiated timeout = 20000
[junit4:junit4] 2> 8451 T2302 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@69db4e84 name:ZooKeeperConnection Watcher:127.0.0.1:40583 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 8451 T2246 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 8452 T2251 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d502f9e170007
[junit4:junit4] 2> 8452 T2302 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 8453 T2248 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:45515 which had sessionid 0x13d502f9e170007
[junit4:junit4] 2> 8452 T2246 oaz.ZooKeeper.close Session: 0x13d502f9e170007 closed
[junit4:junit4] 2> 8453 T2246 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 8457 T2246 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:40583/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@d1744845
[junit4:junit4] 2> 8458 T2303 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:40583. Will not attempt to authenticate using SASL (Access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 8458 T2246 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 8459 T2303 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:40583, initiating session
[junit4:junit4] 2> 8459 T2248 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:45516
[junit4:junit4] 2> 8459 T2248 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:45516
[junit4:junit4] 2> 8460 T2250 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d502f9e170008 with negotiated timeout 20000 for client /127.0.0.1:45516
[junit4:junit4] 2> 8460 T2303 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:40583, sessionid = 0x13d502f9e170008, negotiated timeout = 20000
[junit4:junit4] 2> 8460 T2304 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d1744845 name:ZooKeeperConnection Watcher:127.0.0.1:40583/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 8461 T2246 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 8461 T2251 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d502f9e170008 type:create cxid:0x1 zxid:0x6e txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 8462 T2251 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d502f9e170008 type:create cxid:0x2 zxid:0x6f txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 8463 T2246 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 9466 T2246 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50988_
[junit4:junit4] 2> 9468 T2251 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d502f9e170008 type:delete cxid:0xb zxid:0x70 txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:50988_ Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:50988_
[junit4:junit4] 2> 9469 T2246 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50988_
[junit4:junit4] 2> 9473 T2275 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 9473 T2268 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 9473 T2304 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 9474 T2268 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 9474 T2275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 9474 T2288 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 9476 T2288 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 9482 T2305 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1362849939106/collection1
[junit4:junit4] 2> 9482 T2305 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 9483 T2305 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 9484 T2305 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 9485 T2305 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1362849939106/collection1/'
[junit4:junit4] 2> 9486 T2305 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1362849939106/collection1/lib/README' to classloader
[junit4:junit4] 2> 9486 T2305 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1362849939106/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 9511 T2305 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 9539 T2305 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 9540 T2305 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4] 2> 9544 T2305 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4] 2> 9794 T2269 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 9795 T2269 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "base_url":"http://127.0.0.1:40536",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:40536_",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 9798 T2304 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 9798 T2268 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 9798 T2275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 9798 T2288 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 9867 T2305 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 9872 T2305 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 9874 T2305 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 9883 T2305 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 9886 T2305 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 9890 T2305 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 9891 T2305 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 9891 T2305 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 9892 T2305 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 9892 T2305 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 9893 T2305 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 9893 T2305 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1362849939106/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1362849930771/jetty2/
[junit4:junit4] 2> 9894 T2305 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2059cd3d
[junit4:junit4] 2> 9894 T2305 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 9895 T2305 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-1362849930771/jetty2 forceNew: false
[junit4:junit4] 2> 9895 T2305 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1362849930771/jetty2/index/
[junit4:junit4] 2> 9896 T2305 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1362849930771/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 9896 T2305 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-1362849930771/jetty2/index forceNew: false
[junit4:junit4] 2> 9897 T2305 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@ab5fe65d lockFactory=org.apache.lucene.store.NativeFSLockFactory@5f4638ab),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> 9898 T2305 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4] 2> 9899 T2305 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 9900 T2305 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 9900 T2305 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 9901 T2305 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 9901 T2305 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 9902 T2305 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 9902 T2305 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 9903 T2305 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 9903 T2305 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 9904 T2305 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 9907 T2305 oass.SolrIndexSearcher.<init> Opening Searcher@f076e77b main
[junit4:junit4] 2> 9907 T2305 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1362849930771/jetty2/tlog
[junit4:junit4] 2> 9908 T2305 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 9908 T2305 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 9912 T2306 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@f076e77b main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 9914 T2305 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 9914 T2305 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 11301 T2269 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 11302 T2269 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "base_url":"http://127.0.0.1:50988",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:50988_",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 11302 T2269 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4] 2> 11302 T2269 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4:junit4] 2> 11306 T2268 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 11306 T2275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 11306 T2288 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 11307 T2304 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 11915 T2305 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 11916 T2305 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:50988 collection:collection1 shard:shard2
[junit4:junit4] 2> 11917 T2305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4:junit4] 2> 11925 T2251 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d502f9e170008 type:delete cxid:0x47 zxid:0x7f txntype:-1 reqpath:n/a Error Path:/solr/collections/collection1/leaders/shard2 Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders/shard2
[junit4:junit4] 2> 11926 T2305 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 11927 T2251 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d502f9e170008 type:create cxid:0x48 zxid:0x80 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 11929 T2305 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 11929 T2305 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 11930 T2305 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:50988/collection1/
[junit4:junit4] 2> 11931 T2305 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 11931 T2305 oasc.SyncStrategy.syncToMe http://127.0.0.1:50988/collection1/ has no replicas
[junit4:junit4] 2> 11932 T2305 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:50988/collection1/
[junit4:junit4] 2> 11932 T2305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4] 2> 11937 T2251 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d502f9e170008 type:create cxid:0x51 zxid:0x83 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 12810 T2269 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 12818 T2288 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 12818 T2275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 12818 T2268 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 12818 T2304 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 12842 T2305 oasc.ZkController.register We are http://127.0.0.1:50988/collection1/ and leader is http://127.0.0.1:50988/collection1/
[junit4:junit4] 2> 12842 T2305 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:50988
[junit4:junit4] 2> 12843 T2305 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 12843 T2305 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 12844 T2305 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 12846 T2305 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 12847 T2246 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 12848 T2246 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 12850 T2246 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 12919 T2246 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4] 2> 12921 T2246 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:49682
[junit4:junit4] 2> 12921 T2246 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 12922 T2246 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 12923 T2246 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1362849943624
[junit4:junit4] 2> 12923 T2246 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1362849943624/solr.xml
[junit4:junit4] 2> 12924 T2246 oasc.CoreContainer.<init> New CoreContainer -65400496
[junit4:junit4] 2> 12924 T2246 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1362849943624/'
[junit4:junit4] 2> 12925 T2246 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1362849943624/'
[junit4:junit4] 2> 12938 T2246 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 12939 T2246 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 12939 T2246 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 12940 T2246 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 12940 T2246 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 12941 T2246 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 12941 T2246 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 12942 T2246 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 12942 T2246 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 12943 T2246 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 12950 T2246 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4] 2> 12960 T2246 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:40583/solr
[junit4:junit4] 2> 12961 T2246 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 12961 T2246 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:40583 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@1c2d03a1
[junit4:junit4] 2> 12962 T2317 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:40583. Will not attempt to authenticate using SASL (Access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 12962 T2246 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 12963 T2317 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:40583, initiating session
[junit4:junit4] 2> 12963 T2248 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:45520
[junit4:junit4] 2> 12963 T2248 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:45520
[junit4:junit4] 2> 12964 T2250 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d502f9e170009 with negotiated timeout 20000 for client /127.0.0.1:45520
[junit4:junit4] 2> 12964 T2317 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:40583, sessionid = 0x13d502f9e170009, negotiated timeout = 20000
[junit4:junit4] 2> 12965 T2318 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c2d03a1 name:ZooKeeperConnection Watcher:127.0.0.1:40583 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 12965 T2246 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 12966 T2251 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d502f9e170009
[junit4:junit4] 2> 12967 T2318 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 12967 T2248 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:45520 which had sessionid 0x13d502f9e170009
[junit4:junit4] 2> 12967 T2246 oaz.ZooKeeper.close Session: 0x13d502f9e170009 closed
[junit4:junit4] 2> 12967 T2246 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 12971 T2246 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:40583/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@cf745ad5
[junit4:junit4] 2> 12973 T2319 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:40583. Will not attempt to authenticate using SASL (Access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 12972 T2246 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 12973 T2319 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:40583, initiating session
[junit4:junit4] 2> 12973 T2248 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:45521
[junit4:junit4] 2> 12973 T2248 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:45521
[junit4:junit4] 2> 12974 T2250 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d502f9e17000a with negotiated timeout 20000 for client /127.0.0.1:45521
[junit4:junit4] 2> 12974 T2319 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:40583, sessionid = 0x13d502f9e17000a, negotiated timeout = 20000
[junit4:junit4] 2> 12975 T2320 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@cf745ad5 name:ZooKeeperConnection Watcher:127.0.0.1:40583/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 12975 T2246 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 12976 T2251 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d502f9e17000a type:create cxid:0x1 zxid:0x90 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 12976 T2251 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d502f9e17000a type:create cxid:0x2 zxid:0x91 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 12977 T2246 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 13980 T2246 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49682_
[junit4:junit4] 2> 13981 T2251 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d502f9e17000a type:delete cxid:0xb zxid:0x92 txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:49682_ Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:49682_
[junit4:junit4] 2> 13981 T2246 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49682_
[junit4:junit4] 2> 13983 T2304 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 13983 T2288 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 13983 T2268 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 13984 T2275 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 13984 T2288 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 13984 T2320 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 13984 T2268 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 13984 T2304 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 13984 T2275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 13988 T2321 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1362849943624/collection1
[junit4:junit4] 2> 13988 T2321 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 13988 T2321 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 13989 T2321 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 13990 T2321 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1362849943624/collection1/'
[junit4:junit4] 2> 13990 T2321 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1362849943624/collection1/lib/README' to classloader
[junit4:junit4] 2> 13991 T2321 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1362849943624/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 14015 T2321 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 14044 T2321 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 14045 T2321 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4] 2> 14050 T2321 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4] 2> 14325 T2269 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 14326 T2269 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "base_url":"http://127.0.0.1:50988",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:50988_",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 14328 T2268 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 14328 T2304 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 14328 T2320 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 14328 T2288 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 14328 T2275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 14374 T2321 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 14379 T2321 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 14381 T2321 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 14390 T2321 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 14393 T2321 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 14397 T2321 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 14398 T2321 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 14398 T2321 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 14398 T2321 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 14399 T2321 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 14400 T2321 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 14400 T2321 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1362849943624/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1362849930771/jetty3/
[junit4:junit4] 2> 14400 T2321 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2059cd3d
[junit4:junit4] 2> 14401 T2321 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 14402 T2321 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-1362849930771/jetty3 forceNew: false
[junit4:junit4] 2> 14402 T2321 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1362849930771/jetty3/index/
[junit4:junit4] 2> 14402 T2321 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1362849930771/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 14403 T2321 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-1362849930771/jetty3/index forceNew: false
[junit4:junit4] 2> 14404 T2321 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@84e5d8cc lockFactory=org.apache.lucene.store.NativeFSLockFactory@d822563d),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> 14404 T2321 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4] 2> 14406 T2321 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 14406 T2321 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 14407 T2321 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 14407 T2321 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 14408 T2321 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 14408 T2321 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 14408 T2321 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 14409 T2321 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 14409 T2321 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 14410 T2321 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 14412 T2321 oass.SolrIndexSearcher.<init> Opening Searcher@8805a1c9 main
[junit4:junit4] 2> 14413 T2321 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1362849930771/jetty3/tlog
[junit4:junit4] 2> 14413 T2321 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 14414 T2321 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 14417 T2322 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@8805a1c9 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 14419 T2321 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 14419 T2321 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 15832 T2269 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 15832 T2269 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "base_url":"http://127.0.0.1:49682",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:49682_",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 15833 T2269 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4] 2> 15833 T2269 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4] 2> 15836 T2268 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 15836 T2288 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 15836 T2275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 15836 T2304 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 15836 T2320 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 16420 T2321 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 16421 T2321 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:49682 collection:collection1 shard:shard1
[junit4:junit4] 2> 16426 T2321 oasc.ZkController.register We are http://127.0.0.1:49682/collection1/ and leader is http://127.0.0.1:40536/collection1/
[junit4:junit4] 2> 16427 T2321 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:49682
[junit4:junit4] 2> 16427 T2321 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 16428 T2321 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C442 name=collection1 org.apache.solr.core.SolrCore@f5a1b591 url=http://127.0.0.1:49682/collection1 node=127.0.0.1:49682_ C442_STATE=coll:collection1 core:collection1 props:{base_url=http://127.0.0.1:49682, state=down, node_name=127.0.0.1:49682_, collection=collection1, core=collection1}
[junit4:junit4] 2> 16429 T2323 C442 P49682 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 16429 T2323 C442 P49682 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 16429 T2321 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 16430 T2323 C442 P49682 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 16430 T2323 C442 P49682 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 16432 T2323 C442 P49682 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 16431 T2246 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 16433 T2246 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 16434 T2246 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 16444 T2246 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 16446 T2246 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 16453 T2284 oasc.CoreContainer.create Creating SolrCore 'onenodecollectioncore' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1362849934551/onenodecollectioncore
[junit4:junit4] 2> 16453 T2284 oasc.ZkController.createCollectionZkNode Check for collection zkNode:onenodecollection
[junit4:junit4] 2> 16454 T2284 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:onenodecollection
[junit4:junit4] 2> 16454 T2284 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4] 2> 16456 T2284 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
[junit4:junit4] 2> 16456 T2284 oascc.SolrZkClient.makePath makePath: /collections/onenodecollection
[junit4:junit4] 2> 16458 T2284 oasc.ZkController.readConfigName Load collection config from:/collections/onenodecollection
[junit4:junit4] 2> 16459 T2284 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1362849934551/onenodecollectioncore/'
[junit4:junit4] 2> 16484 T2284 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 16513 T2284 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 16514 T2284 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4] 2> 16518 T2284 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4] 2> 16830 T2284 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 16835 T2284 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 16837 T2284 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 16848 T2284 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 16852 T2284 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 16856 T2284 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 16857 T2284 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 16857 T2284 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 16858 T2284 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 16859 T2284 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 16860 T2284 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 16860 T2284 oasc.SolrCore.<init> [onenodecollectioncore] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1362849934551/onenodecollectioncore/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1362849930771/onenodecollection/
[junit4:junit4] 2> 16861 T2284 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2059cd3d
[junit4:junit4] 2> 16862 T2284 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 16864 T2284 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1362849930771/onenodecollection forceNew: false
[junit4:junit4] 2> 16865 T2284 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1362849930771/onenodecollection/index/
[junit4:junit4] 2> 16866 T2284 oasc.SolrCore.initIndex WARNING [onenodecollectioncore] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1362849930771/onenodecollection/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 16867 T2284 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1362849930771/onenodecollection/index forceNew: false
[junit4:junit4] 2> 16868 T2284 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@88a5546f lockFactory=org.apache.lucene.store.NativeFSLockFactory@7b65d71b),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> 16869 T2284 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4] 2> 16871 T2284 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 16871 T2284 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 16872 T2284 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 16873 T2284 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 16873 T2284 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 16874 T2284 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 16875 T2284 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 16875 T2284 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 16876 T2284 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 16877 T2284 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 16879 T2284 oass.SolrIndexSearcher.<init> Opening Searcher@d499c35 main
[junit4:junit4] 2> 16880 T2284 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1362849930771/onenodecollection/tlog
[junit4:junit4] 2> 16881 T2284 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 16881 T2284 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 16884 T2325 oasc.SolrCore.registerSearcher [onenodecollectioncore] Registered new searcher Searcher@d499c35 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 16886 T2284 oasc.ZkController.publish publishing core=onenodecollectioncore state=down
[junit4:junit4] 2> 17340 T2269 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 17341 T2269 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "base_url":"http://127.0.0.1:49682",
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:49682_",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 17344 T2269 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "base_url":"http://127.0.0.1:40536",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:40536_",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "roles":"none",
[junit4:junit4] 2> "collection":"onenodecollection",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "core":"onenodecollectioncore"}
[junit4:junit4] 2> 17344 T2269 oasc.Overseer$ClusterStateUpdater.createCollection Create collection onenodecollection with numShards 1
[junit4:junit4] 2> 17345 T2269 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4] 2> 17349 T2268 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 17349 T2304 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 17349 T2320 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 17349 T2275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 17349 T2288 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 17441 T2283 oass.SolrDispatchFilter.handleAdminRequest /admin/cores {state=recovering&onlyIfLeader=true&checkLive=true&action=PREPRECOVERY&version=2&coreNodeName=127.0.0.1:49682__collection1&wt=javabin&nodeName=127.0.0.1:49682_&core=collection1} status=0 QTime=1000
[junit4:junit4] 2> 17887 T2284 oasc.CoreContainer.registerCore registering core: onenodecollectioncore
[junit4:junit4] 2> 17888 T2284 oasc.ZkController.register Register replica - core:onenodecollectioncore address:http://127.0.0.1:40536 collection:onenodecollection shard:shard1
[junit4:junit4] 2> 17890 T2284 oascc.SolrZkClient.makePath makePath: /collections/onenodecollection/leader_elect/shard1/election
[junit4:junit4] 2> 17897 T2251 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d502f9e170006 type:delete cxid:0xa1 zxid:0xad txntype:-1 reqpath:n/a Error Path:/solr/collections/onenodecollection/leaders Error:KeeperErrorCode = NoNode for /solr/collections/onenodecollection/leaders
[junit4:junit4] 2> 17898 T2284 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 17900 T2251 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d502f9e170006 type:create cxid:0xa2 zxid:0xae txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 17902 T2284 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 17903 T2284 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 17904 T2284 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:40536/onenodecollectioncore/
[junit4:junit4] 2> 17905 T2284 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 17906 T2284 oasc.SyncStrategy.syncToMe http://127.0.0.1:40536/onenodecollectioncore/ has no replicas
[junit4:junit4] 2> 17907 T2284 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:40536/onenodecollectioncore/
[junit4:junit4] 2> 17908 T2284 oascc.SolrZkClient.makePath makePath: /collections/onenodecollection/leaders/shard1
[junit4:junit4] 2> 17911 T2251 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d502f9e170006 type:create cxid:0xac zxid:0xb2 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 18854 T2269 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 18865 T2268 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 18865 T2288 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 18865 T2275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 18865 T2304 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 18865 T2320 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 18916 T2284 oasc.ZkController.register We are http://127.0.0.1:40536/onenodecollectioncore/ and leader is http://127.0.0.1:40536/onenodecollectioncore/
[junit4:junit4] 2> 18917 T2284 oasc.ZkController.register No LogReplay needed for core=onenodecollectioncore baseURL=http://127.0.0.1:40536
[junit4:junit4] 2> 18917 T2284 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 18917 T2284 oasc.ZkController.publish publishing core=onenodecollectioncore state=active
[junit4:junit4] 2> 18919 T2284 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 18920 T2284 oass.SolrDispatchFilter.handleAdminRequest /admin/cores {dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1362849930771/onenodecollection&action=CREATE&version=2&numShards=1&name=onenodecollectioncore&wt=javabin&roles=none&collection=onenodecollection} status=0 QTime=2467
[junit4:junit4] 2> 18921 T2246 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: onenodecollection failOnTimeout:true timeout (sec):230
[junit4:junit4] 2> 18921 T2246 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> C442_STATE=coll:collection1 core:collection1 props:{base_url=http://127.0.0.1:49682, state=recovering, node_name=127.0.0.1:49682_, collection=collection1, shard=shard1, core=collection1}
[junit4:junit4] 2> 19442 T2323 C442 P49682 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:40536/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 19443 T2323 C442 P49682 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:49682 START replicas=[http://127.0.0.1:40536/collection1/] nUpdates=100
[junit4:junit4] 2> 19443 T2323 C442 P49682 oasu.PeerSync.sync WARNING no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 19444 T2323 C442 P49682 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 19444 T2323 C442 P49682 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 19445 T2323 C442 P49682 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 19445 T2323 C442 P49682 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 19445 T2323 C442 P49682 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:40536/collection1/. core=collection1
[junit4:junit4] 2> 19446 T2323 C442 P49682 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C443 name=collection1 org.apache.solr.core.SolrCore@f8c8120d url=http://127.0.0.1:40536/collection1 node=127.0.0.1:40536_ C443_STATE=coll:collection1 core:collection1 props:{base_url=http://127.0.0.1:40536, state=active, node_name=127.0.0.1:40536_, collection=collection1, shard=shard1, core=collection1, leader=true}
[junit4:junit4] 2> 19558 T2282 C443 P40536 REQ /get {getVersions=100&qt=/get&distrib=false&version=2&wt=javabin} status=0 QTime=108
[junit4:junit4] 2> 19558 T2281 C443 P40536 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 19560 T2281 C443 P40536 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@12f98e4b lockFactory=org.apache.lucene.store.NativeFSLockFactory@9cb4f53b),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> 19560 T2281 C443 P40536 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4] 2> 19561 T2281 C443 P40536 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@12f98e4b lockFactory=org.apache.lucene.store.NativeFSLockFactory@9cb4f53b),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@12f98e4b lockFactory=org.apache.lucene.store.NativeFSLockFactory@9cb4f53b),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4] 2> 19562 T2281 C443 P40536 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[segments_2]
[junit4:junit4] 2> 19562 T2281 C443 P40536 oass.SolrIndexSearcher.<init> Opening Searcher@5788bb4 realtime
[junit4:junit4] 2> 19563 T2281 C443 P40536 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 19564 T2281 C443 P40536 /update {softCommit=false&commit_end_point=true&commit=true&version=2&waitSearcher=true&wt=javabin&openSearcher=false} {commit=} 0 111
[junit4:junit4] 2> 19566 T2323 C442 P49682 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 19566 T2323 C442 P49682 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 19567 T2283 C443 P40536 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 19568 T2283 C443 P40536 REQ /replication {qt=/replication&version=2&command=indexversion&wt=javabin} status=0 QTime=1
[junit4:junit4] 2> 19568 T2323 C442 P49682 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 19569 T2323 C442 P49682 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 19569 T2323 C442 P49682 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 19571 T2282 C443 P40536 REQ /replication {qt=/replication&version=2&generation=2&command=filelist&wt=javabin} status=0 QTime=1
[junit4:junit4] 2> 19572 T2323 C442 P49682 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 19572 T2323 C442 P49682 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-1362849930771/jetty3/index.20130309225550338 forceNew: false
[junit4:junit4] 2> 19573 T2323 C442 P49682 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@6f529c14 lockFactory=org.apache.lucene.store.NativeFSLockFactory@b7aaaeea) fullCopy=false
[junit4:junit4] 2> 19575 T2281 C443 P40536 REQ /replication {checksum=true&qt=/replication&generation=2&command=filecontent&file=segments_2&wt=filestream} status=0 QTime=0
[junit4:junit4] 2> 19576 T2323 C442 P49682 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 19576 T2323 C442 P49682 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 19577 T2323 C442 P49682 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 19578 T2323 C442 P49682 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@84e5d8cc lockFactory=org.apache.lucene.store.NativeFSLockFactory@d822563d),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@84e5d8cc lockFactory=org.apache.lucene.store.NativeFSLockFactory@d822563d),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4] 2> 19578 T2323 C442 P49682 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[segments_2]
[junit4:junit4] 2> 19578 T2323 C442 P49682 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 19579 T2323 C442 P49682 oass.SolrIndexSearcher.<init> Opening Searcher@3c715197 main
[junit4:junit4] 2> 19579 T2322 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3c715197 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 19580 T2323 C442 P49682 oash.SnapPuller.fetchLatestIndex removing temporary index download directory files MockDirWrapper(org.apache.lucene.store.RAMDirectory@6f529c14 lockFactory=org.apache.lucene.store.NativeFSLockFactory@b7aaaeea)
[junit4:junit4] 2> 19580 T2323 C442 P49682 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 19580 T2323 C442 P49682 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 19580 T2323 C442 P49682 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 19581 T2323 C442 P49682 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 19582 T2323 C442 P49682 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 19923 T2246 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 20372 T2269 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 20373 T2269 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "base_url":"http://127.0.0.1:40536",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:40536_",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "roles":"none",
[junit4:junit4] 2> "collection":"onenodecollection",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"onenodecollectioncore"}
[junit4:junit4] 2> 20375 T2269 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "base_url":"http://127.0.0.1:49682",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:49682_",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 20377 T2268 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 20377 T2275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 20377 T2304 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 20377 T2288 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 20377 T2320 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 20925 T2246 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 20926 T2246 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: onenodecollection
[junit4:junit4] 2> 20926 T2246 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 20935 T2297 oejs.Response.sendError WARNING Committed before 404 Can not find: /onenodecollectioncore/update
[junit4:junit4] 2> 20935 T2246 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:40583 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@66e6c31f
[junit4:junit4] 2> 20936 T2297 oejs.ServletHandler.doHandle WARNING /onenodecollectioncore/update java.lang.IllegalStateException: Committed
[junit4:junit4] 2> at org.eclipse.jetty.server.Response.resetBuffer(Response.java:1136)
[junit4:junit4] 2> at org.eclipse.jetty.server.Response.sendError(Response.java:314)
[junit4:junit4] 2> at org.apache.solr.client.solrj.embedded.JettySolrRunner$Servlet404.service(JettySolrRunner.java:465)
[junit4:junit4] 2> at javax.servlet.http.HttpServlet.service(HttpServlet.java:770)
[junit4:junit4] 2> at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:669)
[junit4:junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1336)
[junit4:junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:379)
[junit4:junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:141)
[junit4:junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
[junit4:junit4] 2> at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:135)
[junit4:junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
[junit4:junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453)
[junit4:junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.GzipHandler.handle(GzipHandler.java:275)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1074)
[junit4:junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:382)
[junit4:junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1006)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
[junit4:junit4] 2> at org.eclipse.jetty.server.Server.handle(Server.java:365)
[junit4:junit4] 2> at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:485)
[junit4:junit4] 2> at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:937)
[junit4:junit4] 2> at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:998)
[junit4:junit4] 2> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:948)
[junit4:junit4] 2> at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
[junit4:junit4] 2> at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
[junit4:junit4] 2> at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:628)
[junit4:junit4] 2> at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
[junit4:junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4:junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:780)
[junit4:junit4] 2>
[junit4:junit4] 2> 20936 T2246 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 20937 T2328 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:40583. Will not attempt to authenticate using SASL (Access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 20937 T2328 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:40583, initiating session
[junit4:junit4] 2> 20937 T2248 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:45535
[junit4:junit4] 2> 20938 T2248 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:45535
[junit4:junit4] 2> 20939 T2250 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d502f9e17000b with negotiated timeout 10000 for client /127.0.0.1:45535
[junit4:junit4] 2> 20939 T2328 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:40583, sessionid = 0x13d502f9e17000b, negotiated timeout = 10000
[junit4:junit4] 2> 20939 T2329 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@66e6c31f name:ZooKeeperConnection Watcher:127.0.0.1:40583 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 20940 T2246 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 20965 T2251 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d502f9e17000b
[junit4:junit4] 2> 20966 T2329 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 20966 T2246 oaz.ZooKeeper.close Session: 0x13d502f9e17000b closed
[junit4:junit4] 2> 20966 T2248 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:45535 which had sessionid 0x13d502f9e17000b
[junit4:junit4] 2> 20967 T2251 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d502f9e170004
[junit4:junit4] 2> 20967 T2275 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 20967 T2248 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:45509 which had sessionid 0x13d502f9e170004
[junit4:junit4] 2> 20967 T2246 oaz.ZooKeeper.close Session: 0x13d502f9e170004 closed
[junit4:junit4] 2> 20971 T2246 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 46733
[junit4:junit4] 2> 20971 T2246 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1047548841
[junit4:junit4] 2> 21881 T2269 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 21882 T2269 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "base_url":"http://127.0.0.1:46733",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:46733__collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:46733_",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 21885 T2268 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 21885 T2320 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 21885 T2304 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 21885 T2288 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 22972 T2246 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 22973 T2246 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 22977 T2246 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@ff65327b
[junit4:junit4] 2> 22982 T2246 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 22983 T2246 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 22984 T2246 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 22984 T2246 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 22985 T2246 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 22986 T2246 oasc.CachingDirectoryFactory.close Closing directory when closing factory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-1362849930771/control/data
[junit4:junit4] 2> 22986 T2246 oasc.CachingDirectoryFactory.closeDirectory Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-1362849930771/control/data
[junit4:junit4] 2> 22987 T2246 oasc.CachingDirectoryFactory.close Closing directory when closing factory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-1362849930771/control/data/index
[junit4:junit4] 2> 22987 T2246 oasc.CachingDirectoryFactory.closeDirectory Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-1362849930771/control/data/index
[junit4:junit4] 2> 22988 T2269 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=89315733063270403-127.0.0.1:46733_-n_0000000000) am no longer a leader.
[junit4:junit4] 2> 22988 T2251 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d502f9e170003
[junit4:junit4] 2> 22990 T2304 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 22990 T2248 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:45503 which had sessionid 0x13d502f9e170003
[junit4:junit4] 2> 22990 T2246 oaz.ZooKeeper.close Session: 0x13d502f9e170003 closed
[junit4:junit4] 2> 22991 T2320 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 22991 T2251 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d502f9e170006 type:delete cxid:0xb9 zxid:0xce txntype:-1 reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4] 2> 22991 T2320 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 22991 T2304 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 22991 T2288 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 22993 T2288 oasc.Overseer.start Overseer (id=89315733063270406-127.0.0.1:40536_-n_0000000001) starting
[junit4:junit4] 2> 22994 T2251 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d502f9e170006 type:create cxid:0xbe zxid:0xd0 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 22995 T2251 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d502f9e170006 type:create cxid:0xbf zxid:0xd1 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 22996 T2251 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d502f9e170006 type:create cxid:0xc0 zxid:0xd2 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 22997 T2251 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d502f9e170006 type:create cxid:0xc1 zxid:0xd3 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 22997 T2331 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 22998 T2288 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 22998 T2288 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 22998 T2330 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 23000 T2330 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 23000 T2330 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "base_url":"http://127.0.0.1:46733",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:46733_",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 23003 T2304 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 23003 T2320 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 23003 T2288 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 23011 T2246 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
[junit4:junit4] 2> 23064 T2246 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 40536
[junit4:junit4] 2> 23065 T2246 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=654594879
[junit4:junit4] 2> 24290 T2268 oascc.ZkStateReader$3.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 24290 T2268 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 24291 T2268 oascc.ZkStateReader$2.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 24291 T2268 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 24507 T2330 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 24509 T2330 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "base_url":"http://127.0.0.1:40536",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:40536__onenodecollectioncore",
[junit4:junit4] 2> "node_name":"127.0.0.1:40536_",
[junit4:junit4] 2> "roles":"none",
[junit4:junit4] 2> "collection":"onenodecollection",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"onenodecollectioncore"}
[junit4:junit4] 2> 24513 T2330 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "base_url":"http://127.0.0.1:40536",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:40536__collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:40536_",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 24518 T2304 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 24519 T2320 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 24518 T2288 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 26068 T2246 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 26069 T2246 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 26072 T2246 oasc.ZkController.publish publishing core=onenodecollectioncore state=down
[junit4:junit4] 2> 26075 T2246 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@f8c8120d
[junit4:junit4] 2> 26080 T2246 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 26081 T2246 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 26082 T2246 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 26083 T2246 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 26084 T2246 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 26085 T2246 oasc.CachingDirectoryFactory.close Closing directory when closing factory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-1362849930771/jetty1/index
[junit4:junit4] 2> 26086 T2246 oasc.CachingDirectoryFactory.closeDirectory Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-1362849930771/jetty1/index
[junit4:junit4] 2> 26086 T2246 oasc.CachingDirectoryFactory.close Closing directory when closing factory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-1362849930771/jetty1
[junit4:junit4] 2> 26086 T2246 oasc.CachingDirectoryFactory.closeDirectory Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-1362849930771/jetty1
[junit4:junit4] 2> 26087 T2246 oasc.SolrCore.close [onenodecollectioncore] CLOSING SolrCore org.apache.solr.core.SolrCore@319f1251
[junit4:junit4] 2> 26091 T2246 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 26092 T2246 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 26092 T2246 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 26092 T2246 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 26093 T2246 oasc.SolrCore.closeSearcher [onenodecollectioncore] Closing main searcher on request.
[junit4:junit4] 2> 26094 T2246 oasc.CachingDirectoryFactory.close Closing directory when closing factory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1362849930771/onenodecollection
[junit4:junit4] 2> 26094 T2246 oasc.CachingDirectoryFactory.closeDirectory Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1362849930771/onenodecollection
[junit4:junit4] 2> 26095 T2246 oasc.CachingDirectoryFactory.close Closing directory when closing factory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1362849930771/onenodecollection/index
[junit4:junit4] 2> 26095 T2246 oasc.CachingDirectoryFactory.closeDirectory Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1362849930771/onenodecollection/index
[junit4:junit4] 2> 26096 T2330 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=89315733063270406-127.0.0.1:40536_-n_0000000001) am no longer a leader.
[junit4:junit4] 2> 26096 T2251 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d502f9e170006
[junit4:junit4] 2> 26097 T2304 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 26097 T2248 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:45511 which had sessionid 0x13d502f9e170006
[junit4:junit4] 2> 26097 T2246 oaz.ZooKeeper.close Session: 0x13d502f9e170006 closed
[junit4:junit4] 2> 26098 T2320 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 26098 T2320 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 26099 T2304 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 26099 T2251 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d502f9e170008 type:delete cxid:0x74 zxid:0xe4 txntype:-1 reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4] 2> 26100 T2304 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 26100 T2251 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d502f9e17000a type:delete cxid:0x58 zxid:0xe5 txntype:-1 reqpath:n/a Error Path:/solr/collections/collection1/leaders/shard1 Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders/shard1
[junit4:junit4] 2> 26100 T2320 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 26101 T2251 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d502f9e17000a type:create cxid:0x59 zxid:0xe7 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 26101 T2304 oasc.Overseer.start Overseer (id=89315733063270408-127.0.0.1:50988_-n_0000000002) starting
[junit4:junit4] 2> 26102 T2251 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d502f9e170008 type:create cxid:0x79 zxid:0xe9 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 26102 T2320 oasc.ShardLeaderElectionContext.shouldIBeLeader Checking if I should try and be the leader.
[junit4:junit4] 2> 26102 T2320 oasc.ShardLeaderElectionContext.shouldIBeLeader My last published State was Active, it's okay to be the leader.
[junit4:junit4] 2> 26102 T2251 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d502f9e170008 type:create cxid:0x7a zxid:0xea txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 26102 T2320 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 26103 T2320 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:49682/collection1/
[junit4:junit4] 2> 26103 T2251 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d502f9e170008 type:create cxid:0x7b zxid:0xeb txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 26103 T2320 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 26103 T2320 oasc.SyncStrategy.syncToMe http://127.0.0.1:49682/collection1/ has no replicas
[junit4:junit4] 2> 26104 T2320 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:49682/collection1/
[junit4:junit4] 2> 26104 T2251 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d502f9e170008 type:create cxid:0x7c zxid:0xec txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 26104 T2320 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 26104 T2333 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 26105 T2332 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 26106 T2251 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d502f9e17000a type:create cxid:0x61 zxid:0xee txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 26107 T2332 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 26107 T2332 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "base_url":"http://127.0.0.1:40536",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:40536_",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 26109 T2332 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "base_url":"http://127.0.0.1:40536",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:40536_",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "roles":"none",
[junit4:junit4] 2> "collection":"onenodecollection",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"onenodecollectioncore"}
[junit4:junit4] 2> 26116 T2304 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 26116 T2320 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 26118 T2246 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
[junit4:junit4] 2> 26171 T2246 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 50988
[junit4:junit4] 2> 26171 T2246 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=-1605039329
[junit4:junit4] 2> 27173 T2246 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 27174 T2246 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 27176 T2246 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@278d8498
[junit4:junit4] 2> 27179 T2246 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 27179 T2246 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 27180 T2246 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 27180 T2246 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 27181 T2246 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 27181 T2246 oasc.CachingDirectoryFactory.close Closing directory when closing factory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-1362849930771/jetty2
[junit4:junit4] 2> 27181 T2246 oasc.CachingDirectoryFactory.closeDirectory Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-1362849930771/jetty2
[junit4:junit4] 2> 27182 T2246 oasc.CachingDirectoryFactory.close Closing directory when closing factory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-1362849930771/jetty2/index
[junit4:junit4] 2> 27182 T2246 oasc.CachingDirectoryFactory.closeDirectory Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-1362849930771/jetty2/index
[junit4:junit4] 2> 27183 T2332 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=89315733063270408-127.0.0.1:50988_-n_0000000002) am no longer a leader.
[junit4:junit4] 2> 27183 T2251 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d502f9e170008
[junit4:junit4] 2> 27185 T2304 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 27185 T2248 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:45516 which had sessionid 0x13d502f9e170008
[junit4:junit4] 2> 27185 T2304 oascc.ZkStateReader$2.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 27185 T2246 oaz.ZooKeeper.close Session: 0x13d502f9e170008 closed
[junit4:junit4] 2> 27186 T2320 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4:junit4] 2> 27185 T2304 oascc.ZkStateReader$3.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 27186 T2320 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 27186 T2304 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 27187 T2251 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d502f9e17000a type:delete cxid:0x69 zxid:0x100 txntype:-1 reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4] 2> 27188 T2320 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 27189 T2320 oasc.Overseer.start Overseer (id=89315733063270410-127.0.0.1:49682_-n_0000000003) starting
[junit4:junit4] 2> 27190 T2251 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d502f9e17000a type:create cxid:0x6e zxid:0x102 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 27191 T2251 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d502f9e17000a type:create cxid:0x6f zxid:0x103 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 27191 T2251 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d502f9e17000a type:create cxid:0x70 zxid:0x104 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 27192 T2251 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d502f9e17000a type:create cxid:0x71 zxid:0x105 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 27193 T2335 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 27193 T2334 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 27194 T2334 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 27195 T2334 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "base_url":"http://127.0.0.1:50988",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:50988__collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:50988_",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 27197 T2334 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "base_url":"http://127.0.0.1:50988",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:50988_",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 27200 T2320 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 27206 T2246 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
[junit4:junit4] 2> 27258 T2246 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 49682
[junit4:junit4] 2> 27259 T2246 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=-65400496
[junit4:junit4] 2> 27397 T2288 oascc.ZkStateReader$3.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 27397 T2288 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 27397 T2288 oascc.ZkStateReader$2.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 27398 T2288 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 28261 T2246 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 28261 T2246 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 28264 T2246 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@f5a1b591
[junit4:junit4] 2> 28269 T2246 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 28270 T2246 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 28271 T2246 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 28271 T2246 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 28272 T2246 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 28273 T2246 oasc.CachingDirectoryFactory.close Closing directory when closing factory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-1362849930771/jetty3
[junit4:junit4] 2> 28273 T2246 oasc.CachingDirectoryFactory.closeDirectory Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-1362849930771/jetty3
[junit4:junit4] 2> 28274 T2246 oasc.CachingDirectoryFactory.close Closing directory when closing factory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-1362849930771/jetty3/index.20130309225550338
[junit4:junit4] 2> 28274 T2246 oasc.CachingDirectoryFactory.closeDirectory Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-1362849930771/jetty3/index.20130309225550338
[junit4:junit4] 2> 28275 T2246 oasc.CachingDirectoryFactory.close Closing directory when closing factory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-1362849930771/jetty3/index
[junit4:junit4] 2> 28275 T2246 oasc.CachingDirectoryFactory.closeDirectory Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-1362849930771/jetty3/index
[junit4:junit4] 2> 28276 T2334 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=89315733063270410-127.0.0.1:49682_-n_0000000003) am no longer a leader.
[junit4:junit4] 2> 28276 T2251 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d502f9e17000a
[junit4:junit4] 2> 28277 T2248 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:45521 which had sessionid 0x13d502f9e17000a
[junit4:junit4] 2> 28277 T2246 oaz.ZooKeeper.close Session: 0x13d502f9e17000a closed
[junit4:junit4] 2> 28298 T2246 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
[junit4:junit4] 2> 28353 T2246 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
[junit4:junit4] 2> 28353 T2246 oazs.ZooKeeperServer.shutdown shutting down
[junit4:junit4] 2> 28354 T2246 oazs.SessionTrackerImpl.shutdown Shutting down
[junit4:junit4] 2> 28354 T2246 oazs.PrepRequestProcessor.shutdown Shutting down
[junit4:junit4] 2> 28355 T2251 oazs.PrepRequestProcessor.run PrepRequestProcessor exited loop!
[junit4:junit4] 2> 28355 T2246 oazs.SyncRequestProcessor.shutdown Shutting down
[junit4:junit4] 2> 28355 T2250 oazs.SyncRequestProcessor.run SyncRequestProcessor exited!
[junit4:junit4] 2> 28355 T2246 oazs.FinalRequestProcessor.shutdown shutdown of request processor complete
[junit4:junit4] 2> 28356 T2246 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:40583 40583
[junit4:junit4] 2> 28357 T2248 oazs.NIOServerCnxnFactory.run NIOServerCnxn factory exited run method
[junit4:junit4] 2> 28358 T2246 oazs.ZooKeeperServer.shutdown shutting down
[junit4:junit4] 2> 28358 T2246 oazs.SessionTrackerImpl.shutdown Shutting down
[junit4:junit4] 2> 28358 T2246 oazs.PrepRequestProcessor.shutdown Shutting down
[junit4:junit4] 2> 28359 T2246 oazs.SyncRequestProcessor.shutdown Shutting down
[junit4:junit4] 2> 28359 T2246 oazs.FinalRequestProcessor.shutdown shutdown of request processor complete
[junit4:junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4:junit4] 1> / (2)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1>
[junit4:junit4] 1> /solr (7)
[junit4:junit4] 1> /solr/configs (1)
[junit4:junit4] 1> /solr/configs/conf1 (9)
[junit4:junit4] 1> /solr/configs/conf1/old_synonyms.txt (0)
[junit4:junit4] 1> DATA: ...supressed...
[junit4:junit4] 1> /solr/configs/conf1/protwords.txt (0)
[junit4:junit4] 1> DATA: ...supressed...
[junit4:junit4] 1> /solr/configs/conf1/stopwords.txt (0)
[junit4:junit4] 1> DATA: ...supressed...
[junit4:junit4] 1> /solr/configs/conf1/synonyms.txt (0)
[junit4:junit4] 1> DATA: ...supressed...
[junit4:junit4] 1> /solr/configs/conf1/schema.xml (0)
[junit4:junit4] 1> DATA: ...supressed...
[junit4:junit4] 1> /solr/configs/conf1/currency.xml (0)
[junit4:junit4] 1> DATA: ...supressed...
[junit4:junit4] 1> /solr/configs/conf1/open-exchange-rates.json (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "disclaimer": "This data is not real, it was synthetically created to match currency.xml. It is modeled after the data format available from openexchangerates.org. See https://openexchangerates.org/documentation for details",
[junit4:junit4] 1> "license": "http://www.apache.org/licenses/LICENSE-2.0",
[junit4:junit4] 1> "timestamp": 1332070464,
[junit4:junit4] 1>
[junit4:junit4] 1>
[junit4:junit4] 1> "IMPORTANT NOTE": "In order for tests to work, this data must be kept in sync with ./currency.xml",
[junit4:junit4] 1>
[junit4:junit4] 1>
[junit4:junit4] 1> "base": "USD",
[junit4:junit4] 1> "rates": {
[junit4:junit4] 1> "USD": 1,
[junit4:junit4] 1> "JPY": 81.29,
[junit4:junit4] 1> "EUR": 2.5,
[junit4:junit4] 1> "GBP": 0.5,
[junit4:junit4] 1> "MXN": 2.0
[junit4:junit4] 1> }
[junit4:junit4] 1> }
[junit4:junit4] 1>
[junit4:junit4] 1> /solr/configs/conf1/solrconfig.xml (0)
[junit4:junit4] 1> DATA: ...supressed...
[junit4:junit4] 1> /solr/configs/conf1/mapping-ISOLatin1Accent.txt (0)
[junit4:junit4] 1> DATA: ...supressed...
[junit4:junit4] 1> /solr/overseer (3)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1>
[junit4:junit4] 1> /solr/overseer/queue (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1>
[junit4:junit4] 1> /solr/overseer/collection-queue-work (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1>
[junit4:junit4] 1> /solr/overseer/queue-work (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1>
[junit4:junit4] 1> /solr/overseer_elect (2)
[junit4:junit4] 1> /solr/overseer_elect/leader (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"id":"89315733063270403-127.0.0.1:46733_-n_0000000000"}
[junit4:junit4] 1> /solr/overseer_elect/election (4)
[junit4:junit4] 1> /solr/overseer_elect/election/89315733063270408-127.0.0.1:50988_-n_0000000002 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89315733063270403-127.0.0.1:46733_-n_0000000000 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89315733063270406-127.0.0.1:40536_-n_0000000001 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89315733063270410-127.0.0.1:49682_-n_0000000003 (0)
[junit4:junit4] 1> /solr/collections (3)
[junit4:junit4] 1> /solr/collections/control_collection (3)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"configName":"conf1"}
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect (1)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect/shard1 (1)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect/shard1/election (1)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect/shard1/election/89315733063270403-127.0.0.1:46733__collection1-n_0000000000 (0)
[junit4:junit4] 1> /solr/collections/control_collection/leaders (1)
[junit4:junit4] 1> /solr/collections/control_collection/leaders/shard1 (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "base_url":"http://127.0.0.1:46733",
[junit4:junit4] 1> "node_name":"127.0.0.1:46733_",
[junit4:junit4] 1> "core":"collection1"}
[junit4:junit4] 1> /solr/collections/control_collection/shards (0)
[junit4:junit4] 1> /solr/collections/collection1 (3)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"configName":"conf1"}
[junit4:junit4] 1> /solr/collections/collection1/leader_elect (2)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1 (1)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election (2)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89315733063270406-127.0.0.1:40536__collection1-n_0000000000 (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89315733063270410-127.0.0.1:49682__collection1-n_0000000001 (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard2 (1)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard2/election (1)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard2/election/89315733063270408-127.0.0.1:50988__collection1-n_0000000000 (0)
[junit4:junit4] 1> /solr/collections/collection1/leaders (2)
[junit4:junit4] 1> /solr/collections/collection1/leaders/shard1 (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "base_url":"http://127.0.0.1:40536",
[junit4:junit4] 1> "node_name":"127.0.0.1:40536_",
[junit4:junit4] 1> "core":"collection1"}
[junit4:junit4] 1> /solr/collections/collection1/leaders/shard2 (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "base_url":"http://127.0.0.1:50988",
[junit4:junit4] 1> "node_name":"127.0.0.1:50988_",
[junit4:junit4] 1> "core":"collection1"}
[junit4:junit4] 1> /solr/collections/collection1/shards (0)
[junit4:junit4] 1> /solr/collections/onenodecollection (2)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "router":"compositeId",
[junit4:junit4] 1> "configName":"conf1"}
[junit4:junit4] 1> /solr/collections/onenodecollection/leader_elect (1)
[junit4:junit4] 1> /solr/collections/onenodecollection/leader_elect/shard1 (1)
[junit4:junit4] 1> /solr/collections/onenodecollection/leader_elect/shard1/election (1)
[junit4:junit4] 1> /solr/collections/onenodecollection/leader_elect/shard1/election/89315733063270406-127.0.0.1:40536__onenodecollectioncore-n_0000000000 (0)
[junit4:junit4] 1> /solr/collections/onenodecollection/leaders (1)
[junit4:junit4] 1> /solr/collections/onenodecollection/leaders/shard1 (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "base_url":"http://127.0.0.1:40536",
[junit4:junit4] 1> "node_name":"127.0.0.1:40536_",
[junit4:junit4] 1> "core":"onenodecollectioncore"}
[junit4:junit4] 1> /solr/aliases.json (0)
[junit4:junit4] 1> /solr/live_nodes (4)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:40536_ (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:49682_ (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:50988_ (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:46733_ (0)
[junit4:junit4] 1> /solr/clusterstate.json (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "control_collection":{
[junit4:junit4] 1> "shards":{"shard1":{
[junit4:junit4] 1> "range":"80000000-7fffffff",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "replicas":{"127.0.0.1:46733__collection1":{
[junit4:junit4] 1> "base_url":"http://127.0.0.1:46733",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "node_name":"127.0.0.1:46733_",
[junit4:junit4] 1> "collection":"control_collection",
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "leader":"true"}}}},
[junit4:junit4] 1> "router":"compositeId"},
[junit4:junit4] 1> "onenodecollection":{
[junit4:junit4] 1> "shards":{"shard1":{
[junit4:junit4] 1> "range":"80000000-7fffffff",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "replicas":{"127.0.0.1:40536__onenodecollectioncore":{
[junit4:junit4] 1> "base_url":"http://127.0.0.1:40536",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "node_name":"127.0.0.1:40536_",
[junit4:junit4] 1> "roles":"none",
[junit4:junit4] 1> "collection":"onenodecollection",
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "core":"onenodecollectioncore",
[junit4:junit4] 1> "leader":"true"}}}},
[junit4:junit4] 1> "router":"compositeId"},
[junit4:junit4] 1> "collection1":{
[junit4:junit4] 1> "shards":{
[junit4:junit4] 1> "shard1":{
[junit4:junit4] 1> "range":"80000000-ffffffff",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "replicas":{
[junit4:junit4] 1> "127.0.0.1:40536__collection1":{
[junit4:junit4] 1> "base_url":"http://127.0.0.1:40536",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "node_name":"127.0.0.1:40536_",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "leader":"true"},
[junit4:junit4] 1> "127.0.0.1:49682__collection1":{
[junit4:junit4] 1> "base_url":"http://127.0.0.1:49682",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "node_name":"127.0.0.1:49682_",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "core":"collection1"}}},
[junit4:junit4] 1> "shard2":{
[junit4:junit4] 1> "range":"0-7fffffff",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "replicas":{"127.0.0.1:50988__collection1":{
[junit4:junit4] 1> "base_url":"http://127.0.0.1:50988",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "node_name":"127.0.0.1:50988_",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "shard":"shard2",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "leader":"true"}}}},
[junit4:junit4] 1> "router":"compositeId"}}
[junit4:junit4] 1> /zookeeper (1)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1>
[junit4:junit4] 1>
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=BasicDistributedZk2Test -Dtests.method=testDistribSearch -Dtests.seed=FB5781DC2C96ADC -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ml_IN -Dtests.timezone=IST -Dtests.file.encoding=UTF-8
[junit4:junit4] ERROR 28.4s J1 | BasicDistributedZk2Test.testDistribSearch <<<
[junit4:junit4] > Throwable #1: org.apache.solr.common.SolrException: Server at http://127.0.0.1:50988/onenodecollectioncore returned non ok status:404, message:Can not find: /onenodecollectioncore/update
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([FB5781DC2C96ADC:8E53F605B5960AE0]:0)
[junit4:junit4] > at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:373)
[junit4:junit4] > at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:181)
[junit4:junit4] > at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:117)
[junit4:junit4] > at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:116)
[junit4:junit4] > at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:102)
[junit4:junit4] > at org.apache.solr.cloud.BasicDistributedZk2Test.testNodeWithoutCollectionForwarding(BasicDistributedZk2Test.java:200)
[junit4:junit4] > at org.apache.solr.cloud.BasicDistributedZk2Test.doTest(BasicDistributedZk2Test.java:89)
[junit4:junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:806)
[junit4:junit4] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit4:junit4] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:88)
[junit4:junit4] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
[junit4:junit4] > at java.lang.reflect.Method.invoke(Method.java:613)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
[junit4:junit4] > at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
[junit4:junit4] > at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
[junit4:junit4] > at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4:junit4] > at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
[junit4:junit4] > at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
[junit4:junit4] > at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
[junit4:junit4] > at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
[junit4:junit4] > at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4] > at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
[junit4:junit4] > at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4:junit4] > at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
[junit4:junit4] > at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:780)
[junit4:junit4] 2> 28368 T2246 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4] 2> 28372 T2245 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 3 leaked thread(s).
[junit4:junit4] 2> 29234 T2249 oazs.SessionTrackerImpl.run SessionTrackerImpl exited loop!
[junit4:junit4] 2> 29577 T2320 oascc.ZkStateReader$3.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 29578 T2320 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 29578 T2320 oascc.ZkStateReader$2.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 29579 T2320 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> NOTE: test params are: codec=Lucene42: {}, docValues:{}, sim=DefaultSimilarity, locale=ml_IN, timezone=IST
[junit4:junit4] 2> NOTE: Linux 3.2.0-38-generic x86/IBM Corporation 1.7.0 (32-bit)/cpus=8,threads=1,free=120525696,total=231473152
[junit4:junit4] 2> NOTE: All tests run in this JVM: [SuggesterTSTTest, TestComponentsName, TestSystemIdResolver, StandardRequestHandlerTest, TestAtomicUpdateErrorCases, CollectionsAPIDistributedZkTest, RequiredFieldsTest, StatsComponentTest, DisMaxRequestHandlerTest, SpellCheckCollatorTest, CopyFieldTest, TestHashPartitioner, TestPostingsSolrHighlighter, SpellPossibilityIteratorTest, TestCSVLoader, OpenCloseCoreStressTest, HardAutoCommitTest, TestUpdate, TestMergePolicyConfig, TimeZoneUtilsTest, TestPartialUpdateDeduplication, OpenExchangeRatesOrgProviderTest, SuggesterWFSTTest, BadIndexSchemaTest, SortByFunctionTest, TestUtils, FullSolrCloudDistribCmdsTest, TestDistributedGrouping, ChaosMonkeySafeLeaderTest, DOMUtilTest, SolrIndexConfigTest, MultiTermTest, TestAnalyzedSuggestions, TestSolr4Spatial, TestMultiCoreConfBootstrap, SpellCheckComponentTest, TestWordDelimiterFilterFactory, TestRecovery, DirectSolrSpellCheckerTest, TestQueryTypes, TestBinaryField, FieldAnalysisRequestHandlerTest, TestFiltering, XsltUpdateRequestHandlerTest, TestJmxMonitoredMap, TestIndexingPerformance, DocValuesMultiTest, TestFastOutputStream, DocumentAnalysisRequestHandlerTest, DirectUpdateHandlerTest, SimplePostToolTest, TestCollationField, UpdateParamsTest, DateMathParserTest, TestFieldCollectionResource, ZkSolrClientTest, TestSolrXMLSerializer, CurrencyFieldOpenExchangeTest, BasicDistributedZk2Test]
[junit4:junit4] Completed on J1 in 29.64s, 1 test, 1 error <<< FAILURES!
[...truncated 476 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:381: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:374: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:183: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:449: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1213: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:877: There were test failures: 269 suites, 1137 tests, 1 error, 14 ignored (8 assumptions)
Total time: 45 minutes 14 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Description set: Java: 32bit/ibm-j9-jdk7
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.6.0_41) - Build #
4641 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/4641/
Java: 64bit/jdk1.6.0_41 -XX:+UseParallelGC
2 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.RecoveryZkTest
Error Message:
Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=573, name=coreLoadExecutor-278-thread-1, state=RUNNABLE, group=TGRP-RecoveryZkTest], registration stack trace below.
Stack Trace:
com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=573, name=coreLoadExecutor-278-thread-1, state=RUNNABLE, group=TGRP-RecoveryZkTest], registration stack trace below.
at java.lang.Thread.getStackTrace(Thread.java:1479)
at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:150)
at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:520)
at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:986)
at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:878)
at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:870)
at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:33)
at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:266)
at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:222)
at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:244)
at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:458)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:733)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:619)
at org.apache.solr.core.CoreContainer.createFromZk(CoreContainer.java:874)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:957)
at org.apache.solr.core.CoreContainer$2.call(CoreContainer.java:580)
at org.apache.solr.core.CoreContainer$2.call(CoreContainer.java:575)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.AssertionError: Directory not closed: MockDirWrapper(org.apache.lucene.store.RAMDirectory@559be2c8 lockFactory=org.apache.lucene.store.NativeFSLockFactory@79492ac)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:602)
at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:599)
at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:167)
at com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(RandomizedRunner.java:615)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
... 1 more
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.RecoveryZkTest
Error Message:
Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=575, name=RecoveryThread, state=RUNNABLE, group=TGRP-RecoveryZkTest], registration stack trace below.
Stack Trace:
com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=575, name=RecoveryThread, state=RUNNABLE, group=TGRP-RecoveryZkTest], registration stack trace below.
at java.lang.Thread.getStackTrace(Thread.java:1479)
at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:150)
at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:520)
at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:986)
at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:878)
at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:870)
at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:33)
at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:266)
at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:222)
at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:383)
at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:281)
at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:153)
at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:409)
at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:223)
Caused by: java.lang.AssertionError: Directory not closed: MockDirWrapper(org.apache.lucene.store.RAMDirectory@661cd479 lockFactory=org.apache.lucene.store.NativeFSLockFactory@63dbf1b0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:602)
at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:599)
at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:167)
at com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(RandomizedRunner.java:615)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:662)
Build Log:
[...truncated 8306 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.RecoveryZkTest
[junit4:junit4] 2> 0 T514 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /ev_p/
[junit4:junit4] 2> 4 T514 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-RecoveryZkTest-1362860556925
[junit4:junit4] 2> 5 T514 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 5 T515 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 6 T515 oazs.ZooKeeperServer.setTickTime tickTime set to 1000
[junit4:junit4] 2> 6 T515 oazs.ZooKeeperServer.setMinSessionTimeout minSessionTimeout set to -1
[junit4:junit4] 2> 6 T515 oazs.ZooKeeperServer.setMaxSessionTimeout maxSessionTimeout set to -1
[junit4:junit4] 2> 6 T515 oazs.NIOServerCnxnFactory.configure binding to port 0.0.0.0/0.0.0.0:0
[junit4:junit4] 2> 7 T515 oazsp.FileTxnSnapLog.save Snapshotting: 0x0 to /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-1362860556924/zookeeper/server1/data/version-2/snapshot.0
[junit4:junit4] 2> 105 T514 oasc.ZkTestServer.run start zk server on port:48453
[junit4:junit4] 2> 106 T514 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:48453 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@29794f5b
[junit4:junit4] 2> 107 T514 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 108 T520 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:48453. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4] 2> 108 T520 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:48453, initiating session
[junit4:junit4] 2> 109 T516 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:43080
[junit4:junit4] 2> 109 T516 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:43080
[junit4:junit4] 2> 110 T518 oazsp.FileTxnLog.append Creating new log file: log.1
[junit4:junit4] 2> 200 T518 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d50d1c27f0000 with negotiated timeout 10000 for client /127.0.0.1:43080
[junit4:junit4] 2> 201 T520 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:48453, sessionid = 0x13d50d1c27f0000, negotiated timeout = 10000
[junit4:junit4] 2> 201 T521 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@29794f5b name:ZooKeeperConnection Watcher:127.0.0.1:48453 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 201 T514 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 201 T514 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 204 T519 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d50d1c27f0000
[junit4:junit4] 2> 205 T521 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 205 T514 oaz.ZooKeeper.close Session: 0x13d50d1c27f0000 closed
[junit4:junit4] 2> 205 T516 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:43080 which had sessionid 0x13d50d1c27f0000
[junit4:junit4] 2> 205 T514 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:48453/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@3f2e7707
[junit4:junit4] 2> 206 T514 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 206 T522 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:48453. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4] 2> 207 T522 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:48453, initiating session
[junit4:junit4] 2> 207 T516 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:43081
[junit4:junit4] 2> 207 T516 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:43081
[junit4:junit4] 2> 208 T522 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:48453, sessionid = 0x13d50d1c27f0001, negotiated timeout = 10000
[junit4:junit4] 2> 208 T523 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3f2e7707 name:ZooKeeperConnection Watcher:127.0.0.1:48453/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 208 T518 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d50d1c27f0001 with negotiated timeout 10000 for client /127.0.0.1:43081
[junit4:junit4] 2> 208 T514 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 209 T514 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 211 T514 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 213 T514 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 215 T514 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 217 T514 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 217 T514 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 222 T514 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 223 T514 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 226 T514 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 227 T514 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 229 T514 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 229 T514 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 231 T514 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 232 T514 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 234 T514 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 234 T514 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 237 T514 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 237 T514 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 239 T514 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 239 T514 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 241 T514 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 241 T514 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 243 T519 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d50d1c27f0001
[junit4:junit4] 2> 243 T523 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 243 T514 oaz.ZooKeeper.close Session: 0x13d50d1c27f0001 closed
[junit4:junit4] 2> 244 T516 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:43081 which had sessionid 0x13d50d1c27f0001
[junit4:junit4] 2> 299 T514 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4] 2> 301 T514 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:33529
[junit4:junit4] 2> 302 T514 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 302 T514 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 302 T514 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1362860557164
[junit4:junit4] 2> 303 T514 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1362860557164/solr.xml
[junit4:junit4] 2> 303 T514 oasc.CoreContainer.<init> New CoreContainer 1033421452
[junit4:junit4] 2> 303 T514 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1362860557164/'
[junit4:junit4] 2> 304 T514 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1362860557164/'
[junit4:junit4] 2> 319 T514 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 319 T514 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 319 T514 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 320 T514 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 320 T514 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 320 T514 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 320 T514 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 321 T514 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 321 T514 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 321 T514 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 327 T514 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4] 2> 336 T514 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:48453/solr
[junit4:junit4] 2> 337 T514 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 337 T514 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:48453 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@2050de0
[junit4:junit4] 2> 337 T514 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 338 T533 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:48453. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4] 2> 338 T533 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:48453, initiating session
[junit4:junit4] 2> 338 T516 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:43082
[junit4:junit4] 2> 339 T516 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:43082
[junit4:junit4] 2> 339 T518 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d50d1c27f0002 with negotiated timeout 20000 for client /127.0.0.1:43082
[junit4:junit4] 2> 339 T533 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:48453, sessionid = 0x13d50d1c27f0002, negotiated timeout = 20000
[junit4:junit4] 2> 340 T534 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2050de0 name:ZooKeeperConnection Watcher:127.0.0.1:48453 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 340 T514 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 341 T519 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d50d1c27f0002
[junit4:junit4] 2> 341 T534 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 341 T516 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:43082 which had sessionid 0x13d50d1c27f0002
[junit4:junit4] 2> 341 T514 oaz.ZooKeeper.close Session: 0x13d50d1c27f0002 closed
[junit4:junit4] 2> 342 T514 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 344 T514 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:48453/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@351ff5d1
[junit4:junit4] 2> 345 T514 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 345 T535 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:48453. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4] 2> 345 T535 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:48453, initiating session
[junit4:junit4] 2> 346 T516 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:43083
[junit4:junit4] 2> 346 T516 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:43083
[junit4:junit4] 2> 346 T518 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d50d1c27f0003 with negotiated timeout 20000 for client /127.0.0.1:43083
[junit4:junit4] 2> 346 T535 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:48453, sessionid = 0x13d50d1c27f0003, negotiated timeout = 20000
[junit4:junit4] 2> 347 T536 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@351ff5d1 name:ZooKeeperConnection Watcher:127.0.0.1:48453/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 347 T514 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 348 T519 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d50d1c27f0003 type:create cxid:0x2 zxid:0x1a txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 349 T514 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 351 T514 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33529_ev_p
[junit4:junit4] 2> 351 T519 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d50d1c27f0003 type:delete cxid:0x8 zxid:0x1c txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:33529_ev_p Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:33529_ev_p
[junit4:junit4] 2> 351 T514 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:33529_ev_p
[junit4:junit4] 2> 353 T514 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 356 T519 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d50d1c27f0003 type:delete cxid:0x16 zxid:0x21 txntype:-1 reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4] 2> 357 T514 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 359 T514 oasc.Overseer.start Overseer (id=89316429458767875-127.0.0.1:33529_ev_p-n_0000000000) starting
[junit4:junit4] 2> 359 T519 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d50d1c27f0003 type:create cxid:0x1b zxid:0x23 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 360 T519 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d50d1c27f0003 type:create cxid:0x1c zxid:0x24 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 361 T519 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d50d1c27f0003 type:create cxid:0x1d zxid:0x25 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 361 T519 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d50d1c27f0003 type:create cxid:0x1e zxid:0x26 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 362 T538 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 362 T514 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 364 T514 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 366 T514 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 367 T537 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 369 T539 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1362860557164/collection1
[junit4:junit4] 2> 370 T539 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 370 T539 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 370 T539 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 371 T539 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1362860557164/collection1/'
[junit4:junit4] 2> 371 T539 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-controljetty-1362860557164/collection1/lib/README' to classloader
[junit4:junit4] 2> 372 T539 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-controljetty-1362860557164/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 395 T539 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 427 T539 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 428 T539 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4] 2> 433 T539 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4] 2> 757 T539 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 763 T539 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 765 T539 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 779 T539 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 782 T539 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 786 T539 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 787 T539 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 789 T539 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 789 T539 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 790 T539 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 790 T539 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 790 T539 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1362860557164/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1362860556924/control/data/
[junit4:junit4] 2> 790 T539 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@52458f41
[junit4:junit4] 2> 791 T539 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 791 T539 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-1362860556924/control/data forceNew: false
[junit4:junit4] 2> 791 T539 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1362860556924/control/data/index/
[junit4:junit4] 2> 792 T539 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1362860556924/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 792 T539 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-1362860556924/control/data/index forceNew: false
[junit4:junit4] 2> 793 T539 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5ff7f9f lockFactory=org.apache.lucene.store.NativeFSLockFactory@2825248f),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> 793 T539 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4] 2> 795 T539 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 795 T539 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 796 T539 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 796 T539 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 796 T539 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 796 T539 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 797 T539 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 797 T539 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 797 T539 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 799 T539 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 801 T539 oass.SolrIndexSearcher.<init> Opening Searcher@647d6acc main
[junit4:junit4] 2> 801 T539 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.RecoveryZkTest-1362860556924/control/data/tlog
[junit4:junit4] 2> 802 T539 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 802 T539 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 806 T540 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@647d6acc main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 807 T539 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 807 T539 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 807 T519 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d50d1c27f0003 type:create cxid:0x55 zxid:0x2a txntype:-1 reqpath:n/a Error Path:/solr/overseer/queue Error:KeeperErrorCode = NoNode for /solr/overseer/queue
[junit4:junit4] 2> 1870 T537 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1871 T537 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "node_name":"127.0.0.1:33529_ev_p",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:33529/ev_p"}
[junit4:junit4] 2> 1871 T537 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 1871 T537 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4] 2> 1871 T519 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d50d1c27f0003 type:create cxid:0x5d zxid:0x2d txntype:-1 reqpath:n/a Error Path:/solr/overseer/queue-work Error:KeeperErrorCode = NoNode for /solr/overseer/queue-work
[junit4:junit4] 2> 1875 T536 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 2809 T539 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 2809 T539 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:33529/ev_p collection:control_collection shard:shard1
[junit4:junit4] 2> 2810 T539 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 2815 T519 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d50d1c27f0003 type:delete cxid:0x77 zxid:0x37 txntype:-1 reqpath:n/a Error Path:/solr/collections/control_collection/leaders Error:KeeperErrorCode = NoNode for /solr/collections/control_collection/leaders
[junit4:junit4] 2> 2816 T539 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 2816 T519 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d50d1c27f0003 type:create cxid:0x78 zxid:0x38 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 2817 T539 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 2817 T539 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 2818 T539 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:33529/ev_p/collection1/
[junit4:junit4] 2> 2818 T539 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 2818 T539 oasc.SyncStrategy.syncToMe http://127.0.0.1:33529/ev_p/collection1/ has no replicas
[junit4:junit4] 2> 2818 T539 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:33529/ev_p/collection1/
[junit4:junit4] 2> 2818 T539 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 2821 T519 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d50d1c27f0003 type:create cxid:0x82 zxid:0x3c txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 3377 T537 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 3382 T536 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 3424 T539 oasc.ZkController.register We are http://127.0.0.1:33529/ev_p/collection1/ and leader is http://127.0.0.1:33529/ev_p/collection1/
[junit4:junit4] 2> 3424 T539 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:33529/ev_p
[junit4:junit4] 2> 3424 T539 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 3425 T539 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 3425 T539 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 3426 T539 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 3427 T514 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 3427 T514 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 3428 T514 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 3431 T514 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 3432 T514 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:48453/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@4c01acb7
[junit4:junit4] 2> 3433 T514 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 3433 T542 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:48453. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4] 2> 3433 T542 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:48453, initiating session
[junit4:junit4] 2> 3433 T516 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:43089
[junit4:junit4] 2> 3433 T516 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:43089
[junit4:junit4] 2> 3434 T518 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d50d1c27f0004 with negotiated timeout 10000 for client /127.0.0.1:43089
[junit4:junit4] 2> 3434 T542 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:48453, sessionid = 0x13d50d1c27f0004, negotiated timeout = 10000
[junit4:junit4] 2> 3434 T543 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4c01acb7 name:ZooKeeperConnection Watcher:127.0.0.1:48453/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 3435 T514 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 3435 T514 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 3437 T514 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 3498 T514 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4] 2> 3499 T514 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:41235
[junit4:junit4] 2> 3500 T514 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 3500 T514 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 3501 T514 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1362860560357
[junit4:junit4] 2> 3501 T514 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1362860560357/solr.xml
[junit4:junit4] 2> 3501 T514 oasc.CoreContainer.<init> New CoreContainer 1593181227
[junit4:junit4] 2> 3501 T514 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1362860560357/'
[junit4:junit4] 2> 3502 T514 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1362860560357/'
[junit4:junit4] 2> 3517 T514 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 3517 T514 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 3517 T514 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 3518 T514 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 3518 T514 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 3518 T514 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 3519 T514 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 3519 T514 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 3519 T514 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 3519 T514 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 3525 T514 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4] 2> 3534 T514 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:48453/solr
[junit4:junit4] 2> 3535 T514 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 3535 T514 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:48453 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@5afb5a09
[junit4:junit4] 2> 3536 T514 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 3536 T553 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:48453. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4] 2> 3537 T553 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:48453, initiating session
[junit4:junit4] 2> 3537 T516 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:43090
[junit4:junit4] 2> 3537 T516 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:43090
[junit4:junit4] 2> 3538 T518 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d50d1c27f0005 with negotiated timeout 20000 for client /127.0.0.1:43090
[junit4:junit4] 2> 3538 T553 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:48453, sessionid = 0x13d50d1c27f0005, negotiated timeout = 20000
[junit4:junit4] 2> 3538 T554 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5afb5a09 name:ZooKeeperConnection Watcher:127.0.0.1:48453 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 3538 T514 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 3539 T519 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d50d1c27f0005
[junit4:junit4] 2> 3539 T554 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 3539 T516 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:43090 which had sessionid 0x13d50d1c27f0005
[junit4:junit4] 2> 3539 T514 oaz.ZooKeeper.close Session: 0x13d50d1c27f0005 closed
[junit4:junit4] 2> 3539 T514 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 3542 T514 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:48453/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@7cd07b83
[junit4:junit4] 2> 3543 T514 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 3543 T555 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:48453. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4] 2> 3543 T555 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:48453, initiating session
[junit4:junit4] 2> 3543 T516 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:43091
[junit4:junit4] 2> 3544 T516 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:43091
[junit4:junit4] 2> 3544 T518 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d50d1c27f0006 with negotiated timeout 20000 for client /127.0.0.1:43091
[junit4:junit4] 2> 3544 T555 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:48453, sessionid = 0x13d50d1c27f0006, negotiated timeout = 20000
[junit4:junit4] 2> 3544 T556 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7cd07b83 name:ZooKeeperConnection Watcher:127.0.0.1:48453/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 3545 T514 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 3545 T519 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d50d1c27f0006 type:create cxid:0x1 zxid:0x4a txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 3546 T519 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d50d1c27f0006 type:create cxid:0x2 zxid:0x4b txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 3547 T514 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 4549 T514 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41235_ev_p
[junit4:junit4] 2> 4550 T519 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d50d1c27f0006 type:delete cxid:0xb zxid:0x4c txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:41235_ev_p Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:41235_ev_p
[junit4:junit4] 2> 4551 T514 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:41235_ev_p
[junit4:junit4] 2> 4555 T556 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 4556 T536 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 4555 T543 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 4556 T536 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 4564 T557 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1362860560357/collection1
[junit4:junit4] 2> 4564 T557 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 4565 T557 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 4565 T557 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 4567 T557 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1362860560357/collection1/'
[junit4:junit4] 2> 4568 T557 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty1-1362860560357/collection1/lib/README' to classloader
[junit4:junit4] 2> 4568 T557 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty1-1362860560357/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 4602 T557 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 4629 T557 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 4630 T557 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4] 2> 4634 T557 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4] 2> 4886 T537 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 4887 T537 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "node_name":"127.0.0.1:33529_ev_p",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:33529/ev_p"}
[junit4:junit4] 2> 4890 T556 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 4890 T536 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 4890 T543 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 4948 T557 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 4954 T557 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 4957 T557 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 4968 T557 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 4971 T557 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 4974 T557 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 4974 T557 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 4975 T557 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 4975 T557 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 4976 T557 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 4976 T557 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 4976 T557 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1362860560357/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1362860556924/jetty1/
[junit4:junit4] 2> 4976 T557 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@52458f41
[junit4:junit4] 2> 4977 T557 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 4977 T557 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-1362860556924/jetty1 forceNew: false
[junit4:junit4] 2> 4977 T557 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1362860556924/jetty1/index/
[junit4:junit4] 2> 4978 T557 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1362860556924/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 4978 T557 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-1362860556924/jetty1/index forceNew: false
[junit4:junit4] 2> 4979 T557 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4605346a lockFactory=org.apache.lucene.store.NativeFSLockFactory@12983349),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> 4979 T557 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4] 2> 4981 T557 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 4981 T557 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 4982 T557 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 4982 T557 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 4982 T557 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 4982 T557 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 4983 T557 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 4983 T557 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 4983 T557 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 4985 T557 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 4987 T557 oass.SolrIndexSearcher.<init> Opening Searcher@5431ccfe main
[junit4:junit4] 2> 4987 T557 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.RecoveryZkTest-1362860556924/jetty1/tlog
[junit4:junit4] 2> 4988 T557 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 4988 T557 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 4991 T558 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5431ccfe main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 4994 T557 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 4994 T557 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 6394 T537 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 6395 T537 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:41235_ev_p",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:41235/ev_p"}
[junit4:junit4] 2> 6395 T537 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 1
[junit4:junit4] 2> 6395 T537 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4] 2> 6398 T556 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 6398 T543 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 6398 T536 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 6996 T557 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 6996 T557 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:41235/ev_p collection:collection1 shard:shard1
[junit4:junit4] 2> 6997 T557 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 7001 T519 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d50d1c27f0006 type:delete cxid:0x48 zxid:0x5c txntype:-1 reqpath:n/a Error Path:/solr/collections/collection1/leaders Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders
[junit4:junit4] 2> 7002 T557 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 7002 T519 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d50d1c27f0006 type:create cxid:0x49 zxid:0x5d txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 7003 T557 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 7003 T557 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 7003 T557 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:41235/ev_p/collection1/
[junit4:junit4] 2> 7003 T557 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 7004 T557 oasc.SyncStrategy.syncToMe http://127.0.0.1:41235/ev_p/collection1/ has no replicas
[junit4:junit4] 2> 7004 T557 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:41235/ev_p/collection1/
[junit4:junit4] 2> 7004 T557 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 7007 T519 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d50d1c27f0006 type:create cxid:0x53 zxid:0x61 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 7903 T537 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 7915 T543 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 7915 T556 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 7915 T536 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 7960 T557 oasc.ZkController.register We are http://127.0.0.1:41235/ev_p/collection1/ and leader is http://127.0.0.1:41235/ev_p/collection1/
[junit4:junit4] 2> 7961 T557 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:41235/ev_p
[junit4:junit4] 2> 7961 T557 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 7961 T557 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 7961 T557 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 7964 T557 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 7965 T514 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 7966 T514 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 7967 T514 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 8042 T514 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4] 2> 8044 T514 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:47003
[junit4:junit4] 2> 8045 T514 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 8046 T514 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 8046 T514 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1362860564895
[junit4:junit4] 2> 8046 T514 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1362860564895/solr.xml
[junit4:junit4] 2> 8046 T514 oasc.CoreContainer.<init> New CoreContainer 1554942991
[junit4:junit4] 2> 8047 T514 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1362860564895/'
[junit4:junit4] 2> 8047 T514 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1362860564895/'
[junit4:junit4] 2> 8063 T514 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 8063 T514 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 8063 T514 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 8064 T514 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 8064 T514 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 8064 T514 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 8065 T514 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 8065 T514 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 8065 T514 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 8065 T514 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 8071 T514 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4] 2> 8081 T514 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:48453/solr
[junit4:junit4] 2> 8082 T514 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 8082 T514 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:48453 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@5cc83020
[junit4:junit4] 2> 8083 T514 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 8083 T569 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:48453. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4] 2> 8084 T569 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:48453, initiating session
[junit4:junit4] 2> 8084 T516 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:43097
[junit4:junit4] 2> 8084 T516 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:43097
[junit4:junit4] 2> 8085 T518 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d50d1c27f0007 with negotiated timeout 20000 for client /127.0.0.1:43097
[junit4:junit4] 2> 8085 T569 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:48453, sessionid = 0x13d50d1c27f0007, negotiated timeout = 20000
[junit4:junit4] 2> 8085 T570 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5cc83020 name:ZooKeeperConnection Watcher:127.0.0.1:48453 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 8085 T514 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 8086 T519 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d50d1c27f0007
[junit4:junit4] 2> 8086 T570 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 8086 T516 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:43097 which had sessionid 0x13d50d1c27f0007
[junit4:junit4] 2> 8086 T514 oaz.ZooKeeper.close Session: 0x13d50d1c27f0007 closed
[junit4:junit4] 2> 8086 T514 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 8089 T514 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:48453/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@142ff9fa
[junit4:junit4] 2> 8090 T514 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 8090 T571 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:48453. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4] 2> 8090 T571 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:48453, initiating session
[junit4:junit4] 2> 8091 T516 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:43098
[junit4:junit4] 2> 8091 T516 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:43098
[junit4:junit4] 2> 8092 T518 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d50d1c27f0008 with negotiated timeout 20000 for client /127.0.0.1:43098
[junit4:junit4] 2> 8092 T571 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:48453, sessionid = 0x13d50d1c27f0008, negotiated timeout = 20000
[junit4:junit4] 2> 8092 T572 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@142ff9fa name:ZooKeeperConnection Watcher:127.0.0.1:48453/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 8092 T514 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 8093 T519 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d50d1c27f0008 type:create cxid:0x1 zxid:0x6e txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 8093 T519 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d50d1c27f0008 type:create cxid:0x2 zxid:0x6f txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 8094 T514 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 9097 T514 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:47003_ev_p
[junit4:junit4] 2> 9098 T519 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d50d1c27f0008 type:delete cxid:0xb zxid:0x70 txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:47003_ev_p Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:47003_ev_p
[junit4:junit4] 2> 9100 T514 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:47003_ev_p
[junit4:junit4] 2> 9103 T556 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 9104 T543 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 9104 T536 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 9105 T572 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 9105 T543 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 9105 T536 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 9106 T556 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 9114 T573 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1362860564895/collection1
[junit4:junit4] 2> 9114 T573 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 9115 T573 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 9115 T573 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 9117 T573 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1362860564895/collection1/'
[junit4:junit4] 2> 9118 T573 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty2-1362860564895/collection1/lib/README' to classloader
[junit4:junit4] 2> 9119 T573 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty2-1362860564895/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 9158 T573 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 9190 T573 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 9191 T573 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4] 2> 9196 T573 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4] 2> 9422 T537 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 9422 T537 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:41235_ev_p",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:41235/ev_p"}
[junit4:junit4] 2> 9425 T556 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 9425 T543 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 9426 T572 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 9426 T536 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 9511 T573 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 9517 T573 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 9519 T573 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 9537 T573 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 9541 T573 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 9544 T573 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 9545 T573 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 9546 T573 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 9546 T573 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 9547 T573 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 9547 T573 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 9547 T573 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1362860564895/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1362860556924/jetty2/
[junit4:junit4] 2> 9548 T573 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@52458f41
[junit4:junit4] 2> 9548 T573 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 9549 T573 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-1362860556924/jetty2 forceNew: false
[junit4:junit4] 2> 9549 T573 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1362860556924/jetty2/index/
[junit4:junit4] 2> 9549 T573 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1362860556924/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 9550 T573 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-1362860556924/jetty2/index forceNew: false
[junit4:junit4] 2> 9551 T573 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@28e04e24 lockFactory=org.apache.lucene.store.NativeFSLockFactory@64562b7d),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> 9552 T573 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4] 2> 9554 T573 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 9555 T573 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 9555 T573 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 9556 T573 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 9557 T573 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 9557 T573 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 9557 T573 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 9558 T573 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 9558 T573 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 9561 T573 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 9564 T573 oass.SolrIndexSearcher.<init> Opening Searcher@7d9c25e8 main
[junit4:junit4] 2> 9565 T573 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.RecoveryZkTest-1362860556924/jetty2/tlog
[junit4:junit4] 2> 9565 T573 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 9566 T573 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 9571 T574 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7d9c25e8 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 9573 T573 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 9573 T573 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 10930 T537 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 10931 T537 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:47003_ev_p",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:47003/ev_p"}
[junit4:junit4] 2> 10931 T537 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 10932 T537 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4] 2> 10939 T556 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 10939 T536 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 10939 T572 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 10939 T543 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 11574 T573 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 11574 T573 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:47003/ev_p collection:collection1 shard:shard1
[junit4:junit4] 2> 11577 T573 oasc.ZkController.register We are http://127.0.0.1:47003/ev_p/collection1/ and leader is http://127.0.0.1:41235/ev_p/collection1/
[junit4:junit4] 2> 11577 T573 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:47003/ev_p
[junit4:junit4] 2> 11578 T573 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 11578 T573 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C23 name=collection1 org.apache.solr.core.SolrCore@316ab910 url=http://127.0.0.1:47003/ev_p/collection1 node=127.0.0.1:47003_ev_p C23_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:47003_ev_p, base_url=http://127.0.0.1:47003/ev_p}
[junit4:junit4] 2> 11578 T575 C23 P47003 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 11579 T575 C23 P47003 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 11579 T573 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 11579 T575 C23 P47003 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 11579 T575 C23 P47003 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 11580 T514 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 11580 T575 C23 P47003 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 11580 T514 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 11581 T514 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 11585 T514 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C24 name=collection1 org.apache.solr.core.SolrCore@401c68a0 url=http://127.0.0.1:33529/ev_p/collection1 node=127.0.0.1:33529_ev_p C24_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:33529_ev_p, base_url=http://127.0.0.1:33529/ev_p, leader=true}
[junit4:junit4] 2> 11596 T532 C24 P33529 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5ff7f9f lockFactory=org.apache.lucene.store.NativeFSLockFactory@2825248f),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> 11597 T532 C24 P33529 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4] 2> 11600 T532 C24 P33529 /update {wt=javabin&version=2} {add=[10001 (1429062883491184640)]} 0 5
[junit4:junit4] 2> 11600 T529 C24 P33529 /update {wt=javabin&version=2} {add=[1 (1429062883491184641)]} 0 5
[junit4:junit4] 2> ASYNC NEW_CORE C25 name=collection1 org.apache.solr.core.SolrCore@19a7b6e3 url=http://127.0.0.1:41235/ev_p/collection1 node=127.0.0.1:41235_ev_p C25_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:41235_ev_p, base_url=http://127.0.0.1:41235/ev_p, leader=true}
[junit4:junit4] 2> 11617 T551 C25 P41235 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4605346a lockFactory=org.apache.lucene.store.NativeFSLockFactory@12983349),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> 11617 T551 C25 P41235 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4] 2> 11620 T551 C25 P41235 /update {wt=javabin&version=2} {add=[10001 (1429062883513204736)]} 0 5
[junit4:junit4] 2> 11620 T552 C25 P41235 /update {wt=javabin&version=2} {add=[1 (1429062883513204737)]} 0 5
[junit4:junit4] 2> 11684 T530 C24 P33529 /update {wt=javabin&version=2} {add=[10002 (1429062883582410752)]} 0 2
[junit4:junit4] 2> 11685 T531 C24 P33529 /update {wt=javabin&version=2} {add=[2 (1429062883583459328)]} 0 3
[junit4:junit4] 2> 11689 T549 C25 P41235 /update {wt=javabin&version=2} {add=[10002 (1429062883587653632)]} 0 2
[junit4:junit4] 2> 11689 T551 C25 P41235 /update {wt=javabin&version=2} {add=[2 (1429062883588702208)]} 0 1
[junit4:junit4] 2> 11742 T532 C24 P33529 /update {wt=javabin&version=2} {add=[10003 (1429062883644276736)]} 0 2
[junit4:junit4] 2> 11742 T529 C24 P33529 /update {wt=javabin&version=2} {add=[3 (1429062883644276737)]} 0 2
[junit4:junit4] 2> 11745 T552 C25 P41235 /update {wt=javabin&version=2} {add=[10003 (1429062883647422464)]} 0 1
[junit4:junit4] 2> 11746 T549 C25 P41235 /update {wt=javabin&version=2} {add=[3 (1429062883647422465)]} 0 2
[junit4:junit4] 2> 11755 T530 C24 P33529 /update {wt=javabin&version=2} {delete=[10001 (-1429062883658956800)]} 0 1
[junit4:junit4] 2> 11755 T531 C24 P33529 /update {wt=javabin&version=2} {delete=[1 (-1429062883658956801)]} 0 0
[junit4:junit4] 2> 11757 T551 C25 P41235 /update {wt=javabin&version=2} {delete=[10001 (-1429062883661053952)]} 0 0
[junit4:junit4] 2> 11758 T552 C25 P41235 /update {wt=javabin&version=2} {delete=[1 (-1429062883662102528)]} 0 0
[junit4:junit4] 2> 11760 T529 C24 P33529 /update {wt=javabin&version=2} {add=[10004 (1429062883663151104)]} 0 1
[junit4:junit4] 2> 11761 T532 C24 P33529 /update {wt=javabin&version=2} {add=[4 (1429062883664199680)]} 0 1
[junit4:junit4] 2> 11764 T549 C25 P41235 /update {wt=javabin&version=2} {add=[10004 (1429062883667345408)]} 0 1
[junit4:junit4] 2> 11764 T551 C25 P41235 /update {wt=javabin&version=2} {add=[4 (1429062883667345409)]} 0 1
[junit4:junit4] 2> 11792 T530 C24 P33529 /update {wt=javabin&version=2} {add=[10005 (1429062883696705536)]} 0 2
[junit4:junit4] 2> 11792 T531 C24 P33529 /update {wt=javabin&version=2} {add=[5 (1429062883696705537)]} 0 1
[junit4:junit4] 2> 11795 T552 C25 P41235 /update {wt=javabin&version=2} {add=[10005 (1429062883699851264)]} 0 1
[junit4:junit4] 2> 11795 T549 C25 P41235 /update {wt=javabin&version=2} {add=[5 (1429062883700899840)]} 0 1
[junit4:junit4] 2> 11881 T532 C24 P33529 /update {wt=javabin&version=2} {delete=[5 (-1429062883791077377)]} 0 0
[junit4:junit4] 2> 11881 T529 C24 P33529 /update {wt=javabin&version=2} {delete=[10005 (-1429062883791077376)]} 0 0
[junit4:junit4] 2> 11884 T551 C25 P41235 /update {wt=javabin&version=2} {delete=[10005 (-1429062883794223104)]} 0 1
[junit4:junit4] 2> 11884 T552 C25 P41235 /update {wt=javabin&version=2} {delete=[5 (-1429062883794223105)]} 0 0
[junit4:junit4] 2> 11887 T531 C24 P33529 /update {wt=javabin&version=2} {add=[6 (1429062883796320257)]} 0 1
[junit4:junit4] 2> 11888 T530 C24 P33529 /update {wt=javabin&version=2} {add=[10006 (1429062883796320256)]} 0 2
[junit4:junit4] 2> 11890 T549 C25 P41235 /update {wt=javabin&version=2} {add=[6 (1429062883799465984)]} 0 1
[junit4:junit4] 2> 11891 T551 C25 P41235 /update {wt=javabin&version=2} {add=[10006 (1429062883801563136)]} 0 1
[junit4:junit4] 2> 11919 T532 C24 P33529 /update {wt=javabin&version=2} {add=[7 (1429062883829874688)]} 0 1
[junit4:junit4] 2> 11920 T529 C24 P33529 /update {wt=javabin&version=2} {add=[10007 (1429062883830923264)]} 0 1
[junit4:junit4] 2> 11923 T549 C25 P41235 /update {wt=javabin&version=2} {add=[10007 (1429062883834068992)]} 0 1
[junit4:junit4] 2> 11923 T552 C25 P41235 /update {wt=javabin&version=2} {add=[7 (1429062883834068993)]} 0 1
[junit4:junit4] 2> 11980 T531 C24 P33529 /update {wt=javabin&version=2} {add=[10008 (1429062883893837824)]} 0 1
[junit4:junit4] 2> 11980 T530 C24 P33529 /update {wt=javabin&version=2} {add=[8 (1429062883893837825)]} 0 1
[junit4:junit4] 2> 11983 T551 C25 P41235 /update {wt=javabin&version=2} {add=[10008 (1429062883896983552)]} 0 1
[junit4:junit4] 2> 11984 T549 C25 P41235 /update {wt=javabin&version=2} {add=[8 (1429062883898032128)]} 0 1
[junit4:junit4] 2> 12052 T532 C24 P33529 /update {wt=javabin&version=2} {delete=[10007 (-1429062883970383872)]} 0 0
[junit4:junit4] 2> 12052 T531 C24 P33529 /update {wt=javabin&version=2} {delete=[7 (-1429062883970383873)]} 0 0
[junit4:junit4] 2> 12055 T552 C25 P41235 /update {wt=javabin&version=2} {delete=[10007 (-1429062883973529600)]} 0 1
[junit4:junit4] 2> 12055 T551 C25 P41235 /update {wt=javabin&version=2} {delete=[7 (-1429062883973529601)]} 0 0
[junit4:junit4] 2> 12058 T529 C24 P33529 /update {wt=javabin&version=2} {add=[10009 (1429062883975626752)]} 0 1
[junit4:junit4] 2> 12058 T530 C24 P33529 /update {wt=javabin&version=2} {add=[9 (1429062883975626753)]} 0 1
[junit4:junit4] 2> 12061 T552 C25 P41235 /update {wt=javabin&version=2} {add=[9 (1429062883979821056)]} 0 1
[junit4:junit4] 2> 12061 T549 C25 P41235 /update {wt=javabin&version=2} {add=[10009 (1429062883978772480)]} 0 1
[junit4:junit4] 2> 12115 T531 C24 P33529 /update {wt=javabin&version=2} {delete=[10008 (-1429062884036444160)]} 0 0
[junit4:junit4] 2> 12115 T532 C24 P33529 /update {wt=javabin&version=2} {delete=[8 (-1429062884036444161)]} 0 0
[junit4:junit4] 2> 12118 T551 C25 P41235 /update {wt=javabin&version=2} {delete=[8 (-1429062884039589888)]} 0 0
[junit4:junit4] 2> 12118 T549 C25 P41235 /update {wt=javabin&version=2} {delete=[10008 (-1429062884039589889)]} 0 0
[junit4:junit4] 2> 12121 T529 C24 P33529 /update {wt=javabin&version=2} {add=[10 (1429062884041687040)]} 0 1
[junit4:junit4] 2> 12121 T530 C24 P33529 /update {wt=javabin&version=2} {add=[10010 (1429062884041687041)]} 0 1
[junit4:junit4] 2> 12124 T552 C25 P41235 /update {wt=javabin&version=2} {add=[10 (1429062884045881344)]} 0 1
[junit4:junit4] 2> 12125 T549 C25 P41235 /update {wt=javabin&version=2} {add=[10010 (1429062884045881345)]} 0 1
[junit4:junit4] 2> 12162 T531 C24 P33529 /update {wt=javabin&version=2} {delete=[10 (-1429062884085727232)]} 0 0
[junit4:junit4] 2> 12162 T532 C24 P33529 /update {wt=javabin&version=2} {delete=[10010 (-1429062884085727233)]} 0 0
[junit4:junit4] 2> 12164 T551 C25 P41235 /update {wt=javabin&version=2} {delete=[10 (-1429062884087824384)]} 0 0
[junit4:junit4] 2> 12165 T552 C25 P41235 /update {wt=javabin&version=2} {delete=[10010 (-1429062884087824385)]} 0 1
[junit4:junit4] 2> 12168 T529 C24 P33529 /update {wt=javabin&version=2} {add=[11 (1429062884089921536)]} 0 2
[junit4:junit4] 2> 12169 T530 C24 P33529 /update {wt=javabin&version=2} {add=[10011 (1429062884090970112)]} 0 2
[junit4:junit4] 2> 12173 T549 C25 P41235 /update {wt=javabin&version=2} {add=[11 (1429062884095164416)]} 0 2
[junit4:junit4] 2> 12174 T551 C25 P41235 /update {wt=javabin&version=2} {add=[10011 (1429062884097261568)]} 0 2
[junit4:junit4] 2> 12273 T531 C24 P33529 /update {wt=javabin&version=2} {delete=[11 (-1429062884201070592)]} 0 1
[junit4:junit4] 2> 12274 T532 C24 P33529 /update {wt=javabin&version=2} {delete=[10011 (-1429062884203167744)]} 0 0
[junit4:junit4] 2> 12276 T552 C25 P41235 /update {wt=javabin&version=2} {delete=[11 (-1429062884205264896)]} 0 0
[junit4:junit4] 2> 12278 T549 C25 P41235 /update {wt=javabin&version=2} {delete=[10011 (-1429062884206313472)]} 0 1
[junit4:junit4] 2> 12281 T529 C24 P33529 /update {wt=javabin&version=2} {add=[12 (1429062884209459200)]} 0 2
[junit4:junit4] 2> 12284 T530 C24 P33529 /update {wt=javabin&version=2} {add=[10012 (1429062884210507776)]} 0 3
[junit4:junit4] 2> 12286 T551 C25 P41235 /update {wt=javabin&version=2} {add=[12 (1429062884214702080)]} 0 2
[junit4:junit4] 2> 12289 T552 C25 P41235 /update {wt=javabin&version=2} {add=[10012 (1429062884217847808)]} 0 2
[junit4:junit4] 2> 12344 T531 C24 P33529 /update {wt=javabin&version=2} {delete=[12 (-1429062884276568064)]} 0 0
[junit4:junit4] 2> 12347 T532 C24 P33529 /update {wt=javabin&version=2} {delete=[10012 (-1429062884279713792)]} 0 1
[junit4:junit4] 2> 12347 T549 C25 P41235 /update {wt=javabin&version=2} {delete=[12 (-1429062884279713792)]} 0 0
[junit4:junit4] 2> 12350 T552 C25 P41235 /update {wt=javabin&version=2} {delete=[10012 (-1429062884282859520)]} 0 0
[junit4:junit4] 2> 12353 T529 C24 P33529 /update {wt=javabin&version=2} {add=[13 (1429062884283908096)]} 0 2
[junit4:junit4] 2> 12356 T530 C24 P33529 /update {wt=javabin&version=2} {add=[10013 (1429062884287053824)]} 0 3
[junit4:junit4] 2> 12358 T549 C25 P41235 /update {wt=javabin&version=2} {add=[13 (1429062884289150976)]} 0 2
[junit4:junit4] 2> 12361 T551 C25 P41235 /update {wt=javabin&version=2} {add=[10013 (1429062884292296704)]} 0 2
[junit4:junit4] 2> 12425 T531 C24 P33529 /update {wt=javabin&version=2} {add=[14 (1429062884360454144)]} 0 2
[junit4:junit4] 2> 12428 T532 C24 P33529 /update {wt=javabin&version=2} {add=[10014 (1429062884362551296)]} 0 2
[junit4:junit4] 2> 12430 T552 C25 P41235 /update {wt=javabin&version=2} {add=[14 (1429062884365697024)]} 0 2
[junit4:junit4] 2> 12433 T549 C25 P41235 /update {wt=javabin&version=2} {add=[10014 (1429062884368842752)]} 0 2
[junit4:junit4] 2> 12445 T537 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 12447 T537 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:47003_ev_p",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:47003/ev_p"}
[junit4:junit4] 2> 12452 T556 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 12452 T536 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 12452 T572 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 12452 T543 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 12484 T529 C24 P33529 /update {wt=javabin&version=2} {add=[15 (1429062884422320128)]} 0 1
[junit4:junit4] 2> 12487 T531 C24 P33529 /update {wt=javabin&version=2} {add=[10015 (1429062884425465856)]} 0 2
[junit4:junit4] 2> C23_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:47003_ev_p, base_url=http://127.0.0.1:47003/ev_p}
[junit4:junit4] 2> 12502 T565 C23 P47003 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@28e04e24 lockFactory=org.apache.lucene.store.NativeFSLockFactory@64562b7d),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> 12503 T565 C23 P47003 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4] 2> 12506 T565 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[15 (1429062884427563008)]} 0 7
[junit4:junit4] 2> 12507 T551 C25 P41235 /update {wt=javabin&version=2} {add=[15 (1429062884427563008)]} 0 19
[junit4:junit4] 2> 12507 T566 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10015 (1429062884430708736)]} 0 5
[junit4:junit4] 2> 12508 T552 C25 P41235 /update {wt=javabin&version=2} {add=[10015 (1429062884430708736)]} 0 18
[junit4:junit4] 2> 12514 T530 C24 P33529 /update {wt=javabin&version=2} {add=[16 (1429062884452728832)]} 0 2
[junit4:junit4] 2> 12515 T532 C24 P33529 /update {wt=javabin&version=2} {add=[10016 (1429062884453777408)]} 0 2
[junit4:junit4] 2> 12524 T567 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[16 (1429062884457971712)]} 0 2
[junit4:junit4] 2> 12525 T549 C25 P41235 /update {wt=javabin&version=2} {add=[16 (1429062884457971712)]} 0 8
[junit4:junit4] 2> 12526 T568 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10016 (1429062884459020288)]} 0 3
[junit4:junit4] 2> 12527 T551 C25 P41235 /update {wt=javabin&version=2} {add=[10016 (1429062884459020288)]} 0 9
[junit4:junit4] 2> 12561 T529 C24 P33529 /update {wt=javabin&version=2} {add=[17 (1429062884502011904)]} 0 3
[junit4:junit4] 2> 12562 T531 C24 P33529 /update {wt=javabin&version=2} {add=[10017 (1429062884503060480)]} 0 2
[junit4:junit4] 2> 12570 T565 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[17 (1429062884507254784)]} 0 2
[junit4:junit4] 2> 12571 T552 C25 P41235 /update {wt=javabin&version=2} {add=[17 (1429062884507254784)]} 0 7
[junit4:junit4] 2> 12572 T566 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10017 (1429062884509351936)]} 0 2
[junit4:junit4] 2> 12573 T549 C25 P41235 /update {wt=javabin&version=2} {add=[10017 (1429062884509351936)]} 0 8
[junit4:junit4] 2> 12586 T550 oass.SolrDispatchFilter.handleAdminRequest /admin/cores {coreNodeName=3&state=recovering&nodeName=127.0.0.1:47003_ev_p&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1000
[junit4:junit4] 2> 12622 T530 C24 P33529 /update {wt=javabin&version=2} {delete=[16 (-1429062884567023616)]} 0 1
[junit4:junit4] 2> 12623 T532 C24 P33529 /update {wt=javabin&version=2} {delete=[10016 (-1429062884569120768)]} 0 0
[junit4:junit4] 2> 12631 T567 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10016 (-1429062884572266496)]} 0 1
[junit4:junit4] 2> 12631 T568 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[16 (-1429062884571217920)]} 0 1
[junit4:junit4] 2> 12632 T549 C25 P41235 /update {wt=javabin&version=2} {delete=[10016 (-1429062884572266496)]} 0 6
[junit4:junit4] 2> 12632 T552 C25 P41235 /update {wt=javabin&version=2} {delete=[16 (-1429062884571217920)]} 0 7
[junit4:junit4] 2> 12637 T529 C24 P33529 /update {wt=javabin&version=2} {add=[10018 (1429062884582752256)]} 0 2
[junit4:junit4] 2> 12638 T531 C24 P33529 /update {wt=javabin&version=2} {add=[18 (1429062884582752257)]} 0 3
[junit4:junit4] 2> 12648 T565 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10018 (1429062884587995136)]} 0 3
[junit4:junit4] 2> 12648 T550 C25 P41235 /update {wt=javabin&version=2} {add=[10018 (1429062884587995136)]} 0 8
[junit4:junit4] 2> 12649 T566 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[18 (1429062884587995137)]} 0 2
[junit4:junit4] 2> 12650 T551 C25 P41235 /update {wt=javabin&version=2} {add=[18 (1429062884587995137)]} 0 9
[junit4:junit4] 2> 12715 T530 C24 P33529 /update {wt=javabin&version=2} {delete=[10018 (-1429062884665589760)]} 0 0
[junit4:junit4] 2> 12717 T532 C24 P33529 /update {wt=javabin&version=2} {delete=[18 (-1429062884667686912)]} 0 0
[junit4:junit4] 2> 12722 T567 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10018 (-1429062884669784064)]} 0 0
[junit4:junit4] 2> 12723 T549 C25 P41235 /update {wt=javabin&version=2} {delete=[10018 (-1429062884669784064)]} 0 4
[junit4:junit4] 2> 12724 T567 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[18 (-1429062884671881216)]} 0 1
[junit4:junit4] 2> 12725 T552 C25 P41235 /update {wt=javabin&version=2} {delete=[18 (-1429062884671881216)]} 0 5
[junit4:junit4] 2> 12728 T529 C24 P33529 /update {wt=javabin&version=2} {add=[10019 (1429062884678172672)]} 0 2
[junit4:junit4] 2> 12730 T531 C24 P33529 /update {wt=javabin&version=2} {add=[19 (1429062884679221248)]} 0 2
[junit4:junit4] 2> 12738 T565 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10019 (1429062884683415552)]} 0 2
[junit4:junit4] 2> 12739 T550 C25 P41235 /update {wt=javabin&version=2} {add=[10019 (1429062884683415552)]} 0 8
[junit4:junit4] 2> 12740 T566 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[19 (1429062884685512704)]} 0 3
[junit4:junit4] 2> 12741 T551 C25 P41235 /update {wt=javabin&version=2} {add=[19 (1429062884685512704)]} 0 8
[junit4:junit4] 2> 12782 T530 C24 P33529 /update {wt=javabin&version=2} {add=[10020 (1429062884733747200)]} 0 2
[junit4:junit4] 2> 12783 T532 C24 P33529 /update {wt=javabin&version=2} {add=[20 (1429062884735844352)]} 0 2
[junit4:junit4] 2> 12791 T568 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10020 (1429062884738990080)]} 0 2
[junit4:junit4] 2> 12792 T549 C25 P41235 /update {wt=javabin&version=2} {add=[10020 (1429062884738990080)]} 0 7
[junit4:junit4] 2> 12793 T567 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[20 (1429062884741087232)]} 0 2
[junit4:junit4] 2> 12794 T550 C25 P41235 /update {wt=javabin&version=2} {add=[20 (1429062884741087232)]} 0 7
[junit4:junit4] 2> 12843 T529 C24 P33529 /update {wt=javabin&version=2} {add=[10021 (1429062884797710336)]} 0 2
[junit4:junit4] 2> 12845 T531 C24 P33529 /update {wt=javabin&version=2} {add=[21 (1429062884799807488)]} 0 3
[junit4:junit4] 2> 12852 T565 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10021 (1429062884802953216)]} 0 1
[junit4:junit4] 2> 12853 T552 C25 P41235 /update {wt=javabin&version=2} {add=[10021 (1429062884802953216)]} 0 7
[junit4:junit4] 2> 12853 T568 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[21 (1429062884805050368)]} 0 1
[junit4:junit4] 2> 12853 T551 C25 P41235 /update {wt=javabin&version=2} {add=[21 (1429062884805050368)]} 0 5
[junit4:junit4] 2> 12865 T530 C24 P33529 /update {wt=javabin&version=2} {add=[10022 (1429062884821827584)]} 0 1
[junit4:junit4] 2> 12866 T532 C24 P33529 /update {wt=javabin&version=2} {add=[22 (1429062884822876160)]} 0 2
[junit4:junit4] 2> 12872 T566 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10022 (1429062884826021888)]} 0 2
[junit4:junit4] 2> 12872 T567 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[22 (1429062884826021889)]} 0 1
[junit4:junit4] 2> 12873 T549 C25 P41235 /update {wt=javabin&version=2} {add=[10022 (1429062884826021888)]} 0 6
[junit4:junit4] 2> 12873 T550 C25 P41235 /update {wt=javabin&version=2} {add=[22 (1429062884826021889)]} 0 5
[junit4:junit4] 2> 12938 T529 C24 P33529 /update {wt=javabin&version=2} {add=[10023 (1429062884898373632)]} 0 1
[junit4:junit4] 2> 12938 T530 C24 P33529 /update {wt=javabin&version=2} {add=[23 (1429062884898373633)]} 0 1
[junit4:junit4] 2> 12947 T565 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10023 (1429062884902567936)]} 0 3
[junit4:junit4] 2> 12948 T552 C25 P41235 /update {wt=javabin&version=2} {add=[10023 (1429062884902567936)]} 0 7
[junit4:junit4] 2> 12950 T568 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[23 (1429062884902567937)]} 0 2
[junit4:junit4] 2> 12951 T551 C25 P41235 /update {wt=javabin&version=2} {add=[23 (1429062884902567937)]} 0 10
[junit4:junit4] 2> 13003 T531 C24 P33529 /update {wt=javabin&version=2} {add=[10024 (1429062884966531072)]} 0 1
[junit4:junit4] 2> 13005 T532 C24 P33529 /update {wt=javabin&version=2} {add=[24 (1429062884968628224)]} 0 1
[junit4:junit4] 2> 13011 T566 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10024 (1429062884970725376)]} 0 2
[junit4:junit4] 2> 13012 T549 C25 P41235 /update {wt=javabin&version=2} {add=[10024 (1429062884970725376)]} 0 6
[junit4:junit4] 2> 13012 T567 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[24 (1429062884972822528)]} 0 1
[junit4:junit4] 2> 13013 T550 C25 P41235 /update {wt=javabin&version=2} {add=[24 (1429062884972822528)]} 0 5
[junit4:junit4] 2> 13038 T529 C24 P33529 /update {wt=javabin&version=2} {add=[10025 (1429062885003231232)]} 0 2
[junit4:junit4] 2> 13038 T530 C24 P33529 /update {wt=javabin&version=2} {add=[25 (1429062885003231233)]} 0 1
[junit4:junit4] 2> 13044 T565 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10025 (1429062885006376960)]} 0 1
[junit4:junit4] 2> 13045 T568 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[25 (1429062885007425536)]} 0 2
[junit4:junit4] 2> 13045 T552 C25 P41235 /update {wt=javabin&version=2} {add=[10025 (1429062885006376960)]} 0 5
[junit4:junit4] 2> 13045 T551 C25 P41235 /update {wt=javabin&version=2} {add=[25 (1429062885007425536)]} 0 5
[junit4:junit4] 2> 13067 T531 C24 P33529 /update {wt=javabin&version=2} {add=[10026 (1429062885033639936)]} 0 1
[junit4:junit4] 2> 13068 T532 C24 P33529 /update {wt=javabin&version=2} {add=[26 (1429062885033639937)]} 0 2
[junit4:junit4] 2> 13075 T566 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[26 (1429062885037834241)]} 0 1
[junit4:junit4] 2> 13076 T550 C25 P41235 /update {wt=javabin&version=2} {add=[26 (1429062885037834241)]} 0 6
[junit4:junit4] 2> 13076 T567 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10026 (1429062885037834240)]} 0 2
[junit4:junit4] 2> 13077 T549 C25 P41235 /update {wt=javabin&version=2} {add=[10026 (1429062885037834240)]} 0 8
[junit4:junit4] 2> 13122 T529 C24 P33529 /update {wt=javabin&version=2} {add=[27 (1429062885092360192)]} 0 1
[junit4:junit4] 2> 13123 T530 C24 P33529 /update {wt=javabin&version=2} {add=[10027 (1429062885092360193)]} 0 1
[junit4:junit4] 2> 13129 T565 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[27 (1429062885095505920)]} 0 1
[junit4:junit4] 2> 13130 T552 C25 P41235 /update {wt=javabin&version=2} {add=[27 (1429062885095505920)]} 0 5
[junit4:junit4] 2> 13131 T568 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10027 (1429062885095505921)]} 0 3
[junit4:junit4] 2> 13131 T551 C25 P41235 /update {wt=javabin&version=2} {add=[10027 (1429062885095505921)]} 0 6
[junit4:junit4] 2> 13164 T531 C24 P33529 /update {wt=javabin&version=2} {delete=[24 (-1429062885136400384)]} 0 0
[junit4:junit4] 2> 13165 T532 C24 P33529 /update {wt=javabin&version=2} {delete=[10024 (-1429062885137448960)]} 0 0
[junit4:junit4] 2> 13169 T566 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[24 (-1429062885139546112)]} 0 1
[junit4:junit4] 2> 13169 T567 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10024 (-1429062885139546113)]} 0 0
[junit4:junit4] 2> 13169 T550 C25 P41235 /update {wt=javabin&version=2} {delete=[24 (-1429062885139546112)]} 0 3
[junit4:junit4] 2> 13170 T549 C25 P41235 /update {wt=javabin&version=2} {delete=[10024 (-1429062885139546113)]} 0 3
[junit4:junit4] 2> 13172 T529 C24 P33529 /update {wt=javabin&version=2} {add=[28 (1429062885144788992)]} 0 1
[junit4:junit4] 2> 13173 T530 C24 P33529 /update {wt=javabin&version=2} {add=[10028 (1429062885144788993)]} 0 1
[junit4:junit4] 2> 13178 T565 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[28 (1429062885147934720)]} 0 1
[junit4:junit4] 2> 13179 T568 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10028 (1429062885147934721)]} 0 1
[junit4:junit4] 2> 13179 T552 C25 P41235 /update {wt=javabin&version=2} {add=[28 (1429062885147934720)]} 0 5
[junit4:junit4] 2> 13179 T551 C25 P41235 /update {wt=javabin&version=2} {add=[10028 (1429062885147934721)]} 0 4
[junit4:junit4] 2> 13276 T531 C24 P33529 /update {wt=javabin&version=2} {delete=[27 (-1429062885252792321)]} 0 1
[junit4:junit4] 2> 13276 T532 C24 P33529 /update {wt=javabin&version=2} {delete=[10027 (-1429062885252792320)]} 0 1
[junit4:junit4] 2> 13280 T567 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10027 (-1429062885255938049)]} 0 0
[junit4:junit4] 2> 13280 T566 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[27 (-1429062885255938048)]} 0 0
[junit4:junit4] 2> 13281 T552 C25 P41235 /update {wt=javabin&version=2} {delete=[10027 (-1429062885255938049)]} 0 3
[junit4:junit4] 2> 13281 T550 C25 P41235 /update {wt=javabin&version=2} {delete=[27 (-1429062885255938048)]} 0 3
[junit4:junit4] 2> 13284 T530 C24 P33529 /update {wt=javabin&version=2} {add=[29 (1429062885261180928)]} 0 1
[junit4:junit4] 2> 13284 T529 C24 P33529 /update {wt=javabin&version=2} {add=[10029 (1429062885261180929)]} 0 1
[junit4:junit4] 2> 13290 T568 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[29 (1429062885265375232)]} 0 1
[junit4:junit4] 2> 13290 T551 C25 P41235 /update {wt=javabin&version=2} {add=[29 (1429062885265375232)]} 0 4
[junit4:junit4] 2> 13291 T565 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10029 (1429062885265375233)]} 0 1
[junit4:junit4] 2> 13291 T549 C25 P41235 /update {wt=javabin&version=2} {add=[10029 (1429062885265375233)]} 0 5
[junit4:junit4] 2> 13366 T531 C24 P33529 /update {wt=javabin&version=2} {add=[30 (1429062885347164160)]} 0 1
[junit4:junit4] 2> 13366 T530 C24 P33529 /update {wt=javabin&version=2} {add=[10030 (1429062885347164161)]} 0 1
[junit4:junit4] 2> 13372 T567 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[30 (1429062885350309888)]} 0 1
[junit4:junit4] 2> 13372 T552 C25 P41235 /update {wt=javabin&version=2} {add=[30 (1429062885350309888)]} 0 4
[junit4:junit4] 2> 13372 T566 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10030 (1429062885351358464)]} 0 1
[junit4:junit4] 2> 13373 T550 C25 P41235 /update {wt=javabin&version=2} {add=[10030 (1429062885351358464)]} 0 4
[junit4:junit4] 2> 13430 T532 C24 P33529 /update {wt=javabin&version=2} {add=[31 (1429062885414273024)]} 0 1
[junit4:junit4] 2> 13430 T529 C24 P33529 /update {wt=javabin&version=2} {add=[10031 (1429062885414273025)]} 0 1
[junit4:junit4] 2> 13437 T568 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[31 (1429062885417418752)]} 0 1
[junit4:junit4] 2> 13438 T551 C25 P41235 /update {wt=javabin&version=2} {add=[31 (1429062885417418752)]} 0 6
[junit4:junit4] 2> 13438 T565 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10031 (1429062885418467328)]} 0 1
[junit4:junit4] 2> 13440 T549 C25 P41235 /update {wt=javabin&version=2} {add=[10031 (1429062885418467328)]} 0 8
[junit4:junit4] 2> 13514 T531 C24 P33529 /update {wt=javabin&version=2} {delete=[28 (-1429062885503401984)]} 0 0
[junit4:junit4] 2> 13515 T530 C24 P33529 /update {wt=javabin&version=2} {delete=[10028 (-1429062885504450560)]} 0 0
[junit4:junit4] 2> 13521 T567 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[28 (-1429062885507596288)]} 0 1
[junit4:junit4] 2> 13521 T566 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10028 (-1429062885507596289)]} 0 0
[junit4:junit4] 2> 13522 T552 C25 P41235 /update {wt=javabin&version=2} {delete=[28 (-1429062885507596288)]} 0 4
[junit4:junit4] 2> 13522 T550 C25 P41235 /update {wt=javabin&version=2} {delete=[10028 (-1429062885507596289)]} 0 4
[junit4:junit4] 2> 13527 T532 C24 P33529 /update {wt=javabin&version=2} {add=[32 (1429062885514936320)]} 0 2
[junit4:junit4] 2> 13527 T529 C24 P33529 /update {wt=javabin&version=2} {add=[10032 (1429062885515984896)]} 0 2
[junit4:junit4] 2> 13536 T568 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[32 (1429062885520179200)]} 0 2
[junit4:junit4] 2> 13537 T551 C25 P41235 /update {wt=javabin&version=2} {add=[32 (1429062885520179200)]} 0 7
[junit4:junit4] 2> 13537 T565 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10032 (1429062885521227776)]} 0 2
[junit4:junit4] 2> 13538 T549 C25 P41235 /update {wt=javabin&version=2} {add=[10032 (1429062885521227776)]} 0 8
[junit4:junit4] 2> 13587 T514 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 47003
[junit4:junit4] 2> 13588 T514 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1554942991
[junit4:junit4] 2> 13620 T531 C24 P33529 /update {wt=javabin&version=2} {add=[33 (1429062885613502464)]} 0 1
[junit4:junit4] 2> 13621 T530 C24 P33529 /update {wt=javabin&version=2} {add=[10033 (1429062885614551040)]} 0 1
[junit4:junit4] 2> 13630 T567 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[33 (1429062885618745344)]} 0 2
[junit4:junit4] 2> 13631 T552 C25 P41235 /update {wt=javabin&version=2} {add=[33 (1429062885618745344)]} 0 7
[junit4:junit4] 2> 13631 T566 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10033 (1429062885619793920)]} 0 2
[junit4:junit4] 2> 13633 T550 C25 P41235 /update {wt=javabin&version=2} {add=[10033 (1429062885619793920)]} 0 8
[junit4:junit4] 2> 13668 T532 C24 P33529 /update {wt=javabin&version=2} {delete=[33 (-1429062885664882688)]} 0 0
[junit4:junit4] 2> 13669 T529 C24 P33529 /update {wt=javabin&version=2} {delete=[10033 (-1429062885665931264)]} 0 0
[junit4:junit4] 2> 13675 T568 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[33 (-1429062885668028416)]} 0 1
[junit4:junit4] 2> 13676 T551 C25 P41235 /update {wt=javabin&version=2} {delete=[33 (-1429062885668028416)]} 0 5
[junit4:junit4] 2> 13676 T565 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10033 (-1429062885670125568)]} 0 1
[junit4:junit4] 2> 13677 T549 C25 P41235 /update {wt=javabin&version=2} {delete=[10033 (-1429062885670125568)]} 0 5
[junit4:junit4] 2> 13680 T531 C24 P33529 /update {wt=javabin&version=2} {add=[34 (1429062885676417024)]} 0 1
[junit4:junit4] 2> 13682 T530 C24 P33529 /update {wt=javabin&version=2} {add=[10034 (1429062885677465600)]} 0 2
[junit4:junit4] 2> 13690 T567 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[34 (1429062885681659904)]} 0 2
[junit4:junit4] 2> 13692 T552 C25 P41235 /update {wt=javabin&version=2} {add=[34 (1429062885681659904)]} 0 8
[junit4:junit4] 2> 13692 T568 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10034 (1429062885682708480)]} 0 3
[junit4:junit4] 2> 13693 T551 C25 P41235 /update {wt=javabin&version=2} {add=[10034 (1429062885682708480)]} 0 8
[junit4:junit4] 2> 13716 T532 C24 P33529 /update {wt=javabin&version=2} {add=[35 (1429062885714165760)]} 0 1
[junit4:junit4] 2> 13718 T529 C24 P33529 /update {wt=javabin&version=2} {add=[10035 (1429062885715214336)]} 0 2
[junit4:junit4] 2> 13726 T566 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[35 (1429062885719408640)]} 0 2
[junit4:junit4] 2> 13727 T550 C25 P41235 /update {wt=javabin&version=2} {add=[35 (1429062885719408640)]} 0 7
[junit4:junit4] 2> 13727 T565 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10035 (1429062885720457216)]} 0 2
[junit4:junit4] 2> 13728 T549 C25 P41235 /update {wt=javabin&version=2} {add=[10035 (1429062885720457216)]} 0 7
[junit4:junit4] 2> 13782 T531 C24 P33529 /update {wt=javabin&version=2} {delete=[34 (-1429062885784420352)]} 0 0
[junit4:junit4] 2> 13784 T532 C24 P33529 /update {wt=javabin&version=2} {delete=[10034 (-1429062885785468928)]} 0 1
[junit4:junit4] 2> 13789 T567 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[34 (-1429062885788614656)]} 0 1
[junit4:junit4] 2> 13790 T552 C25 P41235 /update {wt=javabin&version=2} {delete=[34 (-1429062885788614656)]} 0 4
[junit4:junit4] 2> 13790 T568 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10034 (-1429062885789663232)]} 0 1
[junit4:junit4] 2> 13791 T551 C25 P41235 /update {wt=javabin&version=2} {delete=[10034 (-1429062885789663232)]} 0 4
[junit4:junit4] 2> 13795 T530 C24 P33529 /update {wt=javabin&version=2} {add=[36 (1429062885795954688)]} 0 2
[junit4:junit4] 2> 13796 T529 C24 P33529 /update {wt=javabin&version=2} {add=[10036 (1429062885797003264)]} 0 2
[junit4:junit4] 2> 13804 T566 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[36 (1429062885801197568)]} 0 2
[junit4:junit4] 2> 13805 T550 C25 P41235 /update {wt=javabin&version=2} {add=[36 (1429062885801197568)]} 0 7
[junit4:junit4] 2> 13805 T565 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10036 (1429062885802246144)]} 0 2
[junit4:junit4] 2> 13806 T549 C25 P41235 /update {wt=javabin&version=2} {add=[10036 (1429062885802246144)]} 0 7
[junit4:junit4] 2> 13853 T531 C24 P33529 /update {wt=javabin&version=2} {add=[37 (1429062885857820672)]} 0 1
[junit4:junit4] 2> 13854 T532 C24 P33529 /update {wt=javabin&version=2} {add=[10037 (1429062885858869248)]} 0 1
[junit4:junit4] 2> 13859 T567 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[37 (1429062885860966400)]} 0 1
[junit4:junit4] 2> 13860 T568 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10037 (1429062885862014976)]} 0 2
[junit4:junit4] 2> 13860 T552 C25 P41235 /update {wt=javabin&version=2} {add=[37 (1429062885860966400)]} 0 5
[junit4:junit4] 2> 13860 T551 C25 P41235 /update {wt=javabin&version=2} {add=[10037 (1429062885862014976)]} 0 4
[junit4:junit4] 2> 13870 T530 C24 P33529 /update {wt=javabin&version=2} {delete=[35 (-1429062885876695040)]} 0 0
[junit4:junit4] 2> 13871 T529 C24 P33529 /update {wt=javabin&version=2} {delete=[10035 (-1429062885876695041)]} 0 1
[junit4:junit4] 2> 13874 T566 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[35 (-1429062885878792192)]} 0 0
[junit4:junit4] 2> 13875 T565 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10035 (-1429062885879840768)]} 0 1
[junit4:junit4] 2> 13875 T550 C25 P41235 /update {wt=javabin&version=2} {delete=[35 (-1429062885878792192)]} 0 3
[junit4:junit4] 2> 13875 T549 C25 P41235 /update {wt=javabin&version=2} {delete=[10035 (-1429062885879840768)]} 0 2
[junit4:junit4] 2> 13878 T531 C24 P33529 /update {wt=javabin&version=2} {add=[38 (1429062885884035072)]} 0 1
[junit4:junit4] 2> 13878 T532 C24 P33529 /update {wt=javabin&version=2} {add=[10038 (1429062885884035073)]} 0 1
[junit4:junit4] 2> 13885 T567 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[38 (1429062885887180800)]} 0 2
[junit4:junit4] 2> 13886 T568 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10038 (1429062885888229376)]} 0 2
[junit4:junit4] 2> 13886 T552 C25 P41235 /update {wt=javabin&version=2} {add=[38 (1429062885887180800)]} 0 6
[junit4:junit4] 2> 13886 T551 C25 P41235 /update {wt=javabin&version=2} {add=[10038 (1429062885888229376)]} 0 5
[junit4:junit4] 2> 13957 T530 C24 P33529 /update {wt=javabin&version=2} {add=[39 (1429062885965824000)]} 0 2
[junit4:junit4] 2> 13957 T529 C24 P33529 /update {wt=javabin&version=2} {add=[10039 (1429062885966872576)]} 0 2
[junit4:junit4] 2> 13957 T537 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 13958 T537 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:47003_ev_p",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:47003/ev_p"}
[junit4:junit4] 2> 13964 T556 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 13964 T572 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 13964 T536 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 13964 T543 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> C23_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=down, core=collection1, collection=collection1, node_name=127.0.0.1:47003_ev_p, base_url=http://127.0.0.1:47003/ev_p}
[junit4:junit4] 2> 13967 T566 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[39 (1429062885971066880)]} 0 3
[junit4:junit4] 2> 13968 T565 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10039 (1429062885971066881)]} 0 2
[junit4:junit4] 2> 13968 T550 C25 P41235 /update {wt=javabin&version=2} {add=[39 (1429062885971066880)]} 0 8
[junit4:junit4] 2> 13969 T549 C25 P41235 /update {wt=javabin&version=2} {add=[10039 (1429062885971066881)]} 0 9
[junit4:junit4] 2> 14009 T531 C24 P33529 /update {wt=javabin&version=2} {delete=[36 (-1429062886022447104)]} 0 0
[junit4:junit4] 2> 14009 T532 C24 P33529 /update {wt=javabin&version=2} {delete=[10036 (-1429062886022447105)]} 0 0
[junit4:junit4] 2> 14012 T552 C25 P41235 /update {wt=javabin&version=2} {delete=[36 (-1429062886025592832)]} 0 0
[junit4:junit4] 2> 14013 T551 C25 P41235 /update {wt=javabin&version=2} {delete=[10036 (-1429062886026641408)]} 0 1
[junit4:junit4] 2> 14017 T530 C24 P33529 /update {wt=javabin&version=2} {add=[40 (1429062886029787136)]} 0 1
[junit4:junit4] 2> 14017 T529 C24 P33529 /update {wt=javabin&version=2} {add=[10040 (1429062886029787137)]} 0 1
[junit4:junit4] 2> 14022 T550 C25 P41235 /update {wt=javabin&version=2} {add=[40 (1429062886035030016)]} 0 2
[junit4:junit4] 2> 14024 T549 C25 P41235 /update {wt=javabin&version=2} {add=[10040 (1429062886035030017)]} 0 3
[junit4:junit4] 2> 14070 T531 C24 P33529 /update {wt=javabin&version=2} {add=[41 (1429062886085361664)]} 0 1
[junit4:junit4] 2> 14072 T532 C24 P33529 /update {wt=javabin&version=2} {add=[10041 (1429062886086410240)]} 0 2
[junit4:junit4] 2> 14075 T552 C25 P41235 /update {wt=javabin&version=2} {add=[41 (1429062886090604544)]} 0 2
[junit4:junit4] 2> 14076 T551 C25 P41235 /update {wt=javabin&version=2} {add=[10041 (1429062886091653120)]} 0 1
[junit4:junit4] 2> 14087 T530 C24 P33529 /update {wt=javabin&version=2} {delete=[39 (-1429062886103187456)]} 0 1
[junit4:junit4] 2> 14088 T529 C24 P33529 /update {wt=javabin&version=2} {delete=[10039 (-1429062886105284608)]} 0 1
[junit4:junit4] 2> 14090 T550 C25 P41235 /update {wt=javabin&version=2} {delete=[39 (-1429062886107381760)]} 0 1
[junit4:junit4] 2> 14091 T552 C25 P41235 /update {wt=javabin&version=2} {delete=[10039 (-1429062886108430336)]} 0 0
[junit4:junit4] 2> 14094 T531 C24 P33529 /update {wt=javabin&version=2} {add=[42 (1429062886110527488)]} 0 1
[junit4:junit4] 2> 14096 T532 C24 P33529 /update {wt=javabin&version=2} {add=[10042 (1429062886111576064)]} 0 2
[junit4:junit4] 2> 14099 T549 C25 P41235 /update {wt=javabin&version=2} {add=[42 (1429062886115770368)]} 0 2
[junit4:junit4] 2> 14101 T551 C25 P41235 /update {wt=javabin&version=2} {add=[10042 (1429062886116818944)]} 0 2
[junit4:junit4] 2> 14111 T530 C24 P33529 /update {wt=javabin&version=2} {delete=[40 (-1429062886129401856)]} 0 1
[junit4:junit4] 2> 14112 T529 C24 P33529 /update {wt=javabin&version=2} {delete=[10040 (-1429062886130450432)]} 0 0
[junit4:junit4] 2> 14114 T550 C25 P41235 /update {wt=javabin&version=2} {delete=[40 (-1429062886132547584)]} 0 0
[junit4:junit4] 2> 14115 T552 C25 P41235 /update {wt=javabin&version=2} {delete=[10040 (-1429062886133596160)]} 0 0
[junit4:junit4] 2> 14119 T531 C24 P33529 /update {wt=javabin&version=2} {add=[43 (1429062886136741888)]} 0 2
[junit4:junit4] 2> 14120 T532 C24 P33529 /update {wt=javabin&version=2} {add=[10043 (1429062886137790464)]} 0 2
[junit4:junit4] 2> 14124 T549 C25 P41235 /update {wt=javabin&version=2} {add=[43 (1429062886140936192)]} 0 2
[junit4:junit4] 2> 14125 T551 C25 P41235 /update {wt=javabin&version=2} {add=[10043 (1429062886143033344)]} 0 2
[junit4:junit4] 2> 14218 T530 C24 P33529 /update {wt=javabin&version=2} {add=[44 (1429062886240550912)]} 0 1
[junit4:junit4] 2> 14220 T531 C24 P33529 /update {wt=javabin&version=2} {add=[10044 (1429062886241599488)]} 0 2
[junit4:junit4] 2> 14223 T550 C25 P41235 /update {wt=javabin&version=2} {add=[44 (1429062886245793792)]} 0 2
[junit4:junit4] 2> 14224 T552 C25 P41235 /update {wt=javabin&version=2} {add=[10044 (1429062886246842368)]} 0 2
[junit4:junit4] 2> 14274 T529 C24 P33529 /update {wt=javabin&version=2} {add=[45 (1429062886298222592)]} 0 2
[junit4:junit4] 2> 14275 T532 C24 P33529 /update {wt=javabin&version=2} {add=[10045 (1429062886300319744)]} 0 1
[junit4:junit4] 2> 14279 T549 C25 P41235 /update {wt=javabin&version=2} {add=[45 (1429062886303465472)]} 0 2
[junit4:junit4] 2> 14281 T551 C25 P41235 /update {wt=javabin&version=2} {add=[10045 (1429062886305562624)]} 0 2
[junit4:junit4] 2> 14369 T530 C24 P33529 /update {wt=javabin&version=2} {delete=[41 (-1429062886398885888)]} 0 1
[junit4:junit4] 2> 14370 T531 C24 P33529 /update {wt=javabin&version=2} {delete=[10041 (-1429062886400983040)]} 0 0
[junit4:junit4] 2> 14371 T550 C25 P41235 /update {wt=javabin&version=2} {delete=[41 (-1429062886402031616)]} 0 0
[junit4:junit4] 2> 14373 T552 C25 P41235 /update {wt=javabin&version=2} {delete=[10041 (-1429062886403080192)]} 0 1
[junit4:junit4] 2> 14375 T529 C24 P33529 /update {wt=javabin&version=2} {add=[46 (1429062886404128768)]} 0 2
[junit4:junit4] 2> 14375 T532 C24 P33529 /update {wt=javabin&version=2} {add=[10046 (1429062886406225920)]} 0 0
[junit4:junit4] 2> 14378 T549 C25 P41235 /update {wt=javabin&version=2} {add=[46 (1429062886408323072)]} 0 1
[junit4:junit4] 2> 14379 T551 C25 P41235 /update {wt=javabin&version=2} {add=[10046 (1429062886409371648)]} 0 1
[junit4:junit4] 2> 14441 T530 C24 P33529 /update {wt=javabin&version=2} {delete=[43 (-1429062886475431936)]} 0 1
[junit4:junit4] 2> 14441 T531 C24 P33529 /update {wt=javabin&version=2} {delete=[10043 (-1429062886475431937)]} 0 0
[junit4:junit4] 2> 14443 T550 C25 P41235 /update {wt=javabin&version=2} {delete=[43 (-1429062886477529088)]} 0 0
[junit4:junit4] 2> 14444 T552 C25 P41235 /update {wt=javabin&version=2} {delete=[10043 (-1429062886477529089)]} 0 1
[junit4:junit4] 2> 14446 T529 C24 P33529 /update {wt=javabin&version=2} {add=[47 (1429062886480674816)]} 0 0
[junit4:junit4] 2> 14446 T532 C24 P33529 /update {wt=javabin&version=2} {add=[10047 (1429062886480674817)]} 0 0
[junit4:junit4] 2> 14449 T549 C25 P41235 /update {wt=javabin&version=2} {add=[47 (1429062886483820544)]} 0 0
[junit4:junit4] 2> 14450 T551 C25 P41235 /update {wt=javabin&version=2} {add=[10047 (1429062886483820545)]} 0 1
[junit4:junit4] 2> 14513 T530 C24 P33529 /update {wt=javabin&version=2} {delete=[44 (-1429062886550929408)]} 0 0
[junit4:junit4] 2> 14513 T531 C24 P33529 /update {wt=javabin&version=2} {delete=[10044 (-1429062886550929409)]} 0 0
[junit4:junit4] 2> 14516 T550 C25 P41235 /update {wt=javabin&version=2} {delete=[44 (-1429062886553026560)]} 0 1
[junit4:junit4] 2> 14516 T549 C25 P41235 /update {wt=javabin&version=2} {delete=[10044 (-1429062886554075136)]} 0 0
[junit4:junit4] 2> 14519 T529 C24 P33529 /update {wt=javabin&version=2} {add=[48 (1429062886556172288)]} 0 1
[junit4:junit4] 2> 14519 T532 C24 P33529 /update {wt=javabin&version=2} {add=[10048 (1429062886556172289)]} 0 1
[junit4:junit4] 2> 14522 T551 C25 P41235 /update {wt=javabin&version=2} {add=[10048 (1429062886559318017)]} 0 1
[junit4:junit4] 2> 14522 T552 C25 P41235 /update {wt=javabin&version=2} {add=[48 (1429062886559318016)]} 0 1
[junit4:junit4] 2> 14560 T531 C24 P33529 /update {wt=javabin&version=2} {add=[10049 (1429062886600212480)]} 0 0
[junit4:junit4] 2> 14561 T530 C24 P33529 /update {wt=javabin&version=2} {add=[49 (1429062886600212481)]} 0 1
[junit4:junit4] 2> 14564 T549 C25 P41235 /update {wt=javabin&version=2} {add=[49 (1429062886603358209)]} 0 1
[junit4:junit4] 2> 14564 T550 C25 P41235 /update {wt=javabin&version=2} {add=[10049 (1429062886603358208)]} 0 1
[junit4:junit4] 2> 14587 T575 C23 P47003 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:41235/ev_p/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 14588 T575 C23 P47003 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:47003/ev_p START replicas=[http://127.0.0.1:41235/ev_p/collection1/] nUpdates=100
[junit4:junit4] 2> 14589 T575 C23 P47003 oasu.PeerSync.sync WARNING no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 14590 T575 C23 P47003 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 14590 T575 C23 P47003 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 14590 T575 C23 P47003 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 14590 T575 C23 P47003 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=tlog{file=./org.apache.solr.cloud.RecoveryZkTest-1362860556924/jetty2/tlog/tlog.0000000000000000000 refcount=1}}
[junit4:junit4] 2> 14590 T575 C23 P47003 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:41235/ev_p/collection1/. core=collection1
[junit4:junit4] 2> 14590 T575 C23 P47003 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 14590 T551 C25 P41235 REQ /get {getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 14594 T552 C25 P41235 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 14626 T552 C25 P41235 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4605346a lockFactory=org.apache.lucene.store.NativeFSLockFactory@12983349),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4605346a lockFactory=org.apache.lucene.store.NativeFSLockFactory@12983349),segFN=segments_2,generation=2,filenames=[_0_MockVariableIntBlock_0.skp, _1.fnm, _1_MockRandom_0.pos, _0_Pulsing41_0.tim, _0_Pulsing41_0.tip, _1.nvm, _0.fnm, _0_Asserting_0.dvd, _0_Memory_0.ram, _1_MockVariableIntBlock_0.tii, _1_MockRandom_0.tip, _1.nvd, _0_1.del, _1_Pulsing41_0.doc, _1_MockRandom_0.sd, _1_MockRandom_0.tim, _0_Asserting_0.dvm, _1_MockVariableIntBlock_0.tib, _1.fdx, _1_Pulsing41_0.tip, _1.fdt, _1_Pulsing41_0.tim, _0_MockRandom_0.doc, _0_MockVariableIntBlock_0.frq, _0_MockRandom_0.pos, _1_MockRandom_0.doc, _1_MockRandom_0.frq, _1_MockVariableIntBlock_0.skp, _0_MockVariableIntBlock_0.doc, _1_MockVariableIntBlock_0.pyl, _0_MockVariableIntBlock_0.pyl, _1_Memory_0.ram, _0_MockVariableIntBlock_0.tib, _1_MockVariableIntBlock_0.doc, _0.nvd, _0_MockVariableIntBlock_0.pos, _0.si, _0_MockVariableIntBlock_0.tii, _1_MockVariableIntBlock_0.pos, _1_MockRandom_0.skp, _0.nvm, _1_1.del, _1_Pulsing41_0.pos, _1_Asserting_0.dvm, _1_MockVariableIntBlock_0.frq, _0_Pulsing41_0.pos, _0_Pulsing41_0.doc, _0_MockRandom_0.sd, _0_MockRandom_0.tip, _1_MockRandom_0.pyl, _1.si, _1_Asserting_0.dvd, segments_2, _0.fdx, _0_MockRandom_0.tim, _0.fdt]
[junit4:junit4] 2> 14627 T552 C25 P41235 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[_0_MockVariableIntBlock_0.skp, _1.fnm, _1_MockRandom_0.pos, _0_Pulsing41_0.tim, _0_Pulsing41_0.tip, _1.nvm, _0.fnm, _0_Asserting_0.dvd, _0_Memory_0.ram, _1_MockVariableIntBlock_0.tii, _1_MockRandom_0.tip, _1.nvd, _0_1.del, _1_Pulsing41_0.doc, _1_MockRandom_0.sd, _1_MockRandom_0.tim, _0_Asserting_0.dvm, _1_MockVariableIntBlock_0.tib, _1.fdx, _1_Pulsing41_0.tip, _1.fdt, _1_Pulsing41_0.tim, _0_MockRandom_0.doc, _0_MockVariableIntBlock_0.frq, _0_MockRandom_0.pos, _1_MockRandom_0.doc, _1_MockRandom_0.frq, _1_MockVariableIntBlock_0.skp, _0_MockVariableIntBlock_0.doc, _1_MockVariableIntBlock_0.pyl, _0_MockVariableIntBlock_0.pyl, _1_Memory_0.ram, _0_MockVariableIntBlock_0.tib, _1_MockVariableIntBlock_0.doc, _0.nvd, _0_MockVariableIntBlock_0.pos, _0.si, _0_MockVariableIntBlock_0.tii, _1_MockVariableIntBlock_0.pos, _1_MockRandom_0.skp, _0.nvm, _1_1.del, _1_Pulsing41_0.pos, _1_Asserting_0.dvm, _1_MockVariableIntBlock_0.frq, _0_Pulsing41_0.pos, _0_Pulsing41_0.doc, _0_MockRandom_0.sd, _0_MockRandom_0.tip, _1_MockRandom_0.pyl, _1.si, _1_Asserting_0.dvd, segments_2, _0.fdx, _0_MockRandom_0.tim, _0.fdt]
[junit4:junit4] 2> 14634 T552 C25 P41235 oass.SolrIndexSearcher.<init> Opening Searcher@10385cb0 realtime
[junit4:junit4] 2> 14635 T552 C25 P41235 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 14635 T552 C25 P41235 /update {waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 41
[junit4:junit4] 2> 14636 T575 C23 P47003 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 14636 T575 C23 P47003 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 14637 T529 C24 P33529 /update {wt=javabin&version=2} {add=[50 (1429062886679904256)]} 0 1
[junit4:junit4] 2> 14638 T531 C24 P33529 /update {wt=javabin&version=2} {add=[10050 (1429062886680952832)]} 0 1
[junit4:junit4] 2> 14638 T550 C25 P41235 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 14638 T550 C25 P41235 REQ /replication {command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=1
[junit4:junit4] 2> 14639 T575 C23 P47003 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 14639 T575 C23 P47003 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 14639 T575 C23 P47003 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 14641 T549 C25 P41235 REQ /replication {command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0
[junit4:junit4] 2> 14642 T552 C25 P41235 /update {wt=javabin&version=2} {add=[10050 (1429062886684098561)]} 0 2
[junit4:junit4] 2> 14642 T551 C25 P41235 /update {wt=javabin&version=2} {add=[50 (1429062886684098560)]} 0 2
[junit4:junit4] 2> 14643 T575 C23 P47003 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 56
[junit4:junit4] 2> 14644 T575 C23 P47003 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-1362860556924/jetty2/index.20130309202251563 forceNew: false
[junit4:junit4] 2> 14644 T575 C23 P47003 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@661cd479 lockFactory=org.apache.lucene.store.NativeFSLockFactory@63dbf1b0) fullCopy=true
[junit4:junit4] 2> 14646 T550 C25 P41235 REQ /replication {file=_0_MockVariableIntBlock_0.skp&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14649 T549 C25 P41235 REQ /replication {file=_1_MockRandom_0.pos&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14652 T552 C25 P41235 REQ /replication {file=_1.fnm&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14655 T551 C25 P41235 REQ /replication {file=_0_Pulsing41_0.tim&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14658 T550 C25 P41235 REQ /replication {file=_0_Pulsing41_0.tip&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14661 T549 C25 P41235 REQ /replication {file=_1.nvm&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14665 T552 C25 P41235 REQ /replication {file=_0.fnm&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14668 T551 C25 P41235 REQ /replication {file=_0_Asserting_0.dvd&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14671 T550 C25 P41235 REQ /replication {file=_1_MockVariableIntBlock_0.tii&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14674 T549 C25 P41235 REQ /replication {file=_0_Memory_0.ram&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14677 T552 C25 P41235 REQ /replication {file=_1_MockRandom_0.tip&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14680 T551 C25 P41235 REQ /replication {file=_1.nvd&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14684 T550 C25 P41235 REQ /replication {file=_0_1.del&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=1
[junit4:junit4] 2> 14687 T549 C25 P41235 REQ /replication {file=_1_MockRandom_0.tim&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=1
[junit4:junit4] 2> 14690 T552 C25 P41235 REQ /replication {file=_1_MockRandom_0.sd&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=1
[junit4:junit4] 2> 14693 T551 C25 P41235 REQ /replication {file=_1_Pulsing41_0.doc&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14696 T550 C25 P41235 REQ /replication {file=_0_Asserting_0.dvm&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14700 T549 C25 P41235 REQ /replication {file=_1_MockVariableIntBlock_0.tib&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14703 T552 C25 P41235 REQ /replication {file=_1.fdx&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14706 T551 C25 P41235 REQ /replication {file=_1_Pulsing41_0.tip&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14709 T550 C25 P41235 REQ /replication {file=_1.fdt&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14712 T549 C25 P41235 REQ /replication {file=_0_MockRandom_0.doc&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14715 T552 C25 P41235 REQ /replication {file=_1_Pulsing41_0.tim&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14718 T551 C25 P41235 REQ /replication {file=_0_MockRandom_0.pos&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14721 T550 C25 P41235 REQ /replication {file=_0_MockVariableIntBlock_0.frq&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14722 T529 C24 P33529 /update {wt=javabin&version=2} {add=[51 (1429062886769033216)]} 0 1
[junit4:junit4] 2> 14722 T532 C24 P33529 /update {wt=javabin&version=2} {add=[10051 (1429062886769033217)]} 0 1
[junit4:junit4] 2> 14731 T551 C25 P41235 REQ /replication {file=_1_MockRandom_0.doc&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14732 T552 C25 P41235 /update {wt=javabin&version=2} {add=[10051 (1429062886779518976)]} 0 1
[junit4:junit4] 2> 14732 T549 C25 P41235 /update {wt=javabin&version=2} {add=[51 (1429062886779518977)]} 0 1
[junit4:junit4] 2> 14734 T550 C25 P41235 REQ /replication {file=_1_MockRandom_0.frq&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14737 T551 C25 P41235 REQ /replication {file=_1_MockVariableIntBlock_0.skp&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14740 T549 C25 P41235 REQ /replication {file=_0_MockVariableIntBlock_0.doc&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14743 T552 C25 P41235 REQ /replication {file=_1_MockVariableIntBlock_0.pyl&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14744 T575 C23 P47003 oash.SnapPuller$DirectoryFileFetcher.fetchPackets WARNING No content recieved for file: {name=_1_MockVariableIntBlock_0.pyl, size=0}
[junit4:junit4] 2> 14746 T550 C25 P41235 REQ /replication {file=_0_MockVariableIntBlock_0.pyl&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14747 T575 C23 P47003 oash.SnapPuller$DirectoryFileFetcher.fetchPackets WARNING No content recieved for file: {name=_0_MockVariableIntBlock_0.pyl, size=0}
[junit4:junit4] 2> 14749 T551 C25 P41235 REQ /replication {file=_1_Memory_0.ram&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14752 T549 C25 P41235 REQ /replication {file=_0_MockVariableIntBlock_0.tib&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14755 T552 C25 P41235 REQ /replication {file=_1_MockVariableIntBlock_0.doc&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14758 T550 C25 P41235 REQ /replication {file=_0.nvd&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14761 T551 C25 P41235 REQ /replication {file=_0_MockVariableIntBlock_0.pos&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14764 T549 C25 P41235 REQ /replication {file=_0.si&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14767 T552 C25 P41235 REQ /replication {file=_0_MockVariableIntBlock_0.tii&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14770 T550 C25 P41235 REQ /replication {file=_1_MockVariableIntBlock_0.pos&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14773 T551 C25 P41235 REQ /replication {file=_1_MockRandom_0.skp&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=1
[junit4:junit4] 2> 14775 T549 C25 P41235 REQ /replication {file=_1_1.del&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14778 T552 C25 P41235 REQ /replication {file=_0.nvm&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14781 T550 C25 P41235 REQ /replication {file=_1_Pulsing41_0.pos&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14784 T551 C25 P41235 REQ /replication {file=_1_Asserting_0.dvm&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14787 T549 C25 P41235 REQ /replication {file=_1_MockVariableIntBlock_0.frq&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14788 T530 C24 P33529 /update {wt=javabin&version=2} {delete=[45 (-1429062886839287808)]} 0 0
[junit4:junit4] 2> 14788 T531 C24 P33529 /update {wt=javabin&version=2} {delete=[10045 (-1429062886839287809)]} 0 0
[junit4:junit4] 2> 14790 T552 C25 P41235 REQ /replication {file=_0_MockRandom_0.tip&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14791 T551 C25 P41235 /update {wt=javabin&version=2} {delete=[45 (-1429062886842433536)]} 0 0
[junit4:junit4] 2> 14791 T550 C25 P41235 /update {wt=javabin&version=2} {delete=[10045 (-1429062886842433537)]} 0 0
[junit4:junit4] 2> 14794 T549 C25 P41235 REQ /replication {file=_0_MockRandom_0.sd&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=1
[junit4:junit4] 2> 14794 T532 C24 P33529 /update {wt=javabin&version=2} {add=[52 (1429062886845579264)]} 0 1
[junit4:junit4] 2> 14794 T529 C24 P33529 /update {wt=javabin&version=2} {add=[10052 (1429062886845579265)]} 0 1
[junit4:junit4] 2> 14797 T551 C25 P41235 REQ /replication {file=_0_Pulsing41_0.doc&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=1
[junit4:junit4] 2> 14797 T552 C25 P41235 /update {wt=javabin&version=2} {add=[52 (1429062886848724992)]} 0 0
[junit4:junit4] 2> 14798 T550 C25 P41235 /update {wt=javabin&version=2} {add=[10052 (1429062886848724993)]} 0 1
[junit4:junit4] 2> 14799 T549 C25 P41235 REQ /replication {file=_0_Pulsing41_0.pos&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14802 T551 C25 P41235 REQ /replication {file=_1_MockRandom_0.pyl&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14803 T575 C23 P47003 oash.SnapPuller$DirectoryFileFetcher.fetchPackets WARNING No content recieved for file: {name=_1_MockRandom_0.pyl, size=0}
[junit4:junit4] 2> 14806 T552 C25 P41235 REQ /replication {file=_1.si&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14809 T550 C25 P41235 REQ /replication {file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14812 T549 C25 P41235 REQ /replication {file=_1_Asserting_0.dvd&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14815 T551 C25 P41235 REQ /replication {file=_0.fdx&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14818 T552 C25 P41235 REQ /replication {file=_0_MockRandom_0.tim&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14821 T550 C25 P41235 REQ /replication {file=_0.fdt&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14822 T575 C23 P47003 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 14823 T575 C23 P47003 oash.SnapPuller.modifyIndexProps New index installed. Updating index properties... index=index.20130309202251563
[junit4:junit4] 2> 14825 T575 C23 P47003 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 14826 T575 C23 P47003 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 14826 T575 C23 P47003 oasu.DefaultSolrCoreState.newIndexWriter Rollback old IndexWriter... core=collection1
[junit4:junit4] 2> 14827 T575 C23 P47003 oasc.SolrCore.getNewIndexDir New index directory detected: old=./org.apache.solr.cloud.RecoveryZkTest-1362860556924/jetty2/index/ new=./org.apache.solr.cloud.RecoveryZkTest-1362860556924/jetty2/index.20130309202251563
[junit4:junit4] 2> 14828 T575 C23 P47003 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@661cd479 lockFactory=org.apache.lucene.store.NativeFSLockFactory@63dbf1b0),segFN=segments_2,generation=2,filenames=[_0_MockVariableIntBlock_0.skp, _1_MockRandom_0.pos, _1.fnm, _0_Pulsing41_0.tim, _0_Pulsing41_0.tip, _1.nvm, _0.fnm, _0_Asserting_0.dvd, _0_Memory_0.ram, _1_MockVariableIntBlock_0.tii, _1_MockRandom_0.tip, _1.nvd, _0_1.del, _1_Pulsing41_0.doc, _1_MockRandom_0.tim, _1_MockRandom_0.sd, _0_Asserting_0.dvm, _1_MockVariableIntBlock_0.tib, _1.fdx, _1_Pulsing41_0.tip, _1.fdt, _1_Pulsing41_0.tim, _0_MockRandom_0.doc, _0_MockVariableIntBlock_0.frq, _0_MockRandom_0.pos, _1_MockRandom_0.doc, _1_MockRandom_0.frq, _1_MockVariableIntBlock_0.skp, _0_MockVariableIntBlock_0.doc, _1_MockVariableIntBlock_0.pyl, _0_MockVariableIntBlock_0.pyl, _1_Memory_0.ram, _0_MockVariableIntBlock_0.tib, _1_MockVariableIntBlock_0.doc, _0.nvd, _0_MockVariableIntBlock_0.pos, _0.si, _0_MockVariableIntBlock_0.tii, _1_MockVariableIntBlock_0.pos, _1_MockRandom_0.skp, _0.nvm, _1_1.del, _1_Pulsing41_0.pos, _1_Asserting_0.dvm, _1_MockVariableIntBlock_0.frq, _0_Pulsing41_0.pos, _0_Pulsing41_0.doc, _0_MockRandom_0.sd, _0_MockRandom_0.tip, _1_MockRandom_0.pyl, _1.si, _1_Asserting_0.dvd, segments_2, _0.fdx, _0_MockRandom_0.tim, _0.fdt]
[junit4:junit4] 2> 14829 T575 C23 P47003 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[_0_MockVariableIntBlock_0.skp, _1_MockRandom_0.pos, _1.fnm, _0_Pulsing41_0.tim, _0_Pulsing41_0.tip, _1.nvm, _0.fnm, _0_Asserting_0.dvd, _0_Memory_0.ram, _1_MockVariableIntBlock_0.tii, _1_MockRandom_0.tip, _1.nvd, _0_1.del, _1_Pulsing41_0.doc, _1_MockRandom_0.tim, _1_MockRandom_0.sd, _0_Asserting_0.dvm, _1_MockVariableIntBlock_0.tib, _1.fdx, _1_Pulsing41_0.tip, _1.fdt, _1_Pulsing41_0.tim, _0_MockRandom_0.doc, _0_MockVariableIntBlock_0.frq, _0_MockRandom_0.pos, _1_MockRandom_0.doc, _1_MockRandom_0.frq, _1_MockVariableIntBlock_0.skp, _0_MockVariableIntBlock_0.doc, _1_MockVariableIntBlock_0.pyl, _0_MockVariableIntBlock_0.pyl, _1_Memory_0.ram, _0_MockVariableIntBlock_0.tib, _1_MockVariableIntBlock_0.doc, _0.nvd, _0_MockVariableIntBlock_0.pos, _0.si, _0_MockVariableIntBlock_0.tii, _1_MockVariableIntBlock_0.pos, _1_MockRandom_0.skp, _0.nvm, _1_1.del, _1_Pulsing41_0.pos, _1_Asserting_0.dvm, _1_MockVariableIntBlock_0.frq, _0_Pulsing41_0.pos, _0_Pulsing41_0.doc, _0_MockRandom_0.sd, _0_MockRandom_0.tip, _1_MockRandom_0.pyl, _1.si, _1_Asserting_0.dvd, segments_2, _0.fdx, _0_MockRandom_0.tim, _0.fdt]
[junit4:junit4] 2> 14829 T575 C23 P47003 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 14831 T575 C23 P47003 oass.SolrIndexSearcher.<init> Opening Searcher@527cd052 main
[junit4:junit4] 2> 14833 T574 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@527cd052 main{StandardDirectoryReader(segments_2:5:nrt _0(5.0):C75/31 _1(5.0):C23/11)}
[junit4:junit4] 2> 14833 T575 C23 P47003 oash.SnapPuller$2.preClose removing old index files MockDirWrapper(org.apache.lucene.store.RAMDirectory@28e04e24 lockFactory=org.apache.lucene.store.NativeFSLockFactory@64562b7d)
[junit4:junit4] 2> 14833 T575 C23 P47003 oasc.CachingDirectoryFactory.closeDirectory Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-1362860556924/jetty2/index
[junit4:junit4] 2> 14833 T575 C23 P47003 oash.SnapPuller$2.postClose removing old index directory MockDirWrapper(org.apache.lucene.store.RAMDirectory@28e04e24 lockFactory=org.apache.lucene.store.NativeFSLockFactory@64562b7d)
[junit4:junit4] 2> 14836 T575 C23 P47003 oasc.RecoveryStrategy.replay Replaying buffered documents. core=collection1
[junit4:junit4] 2> 14836 T583 C23 P47003 oasu.UpdateLog$LogReplayer.doReplay WARNING Starting log replay tlog{file=./org.apache.solr.cloud.RecoveryZkTest-1362860556924/jetty2/tlog/tlog.0000000000000000000 refcount=2} active=true starting pos=5508
[junit4:junit4] 2> 14837 T583 C23 P47003 oasu.DirectUpdateHandler2.commit start commit{flags=2,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 14838 T583 C23 P47003 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@661cd479 lockFactory=org.apache.lucene.store.NativeFSLockFactory@63dbf1b0),segFN=segments_2,generation=2,filenames=[_0_MockVariableIntBlock_0.skp, _1_MockRandom_0.pos, _1.fnm, _0_Pulsing41_0.tim, _0_Pulsing41_0.tip, _1.nvm, _0.fnm, _0_Asserting_0.dvd, _0_Memory_0.ram, _1_MockVariableIntBlock_0.tii, _1_MockRandom_0.tip, _1.nvd, _0_1.del, _1_Pulsing41_0.doc, _1_MockRandom_0.tim, _1_MockRandom_0.sd, _0_Asserting_0.dvm, _1_MockVariableIntBlock_0.tib, _1.fdx, _1_Pulsing41_0.tip, _1.fdt, _1_Pulsing41_0.tim, _0_MockRandom_0.doc, _0_MockVariableIntBlock_0.frq, _0_MockRandom_0.pos, _1_MockRandom_0.doc, _1_MockRandom_0.frq, _1_MockVariableIntBlock_0.skp, _0_MockVariableIntBlock_0.doc, _1_MockVariableIntBlock_0.pyl, _0_MockVariableIntBlock_0.pyl, _1_Memory_0.ram, _0_MockVariableIntBlock_0.tib, _1_MockVariableIntBlock_0.doc, _0.nvd, _0_MockVariableIntBlock_0.pos, _0.si, _0_MockVariableIntBlock_0.tii, _1_MockVariableIntBlock_0.pos, _1_MockRandom_0.skp, _0.nvm, _1_1.del, _1_Pulsing41_0.pos, _1_Asserting_0.dvm, _1_MockVariableIntBlock_0.frq, _0_Pulsing41_0.pos, _0_Pulsing41_0.doc, _0_MockRandom_0.sd, _0_MockRandom_0.tip, _1_MockRandom_0.pyl, _1.si, _1_Asserting_0.dvd, segments_2, _0.fdx, _0_MockRandom_0.tim, _0.fdt]
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@661cd479 lockFactory=org.apache.lucene.store.NativeFSLockFactory@63dbf1b0),segFN=segments_3,generation=3,filenames=[_0_MockVariableIntBlock_0.skp, _1_MockRandom_0.pos, _1.fnm, _0_Pulsing41_0.tim, _0_Pulsing41_0.tip, _1.nvm, _0.fnm, _0_Asserting_0.dvd, _0_Memory_0.ram, _1_MockVariableIntBlock_0.tii, _1_MockRandom_0.tip, _1.nvd, _0_1.del, _1_Pulsing41_0.doc, _1_MockRandom_0.tim, _1_MockRandom_0.sd, _0_Asserting_0.dvm, _1_MockVariableIntBlock_0.tib, _1.fdx, _1_Pulsing41_0.tip, _1.fdt, _1_Pulsing41_0.tim, _0_MockRandom_0.doc, _0_MockVariableIntBlock_0.frq, _0_MockRandom_0.pos, _1_MockRandom_0.doc, _1_MockRandom_0.frq, _1_MockVariableIntBlock_0.skp, _0_MockVariableIntBlock_0.doc, _1_MockVariableIntBlock_0.pyl, _0_MockVariableIntBlock_0.pyl, _1_Memory_0.ram, _0_MockVariableIntBlock_0.tib, _1_MockVariableIntBlock_0.doc, _0.nvd, _0_MockVariableIntBlock_0.pos, _0.si, _0_MockVariableIntBlock_0.tii, _1_MockVariableIntBlock_0.pos, _1_MockRandom_0.skp, _0.nvm, _1_1.del, _1_Pulsing41_0.pos, _1_Asserting_0.dvm, _1_MockVariableIntBlock_0.frq, _0_Pulsing41_0.pos, _0_Pulsing41_0.doc, _0_MockRandom_0.sd, _0_MockRandom_0.tip, _1_MockRandom_0.pyl, _1.si, _1_Asserting_0.dvd, _0.fdx, _0_MockRandom_0.tim, segments_3, _0.fdt]
[junit4:junit4] 2> 14838 T583 C23 P47003 oasc.SolrDeletionPolicy.updateCommits newest commit = 3[_0_MockVariableIntBlock_0.skp, _1_MockRandom_0.pos, _1.fnm, _0_Pulsing41_0.tim, _0_Pulsing41_0.tip, _1.nvm, _0.fnm, _0_Asserting_0.dvd, _0_Memory_0.ram, _1_MockVariableIntBlock_0.tii, _1_MockRandom_0.tip, _1.nvd, _0_1.del, _1_Pulsing41_0.doc, _1_MockRandom_0.tim, _1_MockRandom_0.sd, _0_Asserting_0.dvm, _1_MockVariableIntBlock_0.tib, _1.fdx, _1_Pulsing41_0.tip, _1.fdt, _1_Pulsing41_0.tim, _0_MockRandom_0.doc, _0_MockVariableIntBlock_0.frq, _0_MockRandom_0.pos, _1_MockRandom_0.doc, _1_MockRandom_0.frq, _1_MockVariableIntBlock_0.skp, _0_MockVariableIntBlock_0.doc, _1_MockVariableIntBlock_0.pyl, _0_MockVariableIntBlock_0.pyl, _1_Memory_0.ram, _0_MockVariableIntBlock_0.tib, _1_MockVariableIntBlock_0.doc, _0.nvd, _0_MockVariableIntBlock_0.pos, _0.si, _0_MockVariableIntBlock_0.tii, _1_MockVariableIntBlock_0.pos, _1_MockRandom_0.skp, _0.nvm, _1_1.del, _1_Pulsing41_0.pos, _1_Asserting_0.dvm, _1_MockVariableIntBlock_0.frq, _0_Pulsing41_0.pos, _0_Pulsing41_0.doc, _0_MockRandom_0.sd, _0_MockRandom_0.tip, _1_MockRandom_0.pyl, _1.si, _1_Asserting_0.dvd, _0.fdx, _0_MockRandom_0.tim, segments_3, _0.fdt]
[junit4:junit4] 2> 14839 T583 C23 P47003 oass.SolrIndexSearcher.<init> Opening Searcher@45d81be9 main
[junit4:junit4] 2> 14839 T583 C23 P47003 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 14840 T574 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@45d81be9 main{StandardDirectoryReader(segments_2:5:nrt _0(5.0):C75/31 _1(5.0):C23/11)}
[junit4:junit4] 2> 14840 T583 C23 P47003 oasu.UpdateLog$LogReplayer.run WARNING Log replay finished. recoveryInfo=RecoveryInfo{adds=0 deletes=0 deleteByQuery=0 errors=0 positionOfStart=5508}
[junit4:junit4] 2> 14840 T575 C23 P47003 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 14841 T575 C23 P47003 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 14841 T575 C23 P47003 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 14843 T575 C23 P47003 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 14891 T530 C24 P33529 /update {wt=javabin&version=2} {delete=[47 (-1429062886947291136)]} 0 1
[junit4:junit4] 2> 14891 T531 C24 P33529 /update {wt=javabin&version=2} {delete=[10047 (-1429062886947291137)]} 0 0
[junit4:junit4] 2> 14895 T549 C25 P41235 /update {wt=javabin&version=2} {delete=[47 (-1429062886950436864)]} 0 1
[junit4:junit4] 2> 14895 T551 C25 P41235 /update {wt=javabin&version=2} {delete=[10047 (-1429062886951485440)]} 0 0
[junit4:junit4] 2> 14899 T532 C24 P33529 /update {wt=javabin&version=2} {add=[53 (1429062886954631168)]} 0 1
[junit4:junit4] 2> 14900 T529 C24 P33529 /update {wt=javabin&version=2} {add=[10053 (1429062886955679744)]} 0 2
[junit4:junit4] 2> 14904 T552 C25 P41235 /update {wt=javabin&version=2} {add=[53 (1429062886959874048)]} 0 1
[junit4:junit4] 2> 14904 T550 C25 P41235 /update {wt=javabin&version=2} {add=[10053 (1429062886959874049)]} 0 1
[junit4:junit4] 2> 14923 T530 C24 P33529 /update {wt=javabin&version=2} {add=[54 (1429062886979796992)]} 0 2
[junit4:junit4] 2> 14924 T531 C24 P33529 /update {wt=javabin&version=2} {add=[10054 (1429062886979796993)]} 0 2
[junit4:junit4] 2> 14928 T549 C25 P41235 /update {wt=javabin&version=2} {add=[54 (1429062886983991296)]} 0 2
[junit4:junit4] 2> 14929 T552 C25 P41235 /update {wt=javabin&version=2} {add=[10054 (1429062886986088448)]} 0 2
[junit4:junit4] 2> 14966 T532 C24 P33529 /update {wt=javabin&version=2} {add=[55 (1429062887024885760)]} 0 1
[junit4:junit4] 2> 14967 T529 C24 P33529 /update {wt=javabin&version=2} {add=[10055 (1429062887025934336)]} 0 1
[junit4:junit4] 2> 14969 T551 C25 P41235 /update {wt=javabin&version=2} {add=[55 (1429062887028031488)]} 0 1
[junit4:junit4] 2> 14970 T550 C25 P41235 /update {wt=javabin&version=2} {add=[10055 (1429062887029080064)]} 0 1
[junit4:junit4] 2> 15027 T530 C24 P33529 /update {wt=javabin&version=2} {add=[56 (1429062887089897472)]} 0 1
[junit4:junit4] 2> 15028 T531 C24 P33529 /update {wt=javabin&version=2} {add=[10056 (1429062887089897473)]} 0 1
[junit4:junit4] 2> 15031 T549 C25 P41235 /update {wt=javabin&version=2} {add=[56 (1429062887093043200)]} 0 1
[junit4:junit4] 2> 15031 T552 C25 P41235 /update {wt=javabin&version=2} {add=[10056 (1429062887094091776)]} 0 1
[junit4:junit4] 2> 15048 T532 C24 P33529 /update {wt=javabin&version=2} {delete=[48 (-1429062887111917568)]} 0 0
[junit4:junit4] 2> 15048 T530 C24 P33529 /update {wt=javabin&version=2} {delete=[10048 (-1429062887111917569)]} 0 0
[junit4:junit4] 2> 15052 T551 C25 P41235 /update {wt=javabin&version=2} {delete=[48 (-1429062887116111872)]} 0 1
[junit4:junit4] 2> 15052 T550 C25 P41235 /update {wt=javabin&version=2} {delete=[10048 (-1429062887116111873)]} 0 0
[junit4:junit4] 2> 15056 T529 C24 P33529 /update {wt=javabin&version=2} {add=[57 (1429062887119257600)]} 0 1
[junit4:junit4] 2> 15056 T531 C24 P33529 /update {wt=javabin&version=2} {add=[10057 (1429062887119257601)]} 0 1
[junit4:junit4] 2> 15061 T552 C25 P41235 /update {wt=javabin&version=2} {add=[10057 (1429062887124500480)]} 0 1
[junit4:junit4] 2> 15061 T549 C25 P41235 /update {wt=javabin&version=2} {add=[57 (1429062887124500481)]} 0 1
[junit4:junit4] 2> 15092 T532 C24 P33529 /update {wt=javabin&version=2} {delete=[10051 (-1429062887157006336)]} 0 1
[junit4:junit4] 2> 15092 T530 C24 P33529 /update {wt=javabin&version=2} {delete=[51 (-1429062887158054912)]} 0 0
[junit4:junit4] 2> 15095 T551 C25 P41235 /update {wt=javabin&version=2} {delete=[10051 (-1429062887161200640)]} 0 0
[junit4:junit4] 2> 15095 T550 C25 P41235 /update {wt=javabin&version=2} {delete=[51 (-1429062887161200641)]} 0 0
[junit4:junit4] 2> 15099 T529 C24 P33529 /update {wt=javabin&version=2} {add=[10058 (1429062887164346368)]} 0 1
[junit4:junit4] 2> 15100 T531 C24 P33529 /update {wt=javabin&version=2} {add=[58 (1429062887165394944)]} 0 1
[junit4:junit4] 2> 15104 T552 C25 P41235 /update {wt=javabin&version=2} {add=[10058 (1429062887169589248)]} 0 1
[junit4:junit4] 2> 15104 T549 C25 P41235 /update {wt=javabin&version=2} {add=[58 (1429062887169589249)]} 0 1
[junit4:junit4] 2> 15132 T532 C24 P33529 /update {wt=javabin&version=2} {add=[10059 (1429062887197900800)]} 0 2
[junit4:junit4] 2> 15133 T530 C24 P33529 /update {wt=javabin&version=2} {add=[59 (1429062887198949376)]} 0 2
[junit4:junit4] 2> 15136 T551 C25 P41235 /update {wt=javabin&version=2} {add=[10059 (1429062887204192256)]} 0 1
[junit4:junit4] 2> 15137 T550 C25 P41235 /update {wt=javabin&version=2} {add=[59 (1429062887204192257)]} 0 1
[junit4:junit4] 2> 15204 T529 C24 P33529 /update {wt=javabin&version=2} {delete=[10052 (-1429062887275495424)]} 0 0
[junit4:junit4] 2> 15205 T531 C24 P33529 /update {wt=javabin&version=2} {delete=[52 (-1429062887275495425)]} 0 1
[junit4:junit4] 2> 15208 T552 C25 P41235 /update {wt=javabin&version=2} {delete=[10052 (-1429062887278641152)]} 0 1
[junit4:junit4] 2> 15208 T549 C25 P41235 /update {wt=javabin&version=2} {delete=[52 (-1429062887279689728)]} 0 0
[junit4:junit4] 2> 15212 T532 C24 P33529 /update {wt=javabin&version=2} {add=[10060 (1429062887282835456)]} 0 1
[junit4:junit4] 2> 15213 T530 C24 P33529 /update {wt=javabin&version=2} {add=[60 (1429062887283884032)]} 0 2
[junit4:junit4] 2> 15217 T551 C25 P41235 /update {wt=javabin&version=2} {add=[10060 (1429062887288078336)]} 0 1
[junit4:junit4] 2> 15218 T550 C25 P41235 /update {wt=javabin&version=2} {add=[60 (1429062887289126912)]} 0 1
[junit4:junit4] 2> 15292 T529 C24 P33529 /update {wt=javabin&version=2} {add=[10061 (1429062887366721536)]} 0 2
[junit4:junit4] 2> 15292 T531 C24 P33529 /update {wt=javabin&version=2} {add=[61 (1429062887366721537)]} 0 1
[junit4:junit4] 2> 15296 T552 C25 P41235 /update {wt=javabin&version=2} {add=[10061 (1429062887370915840)]} 0 1
[junit4:junit4] 2> 15296 T549 C25 P41235 /update {wt=javabin&version=2} {add=[61 (1429062887371964416)]} 0 1
[junit4:junit4] 2> 15389 T532 C24 P33529 /update {wt=javabin&version=2} {add=[10062 (1429062887468433409)]} 0 1
[junit4:junit4] 2> 15390 T530 C24 P33529 /update {wt=javabin&version=2} {add=[62 (1429062887468433408)]} 0 2
[junit4:junit4] 2> 15394 T551 C25 P41235 /update {wt=javabin&version=2} {add=[10062 (1429062887473676288)]} 0 1
[junit4:junit4] 2> 15394 T552 C25 P41235 /update {wt=javabin&version=2} {add=[62 (1429062887473676289)]} 0 1
[junit4:junit4] 2> 15408 T529 C24 P33529 /update {wt=javabin&version=2} {delete=[10054 (-1429062887489404928)]} 0 0
[junit4:junit4] 2> 15409 T531 C24 P33529 /update {wt=javabin&version=2} {delete=[54 (-1429062887490453504)]} 0 1
[junit4:junit4] 2> 15412 T550 C25 P41235 /update {wt=javabin&version=2} {delete=[10054 (-1429062887493599232)]} 0 1
[junit4:junit4] 2> 15412 T549 C25 P41235 /update {wt=javabin&version=2} {delete=[54 (-1429062887493599233)]} 0 0
[junit4:junit4] 2> 15416 T532 C24 P33529 /update {wt=javabin&version=2} {add=[10063 (1429062887496744960)]} 0 1
[junit4:junit4] 2> 15417 T530 C24 P33529 /update {wt=javabin&version=2} {add=[63 (1429062887497793536)]} 0 1
[junit4:junit4] 2> 15420 T551 C25 P41235 /update {wt=javabin&version=2} {add=[10063 (1429062887500939264)]} 0 1
[junit4:junit4] 2> 15421 T552 C25 P41235 /update {wt=javabin&version=2} {add=[63 (1429062887501987840)]} 0 1
[junit4:junit4] 2> 15471 T537 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 15473 T537 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:47003_ev_p",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:47003/ev_p"}
[junit4:junit4] 2> 15479 T556 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 15479 T536 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 15479 T572 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 15479 T543 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 15508 T529 C24 P33529 /update {wt=javabin&version=2} {add=[10064 (1429062887593213952)]} 0 1
[junit4:junit4] 2> 15508 T532 C24 P33529 /update {wt=javabin&version=2} {add=[64 (1429062887593213953)]} 0 1
[junit4:junit4] 2> C23_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:47003_ev_p, base_url=http://127.0.0.1:47003/ev_p}
[junit4:junit4] 2> 15518 T566 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[64 (1429062887598456833)]} 0 2
[junit4:junit4] 2> 15518 T567 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10064 (1429062887598456832)]} 0 3
[junit4:junit4] 2> 15520 T549 C25 P41235 /update {wt=javabin&version=2} {add=[64 (1429062887598456833)]} 0 8
[junit4:junit4] 2> 15520 T550 C25 P41235 /update {wt=javabin&version=2} {add=[10064 (1429062887598456832)]} 0 9
[junit4:junit4] 2> 15557 T531 C24 P33529 /update {wt=javabin&version=2} {add=[65 (1429062887644594176)]} 0 1
[junit4:junit4] 2> 15558 T530 C24 P33529 /update {wt=javabin&version=2} {add=[10065 (1429062887645642752)]} 0 2
[junit4:junit4] 2> 15566 T568 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[65 (1429062887649837056)]} 0 2
[junit4:junit4] 2> 15566 T565 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10065 (1429062887649837057)]} 0 1
[junit4:junit4] 2> 15567 T551 C25 P41235 /update {wt=javabin&version=2} {add=[65 (1429062887649837056)]} 0 6
[junit4:junit4] 2> 15567 T552 C25 P41235 /update {wt=javabin&version=2} {add=[10065 (1429062887649837057)]} 0 6
[junit4:junit4] 2> 15592 T514 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 15593 T514 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 15598 T514 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@316ab910
[junit4:junit4] 2> 15609 T514 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=4,adds=4,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=54,cumulative_deletesById=16,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 15609 T529 C24 P33529 /update {wt=javabin&version=2} {add=[66 (1429062887699120128)]} 0 1
[junit4:junit4] 2> 15620 T514 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 15620 T532 C24 P33529 /update {wt=javabin&version=2} {add=[10066 (1429062887700168704)]} 0 11
[junit4:junit4] 2> 15620 T514 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 15621 T514 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 15628 T566 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {} 0 1
[junit4:junit4] 2> 15629 T567 C23 P47003 /update {distrib.from=http://127.0.0.1:41235/ev_p/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {} 0 1
[junit4:junit4] 2> 15629 T566 C23 P47003 oasc.SolrException.log SEVERE java.lang.RuntimeException: SolrCoreState already closed
[junit4:junit4] 2> at org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:84)
[junit4:junit4] 2> at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:153)
[junit4:junit4] 2> at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:69)
[junit4:junit4] 2> at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51)
[junit4:junit4] 2> at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:451)
[junit4:junit4] 2> at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:587)
[junit4:junit4] 2> at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:346)
[junit4:junit4] 2> at org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:100)
[junit4:junit4] 2> at org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:246)
[junit4:junit4] 2> at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:173)
[junit4:junit4] 2> at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
[junit4:junit4] 2> at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
[junit4:junit4] 2> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
[junit4:junit4] 2> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1800)
[junit4:junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:637)
[junit4:junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:343)
[junit4:junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:141)
[junit4:junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
[junit4:junit4] 2> at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:135)
[junit4:junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
[junit4:junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453)
[junit4:junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.GzipHandler.handle(GzipHandler.java:275)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1074)
[junit4:junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:382)
[junit4:junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1006)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
[junit4:junit4] 2> at org.eclipse.jetty.server.Server.handle(Server.java:365)
[junit4:junit4] 2> at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:485)
[junit4:junit4] 2> at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:937)
[junit4:junit4] 2> at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:998)
[junit4:junit4] 2> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:948)
[junit4:junit4] 2> at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
[junit4:junit4] 2> at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
[junit4:junit4] 2> at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:628)
[junit4:junit4] 2> at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
[junit4:junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4:junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:662)
[junit4:junit4] 2>
[junit4:junit4] 2> 15630 T567 C23 P47003 oasc.SolrException.log SEVERE java.lang.RuntimeException: SolrCoreState already closed
[junit4:junit4] 2> at org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:84)
[junit4:junit4] 2> at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:153)
[junit4:junit4] 2> at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:69)
[junit4:junit4] 2> at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51)
[junit4:junit4] 2> at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:451)
[junit4:junit4] 2> at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:587)
[junit4:junit4] 2> at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:346)
[junit4:junit4] 2> at org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:100)
[junit4:junit4] 2> at org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:246)
[junit4:junit4] 2> at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:173)
[junit4:junit4] 2> at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
[junit4:junit4] 2> at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
[junit4:junit4] 2> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
[junit4:junit4] 2> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1800)
[junit4:junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:637)
[junit4:junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:343)
[junit4:junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:141)
[junit4:junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
[junit4:junit4] 2> at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:135)
[junit4:junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
[junit4:junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453)
[junit4:junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.GzipHandler.handle(GzipHandler.java:275)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1074)
[junit4:junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:382)
[junit4:junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1006)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
[junit4:junit4] 2> at org.eclipse.jetty.server.Server.handle(Server.java:365)
[junit4:junit4] 2> at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:485)
[junit4:junit4] 2> at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:937)
[junit4:junit4] 2> at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:998)
[junit4:junit4] 2> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:948)
[junit4:junit4] 2> at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
[junit4:junit4] 2> at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
[junit4:junit4] 2> at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:628)
[junit4:junit4] 2> at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
[junit4:junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4:junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:662)
[junit4:junit4] 2>
[junit4:junit4] 2> 15632 T566 C23 P47003 oasc.SolrException.log SEVERE null:java.lang.RuntimeException: SolrCoreState already closed
[junit4:junit4] 2> at org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:84)
[junit4:junit4] 2> at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:153)
[junit4:junit4] 2> at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:69)
[junit4:junit4] 2> at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51)
[junit4:junit4] 2> at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:451)
[junit4:junit4] 2> at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:587)
[junit4:junit4] 2> at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:346)
[junit4:junit4] 2> at org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:100)
[junit4:junit4] 2> at org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:246)
[junit4:junit4] 2> at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:173)
[junit4:junit4] 2> at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
[junit4:junit4] 2> at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
[junit4:junit4] 2> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
[junit4:junit4] 2> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1800)
[junit4:junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:637)
[junit4:junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:343)
[junit4:junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:141)
[junit4:junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
[junit4:junit4] 2> at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:135)
[junit4:junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
[junit4:junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453)
[junit4:junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.GzipHandler.handle(GzipHandler.java:275)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1074)
[junit4:junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:382)
[junit4:junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1006)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
[junit4:junit4] 2> at org.eclipse.jetty.server.Server.handle(Server.java:365)
[junit4:junit4] 2> at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:485)
[junit4:junit4] 2> at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:937)
[junit4:junit4] 2> at org.eclipse.jetty.server.AbstractHttpConnection$Req
[...truncated too long message...]
but Solr cannot talk to ZK
[junit4:junit4] 2> 34595 T556 oascc.ZkStateReader$3.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 34596 T556 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 35080 T517 oazs.SessionTrackerImpl.run SessionTrackerImpl exited loop!
[junit4:junit4] 2> NOTE: test params are: codec=Lucene42: {range_facet_l=MockVariableIntBlock(baseBlockSize=126), text=PostingsFormat(name=Memory doPackFST= true), _version_=Pulsing41(freqCutoff=5 minBlockSize=28 maxBlockSize=136), rnd_b=PostingsFormat(name=MockRandom), intDefault=Pulsing41(freqCutoff=5 minBlockSize=28 maxBlockSize=136), id=PostingsFormat(name=MockRandom), timestamp=MockVariableIntBlock(baseBlockSize=126), a_t=MockVariableIntBlock(baseBlockSize=126), range_facet_sl=PostingsFormat(name=MockRandom), range_facet_si=MockVariableIntBlock(baseBlockSize=126), other_tl1=Pulsing41(freqCutoff=5 minBlockSize=28 maxBlockSize=136), multiDefault=MockVariableIntBlock(baseBlockSize=126), a_si=Pulsing41(freqCutoff=5 minBlockSize=28 maxBlockSize=136)}, docValues:{timestamp=DocValuesFormat(name=Asserting)}, sim=RandomSimilarityProvider(queryNorm=false,coord=crazy): {}, locale=mt, timezone=Africa/Abidjan
[junit4:junit4] 2> NOTE: Linux 3.2.0-38-generic amd64/Sun Microsystems Inc. 1.6.0_41 (64-bit)/cpus=8,threads=1,free=46938680,total=169541632
[junit4:junit4] 2> NOTE: All tests run in this JVM: [SystemInfoHandlerTest, TestWriterPerf, TestFieldTypeCollectionResource, LukeRequestHandlerTest, LegacyHTMLStripCharFilterTest, UnloadDistributedZkTest, AlternateDirectoryTest, ZkCLITest, SynonymTokenizerTest, SliceStateUpdateTest, SOLR749Test, BadComponentTest, TestPropInjectDefaults, TestStressVersions, SolrCoreTest, SuggesterTest, RecoveryZkTest]
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=RecoveryZkTest -Dtests.seed=418D481E1F4D5F37 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=mt -Dtests.timezone=Africa/Abidjan -Dtests.file.encoding=UTF-8
[junit4:junit4] ERROR 0.00s J0 | RecoveryZkTest (suite) <<<
[junit4:junit4] > Throwable #1: com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=573, name=coreLoadExecutor-278-thread-1, state=RUNNABLE, group=TGRP-RecoveryZkTest], registration stack trace below.
[junit4:junit4] > at java.lang.Thread.getStackTrace(Thread.java:1479)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:150)
[junit4:junit4] > at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:520)
[junit4:junit4] > at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:986)
[junit4:junit4] > at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:878)
[junit4:junit4] > at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:870)
[junit4:junit4] > at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:33)
[junit4:junit4] > at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:266)
[junit4:junit4] > at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:222)
[junit4:junit4] > at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:244)
[junit4:junit4] > at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:458)
[junit4:junit4] > at org.apache.solr.core.SolrCore.<init>(SolrCore.java:733)
[junit4:junit4] > at org.apache.solr.core.SolrCore.<init>(SolrCore.java:619)
[junit4:junit4] > at org.apache.solr.core.CoreContainer.createFromZk(CoreContainer.java:874)
[junit4:junit4] > at org.apache.solr.core.CoreContainer.create(CoreContainer.java:957)
[junit4:junit4] > at org.apache.solr.core.CoreContainer$2.call(CoreContainer.java:580)
[junit4:junit4] > at org.apache.solr.core.CoreContainer$2.call(CoreContainer.java:575)
[junit4:junit4] > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
[junit4:junit4] > at java.util.concurrent.FutureTask.run(FutureTask.java:138)
[junit4:junit4] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
[junit4:junit4] > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
[junit4:junit4] > at java.util.concurrent.FutureTask.run(FutureTask.java:138)
[junit4:junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
[junit4:junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:662)
[junit4:junit4] > Caused by: java.lang.AssertionError: Directory not closed: MockDirWrapper(org.apache.lucene.store.RAMDirectory@559be2c8 lockFactory=org.apache.lucene.store.NativeFSLockFactory@79492ac)
[junit4:junit4] > at org.junit.Assert.fail(Assert.java:93)
[junit4:junit4] > at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:602)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:599)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:167)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(RandomizedRunner.java:615)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
[junit4:junit4] > ... 1 more
[junit4:junit4] > Throwable #2: com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=575, name=RecoveryThread, state=RUNNABLE, group=TGRP-RecoveryZkTest], registration stack trace below.
[junit4:junit4] > at java.lang.Thread.getStackTrace(Thread.java:1479)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:150)
[junit4:junit4] > at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:520)
[junit4:junit4] > at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:986)
[junit4:junit4] > at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:878)
[junit4:junit4] > at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:870)
[junit4:junit4] > at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:33)
[junit4:junit4] > at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:266)
[junit4:junit4] > at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:222)
[junit4:junit4] > at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:383)
[junit4:junit4] > at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:281)
[junit4:junit4] > at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:153)
[junit4:junit4] > at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:409)
[junit4:junit4] > at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:223)
[junit4:junit4] > Caused by: java.lang.AssertionError: Directory not closed: MockDirWrapper(org.apache.lucene.store.RAMDirectory@661cd479 lockFactory=org.apache.lucene.store.NativeFSLockFactory@63dbf1b0)
[junit4:junit4] > at org.junit.Assert.fail(Assert.java:93)
[junit4:junit4] > at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:602)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:599)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:167)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(RandomizedRunner.java:615)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:662)
[junit4:junit4] Completed on J0 in 35.33s, 1 test, 2 errors <<< FAILURES!
[...truncated 698 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:381: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:361: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:183: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:449: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1213: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:877: There were test failures: 269 suites, 1137 tests, 2 suite-level errors, 13 ignored (7 assumptions)
Total time: 38 minutes 53 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Description set: Java: 64bit/jdk1.6.0_41 -XX:+UseParallelGC
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.8.0-ea-b79) - Build #
4640 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/4640/
Java: 32bit/jdk1.8.0-ea-b79 -server -XX:+UseSerialGC
2 tests failed.
REGRESSION: org.apache.solr.cloud.UnloadDistributedZkTest.testDistribSearch
Error Message:
Still found shard
Stack Trace:
java.lang.AssertionError: Still found shard
at __randomizedtesting.SeedInfo.seed([24CF929680B023C9:A5291C8EF7EF43F5]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.UnloadDistributedZkTest.testUnloadShardAndCollection(UnloadDistributedZkTest.java:124)
at org.apache.solr.cloud.UnloadDistributedZkTest.doTest(UnloadDistributedZkTest.java:77)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:806)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:487)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:722)
FAILED: org.apache.solr.cloud.BasicDistributedZk2Test.testDistribSearch
Error Message:
Server at http://127.0.0.1:45621/c/b/onenodecollectioncore returned non ok status:404, message:Can not find: /c/b/onenodecollectioncore/update
Stack Trace:
org.apache.solr.common.SolrException: Server at http://127.0.0.1:45621/c/b/onenodecollectioncore returned non ok status:404, message:Can not find: /c/b/onenodecollectioncore/update
at __randomizedtesting.SeedInfo.seed([24CF929680B023C9:A5291C8EF7EF43F5]:0)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:375)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:181)
at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:117)
at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:116)
at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:102)
at org.apache.solr.cloud.BasicDistributedZk2Test.testNodeWithoutCollectionForwarding(BasicDistributedZk2Test.java:200)
at org.apache.solr.cloud.BasicDistributedZk2Test.doTest(BasicDistributedZk2Test.java:89)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:806)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:487)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:722)
Build Log:
[...truncated 9013 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.BasicDistributedZk2Test
[junit4:junit4] 2> 0 T689 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /c/b
[junit4:junit4] 2> 3 T689 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1362854978515
[junit4:junit4] 2> 3 T689 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 4 T690 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 4 T690 oazs.ZooKeeperServer.setTickTime tickTime set to 1000
[junit4:junit4] 2> 5 T690 oazs.ZooKeeperServer.setMinSessionTimeout minSessionTimeout set to -1
[junit4:junit4] 2> 5 T690 oazs.ZooKeeperServer.setMaxSessionTimeout maxSessionTimeout set to -1
[junit4:junit4] 2> 5 T690 oazs.NIOServerCnxnFactory.configure binding to port 0.0.0.0/0.0.0.0:0
[junit4:junit4] 2> 5 T690 oazsp.FileTxnSnapLog.save Snapshotting: 0x0 to /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-1362854978515/zookeeper/server1/data/version-2/snapshot.0
[junit4:junit4] 2> 104 T689 oasc.ZkTestServer.run start zk server on port:51553
[junit4:junit4] 2> 104 T689 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:51553 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@150dddb
[junit4:junit4] 2> 105 T689 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 105 T695 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:51553. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 106 T695 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:51553, initiating session
[junit4:junit4] 2> 106 T691 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:57007
[junit4:junit4] 2> 106 T691 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:57007
[junit4:junit4] 2> 106 T693 oazsp.FileTxnLog.append Creating new log file: log.1
[junit4:junit4] 2> 108 T693 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d507ca3d60000 with negotiated timeout 10000 for client /127.0.0.1:57007
[junit4:junit4] 2> 108 T695 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:51553, sessionid = 0x13d507ca3d60000, negotiated timeout = 10000
[junit4:junit4] 2> 108 T696 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@150dddb name:ZooKeeperConnection Watcher:127.0.0.1:51553 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 109 T689 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 109 T689 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 111 T694 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d507ca3d60000
[junit4:junit4] 2> 112 T696 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 112 T689 oaz.ZooKeeper.close Session: 0x13d507ca3d60000 closed
[junit4:junit4] 2> 112 T691 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:57007 which had sessionid 0x13d507ca3d60000
[junit4:junit4] 2> 112 T689 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:51553/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@19724c1
[junit4:junit4] 2> 113 T697 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:51553. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 113 T689 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 114 T697 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:51553, initiating session
[junit4:junit4] 2> 114 T691 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:57008
[junit4:junit4] 2> 114 T691 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:57008
[junit4:junit4] 2> 114 T693 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d507ca3d60001 with negotiated timeout 10000 for client /127.0.0.1:57008
[junit4:junit4] 2> 114 T697 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:51553, sessionid = 0x13d507ca3d60001, negotiated timeout = 10000
[junit4:junit4] 2> 115 T698 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@19724c1 name:ZooKeeperConnection Watcher:127.0.0.1:51553/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 115 T689 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 115 T689 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 118 T689 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 120 T689 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 121 T689 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 123 T689 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 124 T689 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 127 T689 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 127 T689 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 130 T689 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 130 T689 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 132 T689 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 132 T689 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 134 T689 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 135 T689 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 137 T689 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 137 T689 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 139 T689 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 140 T689 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 143 T689 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 144 T689 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 148 T689 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 148 T689 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 150 T694 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d507ca3d60001
[junit4:junit4] 2> 150 T698 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 150 T691 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:57008 which had sessionid 0x13d507ca3d60001
[junit4:junit4] 2> 150 T689 oaz.ZooKeeper.close Session: 0x13d507ca3d60001 closed
[junit4:junit4] 2> 230 T689 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4] 2> 237 T689 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:47673
[junit4:junit4] 2> 238 T689 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 238 T689 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 238 T689 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1362854978663
[junit4:junit4] 2> 239 T689 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1362854978663/solr.xml
[junit4:junit4] 2> 239 T689 oasc.CoreContainer.<init> New CoreContainer 33416334
[junit4:junit4] 2> 239 T689 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1362854978663/'
[junit4:junit4] 2> 240 T689 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1362854978663/'
[junit4:junit4] 2> 256 T689 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 257 T689 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 257 T689 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 257 T689 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 258 T689 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 258 T689 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 258 T689 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 259 T689 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 259 T689 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 259 T689 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 265 T689 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4] 2> 283 T689 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:51553/solr
[junit4:junit4] 2> 283 T689 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 284 T689 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:51553 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@1746e27
[junit4:junit4] 2> 285 T689 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 285 T708 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:51553. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 286 T708 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:51553, initiating session
[junit4:junit4] 2> 286 T691 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:57009
[junit4:junit4] 2> 286 T691 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:57009
[junit4:junit4] 2> 287 T693 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d507ca3d60002 with negotiated timeout 20000 for client /127.0.0.1:57009
[junit4:junit4] 2> 287 T708 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:51553, sessionid = 0x13d507ca3d60002, negotiated timeout = 20000
[junit4:junit4] 2> 287 T709 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1746e27 name:ZooKeeperConnection Watcher:127.0.0.1:51553 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 287 T689 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 289 T694 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d507ca3d60002
[junit4:junit4] 2> 289 T709 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 289 T691 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:57009 which had sessionid 0x13d507ca3d60002
[junit4:junit4] 2> 289 T689 oaz.ZooKeeper.close Session: 0x13d507ca3d60002 closed
[junit4:junit4] 2> 290 T689 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 294 T689 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:51553/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@d22850
[junit4:junit4] 2> 295 T710 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:51553. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 295 T689 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 295 T710 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:51553, initiating session
[junit4:junit4] 2> 295 T691 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:57010
[junit4:junit4] 2> 296 T691 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:57010
[junit4:junit4] 2> 296 T693 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d507ca3d60003 with negotiated timeout 20000 for client /127.0.0.1:57010
[junit4:junit4] 2> 297 T710 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:51553, sessionid = 0x13d507ca3d60003, negotiated timeout = 20000
[junit4:junit4] 2> 297 T711 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d22850 name:ZooKeeperConnection Watcher:127.0.0.1:51553/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 297 T689 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 299 T694 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d507ca3d60003 type:create cxid:0x2 zxid:0x1a txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 300 T689 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 302 T689 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:47673_c%2Fb
[junit4:junit4] 2> 303 T694 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d507ca3d60003 type:delete cxid:0x8 zxid:0x1c txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:47673_c%2Fb Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:47673_c%2Fb
[junit4:junit4] 2> 304 T689 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:47673_c%2Fb
[junit4:junit4] 2> 307 T689 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 313 T694 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d507ca3d60003 type:delete cxid:0x16 zxid:0x21 txntype:-1 reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4] 2> 313 T689 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 316 T689 oasc.Overseer.start Overseer (id=89316063872155651-127.0.0.1:47673_c%2Fb-n_0000000000) starting
[junit4:junit4] 2> 317 T694 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d507ca3d60003 type:create cxid:0x1b zxid:0x23 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 318 T694 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d507ca3d60003 type:create cxid:0x1c zxid:0x24 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 319 T694 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d507ca3d60003 type:create cxid:0x1d zxid:0x25 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 320 T694 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d507ca3d60003 type:create cxid:0x1e zxid:0x26 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 320 T713 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 321 T689 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 323 T689 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 324 T689 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 326 T712 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 328 T714 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1362854978663/collection1
[junit4:junit4] 2> 328 T714 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 328 T714 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 329 T714 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 329 T714 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1362854978663/collection1/'
[junit4:junit4] 2> 330 T714 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1362854978663/collection1/lib/README' to classloader
[junit4:junit4] 2> 330 T714 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1362854978663/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 351 T714 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 385 T714 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 386 T714 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4] 2> 391 T714 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4] 2> 711 T714 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 716 T714 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 725 T714 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 738 T714 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 741 T714 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 743 T714 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 744 T714 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 745 T714 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 745 T714 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 746 T714 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 746 T714 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 747 T714 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1362854978663/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1362854978515/control/data/
[junit4:junit4] 2> 747 T714 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1374946
[junit4:junit4] 2> 747 T714 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 748 T714 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-1362854978515/control/data forceNew: false
[junit4:junit4] 2> 748 T714 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1362854978515/control/data/index/
[junit4:junit4] 2> 748 T714 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1362854978515/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 748 T714 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-1362854978515/control/data/index forceNew: false
[junit4:junit4] 2> 749 T714 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@16758f0 lockFactory=org.apache.lucene.store.NativeFSLockFactory@f94b08),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> 750 T714 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4] 2> 751 T714 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 751 T714 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 751 T714 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 752 T714 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 752 T714 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 752 T714 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 752 T714 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 753 T714 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 753 T714 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 755 T714 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 757 T714 oass.SolrIndexSearcher.<init> Opening Searcher@1ea7b1e main
[junit4:junit4] 2> 757 T714 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1362854978515/control/data/tlog
[junit4:junit4] 2> 758 T714 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 758 T714 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 761 T715 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1ea7b1e main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 762 T714 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 762 T714 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 763 T694 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d507ca3d60003 type:create cxid:0x55 zxid:0x2a txntype:-1 reqpath:n/a Error Path:/solr/overseer/queue Error:KeeperErrorCode = NoNode for /solr/overseer/queue
[junit4:junit4] 2> 1829 T712 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1830 T712 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:47673/c/b",
[junit4:junit4] 2> "node_name":"127.0.0.1:47673_c%2Fb",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "roles":null}
[junit4:junit4] 2> 1830 T712 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 1830 T712 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4] 2> 1830 T694 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d507ca3d60003 type:create cxid:0x5d zxid:0x2d txntype:-1 reqpath:n/a Error Path:/solr/overseer/queue-work Error:KeeperErrorCode = NoNode for /solr/overseer/queue-work
[junit4:junit4] 2> 1835 T711 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 2765 T714 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 2766 T714 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:47673/c/b collection:control_collection shard:shard1
[junit4:junit4] 2> 2767 T714 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 2775 T694 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d507ca3d60003 type:delete cxid:0x77 zxid:0x37 txntype:-1 reqpath:n/a Error Path:/solr/collections/control_collection/leaders Error:KeeperErrorCode = NoNode for /solr/collections/control_collection/leaders
[junit4:junit4] 2> 2776 T714 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 2776 T694 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d507ca3d60003 type:create cxid:0x78 zxid:0x38 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 2779 T714 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 2779 T714 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 2779 T714 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:47673/c/b/collection1/
[junit4:junit4] 2> 2779 T714 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 2780 T714 oasc.SyncStrategy.syncToMe http://127.0.0.1:47673/c/b/collection1/ has no replicas
[junit4:junit4] 2> 2780 T714 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:47673/c/b/collection1/
[junit4:junit4] 2> 2780 T714 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 2785 T694 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d507ca3d60003 type:create cxid:0x82 zxid:0x3c txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 3339 T712 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 3345 T711 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 3389 T714 oasc.ZkController.register We are http://127.0.0.1:47673/c/b/collection1/ and leader is http://127.0.0.1:47673/c/b/collection1/
[junit4:junit4] 2> 3389 T714 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:47673/c/b
[junit4:junit4] 2> 3389 T714 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 3390 T714 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 3390 T714 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 3392 T714 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 3393 T689 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 3393 T689 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 3394 T689 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 3398 T689 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 3399 T689 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:51553/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@1701dbb
[junit4:junit4] 2> 3399 T689 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 3399 T717 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:51553. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 3400 T717 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:51553, initiating session
[junit4:junit4] 2> 3400 T691 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:57012
[junit4:junit4] 2> 3400 T691 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:57012
[junit4:junit4] 2> 3401 T693 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d507ca3d60004 with negotiated timeout 10000 for client /127.0.0.1:57012
[junit4:junit4] 2> 3401 T717 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:51553, sessionid = 0x13d507ca3d60004, negotiated timeout = 10000
[junit4:junit4] 2> 3401 T718 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1701dbb name:ZooKeeperConnection Watcher:127.0.0.1:51553/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 3402 T689 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 3403 T689 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 3405 T689 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 3473 T689 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4] 2> 3475 T689 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:49575
[junit4:junit4] 2> 3476 T689 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 3477 T689 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 3477 T689 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1362854981917
[junit4:junit4] 2> 3477 T689 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1362854981917/solr.xml
[junit4:junit4] 2> 3478 T689 oasc.CoreContainer.<init> New CoreContainer 22673042
[junit4:junit4] 2> 3478 T689 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1362854981917/'
[junit4:junit4] 2> 3478 T689 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1362854981917/'
[junit4:junit4] 2> 3496 T689 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 3496 T689 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 3496 T689 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 3497 T689 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 3497 T689 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 3497 T689 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 3498 T689 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 3498 T689 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 3498 T689 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 3499 T689 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 3505 T689 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4] 2> 3514 T689 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:51553/solr
[junit4:junit4] 2> 3514 T689 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 3514 T689 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:51553 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@1d52be8
[junit4:junit4] 2> 3515 T689 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 3515 T728 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:51553. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 3516 T728 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:51553, initiating session
[junit4:junit4] 2> 3516 T691 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:57013
[junit4:junit4] 2> 3516 T691 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:57013
[junit4:junit4] 2> 3517 T693 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d507ca3d60005 with negotiated timeout 20000 for client /127.0.0.1:57013
[junit4:junit4] 2> 3517 T728 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:51553, sessionid = 0x13d507ca3d60005, negotiated timeout = 20000
[junit4:junit4] 2> 3518 T729 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d52be8 name:ZooKeeperConnection Watcher:127.0.0.1:51553 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 3518 T689 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 3519 T694 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d507ca3d60005
[junit4:junit4] 2> 3519 T729 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 3519 T691 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:57013 which had sessionid 0x13d507ca3d60005
[junit4:junit4] 2> 3519 T689 oaz.ZooKeeper.close Session: 0x13d507ca3d60005 closed
[junit4:junit4] 2> 3520 T689 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 3523 T689 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:51553/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@19d5009
[junit4:junit4] 2> 3523 T689 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 3524 T730 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:51553. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 3524 T730 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:51553, initiating session
[junit4:junit4] 2> 3524 T691 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:57014
[junit4:junit4] 2> 3524 T691 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:57014
[junit4:junit4] 2> 3525 T693 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d507ca3d60006 with negotiated timeout 20000 for client /127.0.0.1:57014
[junit4:junit4] 2> 3525 T730 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:51553, sessionid = 0x13d507ca3d60006, negotiated timeout = 20000
[junit4:junit4] 2> 3525 T731 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@19d5009 name:ZooKeeperConnection Watcher:127.0.0.1:51553/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 3526 T689 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 3527 T694 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d507ca3d60006 type:create cxid:0x1 zxid:0x4a txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 3527 T694 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d507ca3d60006 type:create cxid:0x2 zxid:0x4b txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 3529 T689 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 4531 T689 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49575_c%2Fb
[junit4:junit4] 2> 4532 T694 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d507ca3d60006 type:delete cxid:0xb zxid:0x4c txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:49575_c%2Fb Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:49575_c%2Fb
[junit4:junit4] 2> 4533 T689 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49575_c%2Fb
[junit4:junit4] 2> 4537 T711 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 4538 T718 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 4538 T731 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 4540 T711 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 4546 T732 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1362854981917/collection1
[junit4:junit4] 2> 4547 T732 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 4548 T732 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 4548 T732 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 4550 T732 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1362854981917/collection1/'
[junit4:junit4] 2> 4551 T732 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1362854981917/collection1/lib/README' to classloader
[junit4:junit4] 2> 4551 T732 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1362854981917/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 4594 T732 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 4626 T732 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 4627 T732 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4] 2> 4631 T732 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4] 2> 4861 T712 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 4862 T712 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:47673/c/b",
[junit4:junit4] 2> "node_name":"127.0.0.1:47673_c%2Fb",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "roles":null}
[junit4:junit4] 2> 4866 T711 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 4866 T718 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 4866 T731 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 4985 T732 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 4990 T732 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 4992 T732 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 5003 T732 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 5006 T732 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 5009 T732 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 5010 T732 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 5010 T732 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 5010 T732 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 5011 T732 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 5011 T732 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 5012 T732 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1362854981917/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1362854978515/jetty1/
[junit4:junit4] 2> 5012 T732 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1374946
[junit4:junit4] 2> 5012 T732 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 5013 T732 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-1362854978515/jetty1 forceNew: false
[junit4:junit4] 2> 5013 T732 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1362854978515/jetty1/index/
[junit4:junit4] 2> 5013 T732 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1362854978515/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 5014 T732 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-1362854978515/jetty1/index forceNew: false
[junit4:junit4] 2> 5015 T732 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@17969a8 lockFactory=org.apache.lucene.store.NativeFSLockFactory@eafaaa),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> 5015 T732 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4] 2> 5017 T732 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 5017 T732 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 5018 T732 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 5018 T732 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 5018 T732 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 5019 T732 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 5019 T732 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 5019 T732 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 5020 T732 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 5022 T732 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 5024 T732 oass.SolrIndexSearcher.<init> Opening Searcher@16f4184 main
[junit4:junit4] 2> 5024 T732 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1362854978515/jetty1/tlog
[junit4:junit4] 2> 5024 T732 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 5024 T732 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 5028 T733 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@16f4184 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 5029 T732 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 5030 T732 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 6371 T712 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 6372 T712 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:49575/c/b",
[junit4:junit4] 2> "node_name":"127.0.0.1:49575_c%2Fb",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "roles":null}
[junit4:junit4] 2> 6372 T712 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 2
[junit4:junit4] 2> 6372 T712 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4:junit4] 2> 6376 T731 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 6376 T718 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 6376 T711 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 7032 T732 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 7032 T732 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:49575/c/b collection:collection1 shard:shard2
[junit4:junit4] 2> 7033 T732 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4:junit4] 2> 7038 T694 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d507ca3d60006 type:delete cxid:0x48 zxid:0x5c txntype:-1 reqpath:n/a Error Path:/solr/collections/collection1/leaders Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders
[junit4:junit4] 2> 7039 T732 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 7039 T694 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d507ca3d60006 type:create cxid:0x49 zxid:0x5d txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 7041 T732 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 7041 T732 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 7041 T732 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:49575/c/b/collection1/
[junit4:junit4] 2> 7042 T732 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 7042 T732 oasc.SyncStrategy.syncToMe http://127.0.0.1:49575/c/b/collection1/ has no replicas
[junit4:junit4] 2> 7042 T732 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:49575/c/b/collection1/
[junit4:junit4] 2> 7042 T732 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4] 2> 7046 T694 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d507ca3d60006 type:create cxid:0x53 zxid:0x61 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 7879 T712 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 7885 T711 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 7885 T718 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 7885 T731 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 7899 T732 oasc.ZkController.register We are http://127.0.0.1:49575/c/b/collection1/ and leader is http://127.0.0.1:49575/c/b/collection1/
[junit4:junit4] 2> 7899 T732 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:49575/c/b
[junit4:junit4] 2> 7899 T732 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 7899 T732 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 7899 T732 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 7901 T732 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 7901 T689 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 7902 T689 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 7902 T689 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 7976 T689 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4] 2> 7978 T689 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:45621
[junit4:junit4] 2> 7978 T689 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 7978 T689 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 7979 T689 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1362854986418
[junit4:junit4] 2> 7979 T689 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1362854986418/solr.xml
[junit4:junit4] 2> 7979 T689 oasc.CoreContainer.<init> New CoreContainer 9207106
[junit4:junit4] 2> 7980 T689 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1362854986418/'
[junit4:junit4] 2> 7980 T689 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1362854986418/'
[junit4:junit4] 2> 8004 T689 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 8004 T689 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 8005 T689 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 8005 T689 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 8005 T689 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 8005 T689 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 8006 T689 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 8006 T689 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 8006 T689 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 8007 T689 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 8012 T689 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4] 2> 8021 T689 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:51553/solr
[junit4:junit4] 2> 8021 T689 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 8021 T689 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:51553 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@9554f0
[junit4:junit4] 2> 8022 T689 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 8022 T744 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:51553. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 8023 T744 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:51553, initiating session
[junit4:junit4] 2> 8023 T691 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:57026
[junit4:junit4] 2> 8023 T691 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:57026
[junit4:junit4] 2> 8024 T693 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d507ca3d60007 with negotiated timeout 20000 for client /127.0.0.1:57026
[junit4:junit4] 2> 8024 T744 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:51553, sessionid = 0x13d507ca3d60007, negotiated timeout = 20000
[junit4:junit4] 2> 8024 T745 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@9554f0 name:ZooKeeperConnection Watcher:127.0.0.1:51553 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 8025 T689 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 8026 T694 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d507ca3d60007
[junit4:junit4] 2> 8026 T745 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 8026 T691 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:57026 which had sessionid 0x13d507ca3d60007
[junit4:junit4] 2> 8026 T689 oaz.ZooKeeper.close Session: 0x13d507ca3d60007 closed
[junit4:junit4] 2> 8027 T689 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 8030 T689 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:51553/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@857e91
[junit4:junit4] 2> 8030 T689 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 8031 T746 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:51553. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 8031 T746 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:51553, initiating session
[junit4:junit4] 2> 8031 T691 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:57027
[junit4:junit4] 2> 8031 T691 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:57027
[junit4:junit4] 2> 8032 T693 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d507ca3d60008 with negotiated timeout 20000 for client /127.0.0.1:57027
[junit4:junit4] 2> 8032 T746 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:51553, sessionid = 0x13d507ca3d60008, negotiated timeout = 20000
[junit4:junit4] 2> 8032 T747 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@857e91 name:ZooKeeperConnection Watcher:127.0.0.1:51553/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 8033 T689 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 8034 T694 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d507ca3d60008 type:create cxid:0x1 zxid:0x6e txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 8034 T694 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d507ca3d60008 type:create cxid:0x2 zxid:0x6f txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 8036 T689 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 9039 T689 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45621_c%2Fb
[junit4:junit4] 2> 9039 T694 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d507ca3d60008 type:delete cxid:0xb zxid:0x70 txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:45621_c%2Fb Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:45621_c%2Fb
[junit4:junit4] 2> 9040 T689 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:45621_c%2Fb
[junit4:junit4] 2> 9042 T718 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 9043 T711 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 9043 T731 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 9043 T747 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 9043 T731 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 9044 T718 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 9044 T711 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 9047 T748 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1362854986418/collection1
[junit4:junit4] 2> 9048 T748 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 9048 T748 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 9048 T748 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 9049 T748 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1362854986418/collection1/'
[junit4:junit4] 2> 9050 T748 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1362854986418/collection1/lib/README' to classloader
[junit4:junit4] 2> 9050 T748 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1362854986418/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 9071 T748 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 9104 T748 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 9105 T748 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4] 2> 9109 T748 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4] 2> 9389 T712 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 9390 T712 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:49575/c/b",
[junit4:junit4] 2> "node_name":"127.0.0.1:49575_c%2Fb",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "roles":null}
[junit4:junit4] 2> 9392 T718 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 9393 T747 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 9393 T711 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 9392 T731 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 9436 T748 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 9442 T748 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 9444 T748 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 9460 T748 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 9464 T748 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 9469 T748 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 9470 T748 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 9470 T748 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 9470 T748 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 9471 T748 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 9471 T748 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 9471 T748 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1362854986418/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1362854978515/jetty2/
[junit4:junit4] 2> 9471 T748 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1374946
[junit4:junit4] 2> 9472 T748 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 9472 T748 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-1362854978515/jetty2 forceNew: false
[junit4:junit4] 2> 9472 T748 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1362854978515/jetty2/index/
[junit4:junit4] 2> 9473 T748 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1362854978515/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 9473 T748 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-1362854978515/jetty2/index forceNew: false
[junit4:junit4] 2> 9474 T748 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@14a3dcb lockFactory=org.apache.lucene.store.NativeFSLockFactory@dcc7b3),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> 9474 T748 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4] 2> 9476 T748 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 9476 T748 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 9476 T748 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 9477 T748 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 9477 T748 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 9477 T748 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 9477 T748 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 9478 T748 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 9478 T748 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 9480 T748 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 9482 T748 oass.SolrIndexSearcher.<init> Opening Searcher@1e49110 main
[junit4:junit4] 2> 9483 T748 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1362854978515/jetty2/tlog
[junit4:junit4] 2> 9483 T748 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 9483 T748 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 9487 T749 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1e49110 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 9489 T748 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 9489 T748 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 10897 T712 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 10898 T712 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:45621/c/b",
[junit4:junit4] 2> "node_name":"127.0.0.1:45621_c%2Fb",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "roles":null}
[junit4:junit4] 2> 10898 T712 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4] 2> 10898 T712 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4] 2> 10902 T711 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 10902 T747 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 10902 T718 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 10902 T731 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 11490 T748 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 11491 T748 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:45621/c/b collection:collection1 shard:shard1
[junit4:junit4] 2> 11492 T748 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 11498 T694 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d507ca3d60008 type:delete cxid:0x47 zxid:0x7f txntype:-1 reqpath:n/a Error Path:/solr/collections/collection1/leaders/shard1 Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders/shard1
[junit4:junit4] 2> 11498 T748 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 11499 T694 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d507ca3d60008 type:create cxid:0x48 zxid:0x80 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 11500 T748 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 11501 T748 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 11501 T748 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:45621/c/b/collection1/
[junit4:junit4] 2> 11501 T748 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 11501 T748 oasc.SyncStrategy.syncToMe http://127.0.0.1:45621/c/b/collection1/ has no replicas
[junit4:junit4] 2> 11501 T748 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:45621/c/b/collection1/
[junit4:junit4] 2> 11502 T748 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 11505 T694 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d507ca3d60008 type:create cxid:0x51 zxid:0x83 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 12406 T712 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 12413 T731 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 12413 T747 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 12413 T718 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 12413 T711 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 12458 T748 oasc.ZkController.register We are http://127.0.0.1:45621/c/b/collection1/ and leader is http://127.0.0.1:45621/c/b/collection1/
[junit4:junit4] 2> 12459 T748 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:45621/c/b
[junit4:junit4] 2> 12459 T748 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 12459 T748 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 12459 T748 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 12461 T748 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 12462 T689 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 12462 T689 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 12463 T689 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 12525 T689 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4] 2> 12527 T689 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:50207
[junit4:junit4] 2> 12527 T689 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 12528 T689 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 12528 T689 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1362854990979
[junit4:junit4] 2> 12528 T689 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1362854990979/solr.xml
[junit4:junit4] 2> 12528 T689 oasc.CoreContainer.<init> New CoreContainer 22768413
[junit4:junit4] 2> 12529 T689 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1362854990979/'
[junit4:junit4] 2> 12529 T689 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1362854990979/'
[junit4:junit4] 2> 12546 T689 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 12546 T689 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 12546 T689 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 12547 T689 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 12547 T689 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 12547 T689 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 12548 T689 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 12548 T689 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 12548 T689 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 12548 T689 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 12554 T689 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4] 2> 12563 T689 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:51553/solr
[junit4:junit4] 2> 12563 T689 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 12564 T689 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:51553 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@cf8acd
[junit4:junit4] 2> 12564 T689 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 12564 T760 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:51553. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 12565 T760 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:51553, initiating session
[junit4:junit4] 2> 12565 T691 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:57034
[junit4:junit4] 2> 12566 T691 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:57034
[junit4:junit4] 2> 12566 T693 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d507ca3d60009 with negotiated timeout 20000 for client /127.0.0.1:57034
[junit4:junit4] 2> 12566 T760 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:51553, sessionid = 0x13d507ca3d60009, negotiated timeout = 20000
[junit4:junit4] 2> 12567 T761 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@cf8acd name:ZooKeeperConnection Watcher:127.0.0.1:51553 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 12567 T689 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 12568 T694 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d507ca3d60009
[junit4:junit4] 2> 12569 T761 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 12569 T691 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:57034 which had sessionid 0x13d507ca3d60009
[junit4:junit4] 2> 12568 T689 oaz.ZooKeeper.close Session: 0x13d507ca3d60009 closed
[junit4:junit4] 2> 12569 T689 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 12573 T689 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:51553/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@1696b6b
[junit4:junit4] 2> 12574 T689 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 12574 T762 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:51553. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 12574 T762 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:51553, initiating session
[junit4:junit4] 2> 12574 T691 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:57035
[junit4:junit4] 2> 12575 T691 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:57035
[junit4:junit4] 2> 12575 T693 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d507ca3d6000a with negotiated timeout 20000 for client /127.0.0.1:57035
[junit4:junit4] 2> 12575 T762 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:51553, sessionid = 0x13d507ca3d6000a, negotiated timeout = 20000
[junit4:junit4] 2> 12576 T763 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1696b6b name:ZooKeeperConnection Watcher:127.0.0.1:51553/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 12576 T689 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 12577 T694 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d507ca3d6000a type:create cxid:0x1 zxid:0x90 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 12579 T694 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d507ca3d6000a type:create cxid:0x2 zxid:0x91 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 12581 T689 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 13583 T689 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50207_c%2Fb
[junit4:junit4] 2> 13584 T694 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d507ca3d6000a type:delete cxid:0xb zxid:0x92 txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:50207_c%2Fb Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:50207_c%2Fb
[junit4:junit4] 2> 13585 T689 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50207_c%2Fb
[junit4:junit4] 2> 13587 T747 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 13587 T711 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 13588 T731 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 13588 T718 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 13589 T763 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 13589 T718 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 13589 T731 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 13590 T747 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 13590 T711 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 13593 T764 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1362854990979/collection1
[junit4:junit4] 2> 13594 T764 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 13594 T764 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 13594 T764 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 13596 T764 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1362854990979/collection1/'
[junit4:junit4] 2> 13596 T764 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1362854990979/collection1/lib/README' to classloader
[junit4:junit4] 2> 13596 T764 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1362854990979/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 13626 T764 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 13652 T764 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 13653 T764 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4] 2> 13657 T764 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4] 2> 13918 T712 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 13918 T712 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:45621/c/b",
[junit4:junit4] 2> "node_name":"127.0.0.1:45621_c%2Fb",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "roles":null}
[junit4:junit4] 2> 13922 T731 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 13922 T711 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 13922 T763 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 13922 T747 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 13922 T718 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 14022 T764 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 14027 T764 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 14030 T764 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 14041 T764 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 14044 T764 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 14047 T764 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 14048 T764 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 14048 T764 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 14048 T764 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 14049 T764 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 14050 T764 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 14050 T764 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1362854990979/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1362854978515/jetty3/
[junit4:junit4] 2> 14050 T764 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1374946
[junit4:junit4] 2> 14050 T764 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 14051 T764 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-1362854978515/jetty3 forceNew: false
[junit4:junit4] 2> 14051 T764 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1362854978515/jetty3/index/
[junit4:junit4] 2> 14051 T764 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1362854978515/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 14051 T764 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-1362854978515/jetty3/index forceNew: false
[junit4:junit4] 2> 14053 T764 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1a8aa2d lockFactory=org.apache.lucene.store.NativeFSLockFactory@1fe691c),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> 14053 T764 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4] 2> 14054 T764 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 14054 T764 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 14055 T764 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 14055 T764 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 14056 T764 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 14056 T764 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 14056 T764 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 14056 T764 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 14057 T764 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 14059 T764 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 14061 T764 oass.SolrIndexSearcher.<init> Opening Searcher@870347 main
[junit4:junit4] 2> 14061 T764 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1362854978515/jetty3/tlog
[junit4:junit4] 2> 14062 T764 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 14062 T764 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 14065 T765 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@870347 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 14066 T764 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 14066 T764 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 15427 T712 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 15427 T712 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:50207/c/b",
[junit4:junit4] 2> "node_name":"127.0.0.1:50207_c%2Fb",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "roles":null}
[junit4:junit4] 2> 15427 T712 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4] 2> 15428 T712 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4:junit4] 2> 15431 T731 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 15431 T763 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 15431 T718 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 15431 T747 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 15431 T711 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 16068 T764 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 16068 T764 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:50207/c/b collection:collection1 shard:shard2
[junit4:junit4] 2> 16070 T764 oasc.ZkController.register We are http://127.0.0.1:50207/c/b/collection1/ and leader is http://127.0.0.1:49575/c/b/collection1/
[junit4:junit4] 2> 16070 T764 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:50207/c/b
[junit4:junit4] 2> 16070 T764 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 16070 T764 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C47 name=collection1 org.apache.solr.core.SolrCore@2d4265 url=http://127.0.0.1:50207/c/b/collection1 node=127.0.0.1:50207_c%2Fb C47_STATE=coll:collection1 core:collection1 props:{collection=collection1, base_url=http://127.0.0.1:50207/c/b, node_name=127.0.0.1:50207_c%2Fb, state=down, core=collection1}
[junit4:junit4] 2> 16071 T766 C47 P50207 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 16071 T766 C47 P50207 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 16071 T764 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 16072 T766 C47 P50207 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 16072 T766 C47 P50207 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 16072 T689 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 16072 T766 C47 P50207 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 16073 T689 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 16073 T689 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 16077 T689 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 16079 T689 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 16084 T726 oasc.CoreContainer.create Creating SolrCore 'onenodecollectioncore' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1362854981917/onenodecollectioncore
[junit4:junit4] 2> 16084 T726 oasc.ZkController.createCollectionZkNode Check for collection zkNode:onenodecollection
[junit4:junit4] 2> 16085 T726 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:onenodecollection
[junit4:junit4] 2> 16085 T726 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4] 2> 16086 T726 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
[junit4:junit4] 2> 16087 T726 oascc.SolrZkClient.makePath makePath: /collections/onenodecollection
[junit4:junit4] 2> 16088 T726 oasc.ZkController.readConfigName Load collection config from:/collections/onenodecollection
[junit4:junit4] 2> 16089 T726 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1362854981917/onenodecollectioncore/'
[junit4:junit4] 2> 16126 T726 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 16154 T726 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 16155 T726 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4] 2> 16159 T726 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4] 2> 16479 T726 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 16483 T726 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 16486 T726 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 16498 T726 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 16501 T726 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 16504 T726 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 16505 T726 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 16506 T726 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 16506 T726 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 16507 T726 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 16507 T726 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 16508 T726 oasc.SolrCore.<init> [onenodecollectioncore] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1362854981917/onenodecollectioncore/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1362854978515/onenodecollection/
[junit4:junit4] 2> 16508 T726 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1374946
[junit4:junit4] 2> 16508 T726 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 16509 T726 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1362854978515/onenodecollection forceNew: false
[junit4:junit4] 2> 16509 T726 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1362854978515/onenodecollection/index/
[junit4:junit4] 2> 16510 T726 oasc.SolrCore.initIndex WARNING [onenodecollectioncore] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1362854978515/onenodecollection/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 16510 T726 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1362854978515/onenodecollection/index forceNew: false
[junit4:junit4] 2> 16511 T726 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@b6d46c lockFactory=org.apache.lucene.store.NativeFSLockFactory@1a19493),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> 16512 T726 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4] 2> 16513 T726 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 16514 T726 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 16514 T726 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 16514 T726 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 16515 T726 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 16515 T726 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 16515 T726 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 16516 T726 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 16516 T726 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 16519 T726 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 16521 T726 oass.SolrIndexSearcher.<init> Opening Searcher@2ced9f main
[junit4:junit4] 2> 16521 T726 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1362854978515/onenodecollection/tlog
[junit4:junit4] 2> 16522 T726 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 16522 T726 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 16525 T768 oasc.SolrCore.registerSearcher [onenodecollectioncore] Registered new searcher Searcher@2ced9f main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 16525 T726 oasc.ZkController.publish publishing core=onenodecollectioncore state=down
[junit4:junit4] 2> 16935 T712 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 16936 T712 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:50207/c/b",
[junit4:junit4] 2> "node_name":"127.0.0.1:50207_c%2Fb",
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "roles":null}
[junit4:junit4] 2> 16939 T712 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "collection":"onenodecollection",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:49575/c/b",
[junit4:junit4] 2> "node_name":"127.0.0.1:49575_c%2Fb",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "core":"onenodecollectioncore",
[junit4:junit4] 2> "roles":"none"}
[junit4:junit4] 2> 16940 T712 oasc.Overseer$ClusterStateUpdater.createCollection Create collection onenodecollection with numShards 1
[junit4:junit4] 2> 16940 T712 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4] 2> 16943 T711 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 16943 T718 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 16943 T731 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 16943 T763 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 16943 T747 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 17077 T725 oass.SolrDispatchFilter.handleAdminRequest /admin/cores {coreNodeName=127.0.0.1:50207_c%252Fb_collection1&state=recovering&action=PREPRECOVERY&nodeName=127.0.0.1:50207_c%252Fb&core=collection1&checkLive=true&onlyIfLeader=true&version=2&wt=javabin} status=0 QTime=1000
[junit4:junit4] 2> 17527 T726 oasc.CoreContainer.registerCore registering core: onenodecollectioncore
[junit4:junit4] 2> 17528 T726 oasc.ZkController.register Register replica - core:onenodecollectioncore address:http://127.0.0.1:49575/c/b collection:onenodecollection shard:shard1
[junit4:junit4] 2> 17529 T726 oascc.SolrZkClient.makePath makePath: /collections/onenodecollection/leader_elect/shard1/election
[junit4:junit4] 2> 17535 T694 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d507ca3d60006 type:delete cxid:0xa1 zxid:0xad txntype:-1 reqpath:n/a Error Path:/solr/collections/onenodecollection/leaders Error:KeeperErrorCode = NoNode for /solr/collections/onenodecollection/leaders
[junit4:junit4] 2> 17536 T726 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 17536 T694 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d507ca3d60006 type:create cxid:0xa2 zxid:0xae txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 17538 T726 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 17538 T726 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 17539 T726 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:49575/c/b/onenodecollectioncore/
[junit4:junit4] 2> 17539 T726 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 17539 T726 oasc.SyncStrategy.syncToMe http://127.0.0.1:49575/c/b/onenodecollectioncore/ has no replicas
[junit4:junit4] 2> 17540 T726 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:49575/c/b/onenodecollectioncore/
[junit4:junit4] 2> 17540 T726 oascc.SolrZkClient.makePath makePath: /collections/onenodecollection/leaders/shard1
[junit4:junit4] 2> 17544 T694 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d507ca3d60006 type:create cxid:0xac zxid:0xb2 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 18448 T712 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 18454 T731 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 18454 T718 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 18454 T763 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 18454 T747 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 18455 T711 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 18497 T726 oasc.ZkController.register We are http://127.0.0.1:49575/c/b/onenodecollectioncore/ and leader is http://127.0.0.1:49575/c/b/onenodecollectioncore/
[junit4:junit4] 2> 18497 T726 oasc.ZkController.register No LogReplay needed for core=onenodecollectioncore baseURL=http://127.0.0.1:49575/c/b
[junit4:junit4] 2> 18497 T726 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 18497 T726 oasc.ZkController.publish publishing core=onenodecollectioncore state=active
[junit4:junit4] 2> 18499 T726 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 18500 T726 oass.SolrDispatchFilter.handleAdminRequest /admin/cores {numShards=1&collection=onenodecollection&dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1362854978515/onenodecollection&action=CREATE&name=onenodecollectioncore&version=2&roles=none&wt=javabin} status=0 QTime=2416
[junit4:junit4] 2> 18500 T689 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: onenodecollection failOnTimeout:true timeout (sec):230
[junit4:junit4] 2> 18501 T689 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> C47_STATE=coll:collection1 core:collection1 props:{collection=collection1, shard=shard2, base_url=http://127.0.0.1:50207/c/b, node_name=127.0.0.1:50207_c%2Fb, state=recovering, core=collection1}
[junit4:junit4] 2> 19078 T766 C47 P50207 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:49575/c/b/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 19078 T766 C47 P50207 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:50207/c/b START replicas=[http://127.0.0.1:49575/c/b/collection1/] nUpdates=100
[junit4:junit4] 2> 19079 T766 C47 P50207 oasu.PeerSync.sync WARNING no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 19079 T766 C47 P50207 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 19079 T766 C47 P50207 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 19079 T766 C47 P50207 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 19079 T766 C47 P50207 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 19079 T766 C47 P50207 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:49575/c/b/collection1/. core=collection1
[junit4:junit4] 2> 19080 T766 C47 P50207 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C48 name=collection1 org.apache.solr.core.SolrCore@fb2072 url=http://127.0.0.1:49575/c/b/collection1 node=127.0.0.1:49575_c%2Fb C48_STATE=coll:collection1 core:collection1 props:{collection=collection1, shard=shard2, base_url=http://127.0.0.1:49575/c/b, node_name=127.0.0.1:49575_c%2Fb, state=active, core=collection1, leader=true}
[junit4:junit4] 2> 19093 T725 C48 P49575 REQ /get {distrib=false&getVersions=100&version=2&qt=/get&wt=javabin} status=0 QTime=1
[junit4:junit4] 2> 19094 T724 C48 P49575 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 19095 T724 C48 P49575 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@17969a8 lockFactory=org.apache.lucene.store.NativeFSLockFactory@eafaaa),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> 19095 T724 C48 P49575 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4] 2> 19096 T724 C48 P49575 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@17969a8 lockFactory=org.apache.lucene.store.NativeFSLockFactory@eafaaa),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@17969a8 lockFactory=org.apache.lucene.store.NativeFSLockFactory@eafaaa),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4] 2> 19096 T724 C48 P49575 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[segments_2]
[junit4:junit4] 2> 19097 T724 C48 P49575 oass.SolrIndexSearcher.<init> Opening Searcher@dd1274 realtime
[junit4:junit4] 2> 19097 T724 C48 P49575 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 19097 T724 C48 P49575 /update {waitSearcher=true&softCommit=false&commit_end_point=true&openSearcher=false&version=2&wt=javabin&commit=true} {commit=} 0 3
[junit4:junit4] 2> 19098 T766 C47 P50207 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 19098 T766 C47 P50207 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 19102 T726 C48 P49575 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 19103 T726 C48 P49575 REQ /replication {command=indexversion&version=2&qt=/replication&wt=javabin} status=0 QTime=4
[junit4:junit4] 2> 19104 T766 C47 P50207 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 19104 T766 C47 P50207 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 19104 T766 C47 P50207 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 19106 T725 C48 P49575 REQ /replication {command=filelist&generation=2&version=2&qt=/replication&wt=javabin} status=0 QTime=0
[junit4:junit4] 2> 19106 T766 C47 P50207 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 19107 T766 C47 P50207 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-1362854978515/jetty3/index.20130310064957618 forceNew: false
[junit4:junit4] 2> 19107 T766 C47 P50207 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@1c8b11e lockFactory=org.apache.lucene.store.NativeFSLockFactory@c862ac)) fullCopy=false
[junit4:junit4] 2> 19109 T724 C48 P49575 REQ /replication {command=filecontent&checksum=true&file=segments_2&generation=2&qt=/replication&wt=filestream} status=0 QTime=0
[junit4:junit4] 2> 19110 T766 C47 P50207 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 19111 T766 C47 P50207 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 19111 T766 C47 P50207 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 19112 T766 C47 P50207 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1a8aa2d lockFactory=org.apache.lucene.store.NativeFSLockFactory@1fe691c),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1a8aa2d lockFactory=org.apache.lucene.store.NativeFSLockFactory@1fe691c),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4] 2> 19112 T766 C47 P50207 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[segments_2]
[junit4:junit4] 2> 19112 T766 C47 P50207 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 19112 T766 C47 P50207 oass.SolrIndexSearcher.<init> Opening Searcher@1e1adcf main
[junit4:junit4] 2> 19113 T765 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1e1adcf main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 19113 T766 C47 P50207 oash.SnapPuller.fetchLatestIndex removing temporary index download directory files MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@1c8b11e lockFactory=org.apache.lucene.store.NativeFSLockFactory@c862ac))
[junit4:junit4] 2> 19113 T766 C47 P50207 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 19114 T766 C47 P50207 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 19114 T766 C47 P50207 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 19114 T766 C47 P50207 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 19115 T766 C47 P50207 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 19503 T689 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 19960 T712 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 19960 T712 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "collection":"onenodecollection",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:49575/c/b",
[junit4:junit4] 2> "node_name":"127.0.0.1:49575_c%2Fb",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "core":"onenodecollectioncore",
[junit4:junit4] 2> "roles":"none"}
[junit4:junit4] 2> 19963 T712 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:50207/c/b",
[junit4:junit4] 2> "node_name":"127.0.0.1:50207_c%2Fb",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "roles":null}
[junit4:junit4] 2> 19967 T711 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 19967 T747 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 19967 T763 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 19967 T718 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 19967 T731 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 20504 T689 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 20505 T689 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: onenodecollection
[junit4:junit4] 2> 20506 T689 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 20517 T742 oejs.Response.sendError WARNING Committed before 404 Can not find: /c/b/onenodecollectioncore/update
[junit4:junit4] 2> 20517 T689 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:51553 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@1a0b7e4
[junit4:junit4] 2> 20517 T742 oejs.ServletHandler.doHandle WARNING /c/b/onenodecollectioncore/update java.lang.IllegalStateException: Committed
[junit4:junit4] 2> at org.eclipse.jetty.server.Response.resetBuffer(Response.java:1136)
[junit4:junit4] 2> at org.eclipse.jetty.server.Response.sendError(Response.java:314)
[junit4:junit4] 2> at org.apache.solr.client.solrj.embedded.JettySolrRunner$Servlet404.service(JettySolrRunner.java:465)
[junit4:junit4] 2> at javax.servlet.http.HttpServlet.service(HttpServlet.java:770)
[junit4:junit4] 2> at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:669)
[junit4:junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1336)
[junit4:junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:379)
[junit4:junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:141)
[junit4:junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
[junit4:junit4] 2> at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:135)
[junit4:junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
[junit4:junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453)
[junit4:junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.GzipHandler.handle(GzipHandler.java:275)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1074)
[junit4:junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:382)
[junit4:junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1006)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
[junit4:junit4] 2> at org.eclipse.jetty.server.Server.handle(Server.java:365)
[junit4:junit4] 2> at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:485)
[junit4:junit4] 2> at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:937)
[junit4:junit4] 2> at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:998)
[junit4:junit4] 2> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:948)
[junit4:junit4] 2> at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
[junit4:junit4] 2> at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
[junit4:junit4] 2> at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:628)
[junit4:junit4] 2> at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
[junit4:junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4:junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:722)
[junit4:junit4] 2>
[junit4:junit4] 2> 20518 T771 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:51553. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 20518 T689 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 20518 T771 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:51553, initiating session
[junit4:junit4] 2> 20518 T691 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:57061
[junit4:junit4] 2> 20518 T691 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:57061
[junit4:junit4] 2> 20519 T693 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d507ca3d6000b with negotiated timeout 10000 for client /127.0.0.1:57061
[junit4:junit4] 2> 20519 T771 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:51553, sessionid = 0x13d507ca3d6000b, negotiated timeout = 10000
[junit4:junit4] 2> 20520 T772 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a0b7e4 name:ZooKeeperConnection Watcher:127.0.0.1:51553 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 20520 T689 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 20559 T694 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d507ca3d6000b
[junit4:junit4] 2> 20559 T772 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 20559 T691 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:57061 which had sessionid 0x13d507ca3d6000b
[junit4:junit4] 2> 20559 T689 oaz.ZooKeeper.close Session: 0x13d507ca3d6000b closed
[junit4:junit4] 2> 20560 T694 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d507ca3d60004
[junit4:junit4] 2> 20560 T718 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 20560 T689 oaz.ZooKeeper.close Session: 0x13d507ca3d60004 closed
[junit4:junit4] 2> 20560 T691 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:57012 which had sessionid 0x13d507ca3d60004
[junit4:junit4] 2> 20563 T689 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 47673
[junit4:junit4] 2> 20563 T689 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=33416334
[junit4:junit4] 2> 21472 T712 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 21473 T712 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:47673/c/b",
[junit4:junit4] 2> "node_name":"127.0.0.1:47673_c%2Fb",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:47673_c%2Fb_collection1",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "roles":null}
[junit4:junit4] 2> 21478 T747 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 21478 T711 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 21478 T731 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 21478 T763 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 22565 T689 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 22565 T689 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 22567 T689 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@85151b
[junit4:junit4] 2> 22570 T689 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 22570 T689 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 22571 T689 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 22571 T689 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 22571 T689 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 22572 T689 oasc.CachingDirectoryFactory.close Closing directory when closing factory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-1362854978515/control/data/index
[junit4:junit4] 2> 22572 T689 oasc.CachingDirectoryFactory.closeDirectory Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-1362854978515/control/data/index
[junit4:junit4] 2> 22572 T689 oasc.CachingDirectoryFactory.close Closing directory when closing factory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-1362854978515/control/data
[junit4:junit4] 2> 22573 T689 oasc.CachingDirectoryFactory.closeDirectory Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-1362854978515/control/data
[junit4:junit4] 2> 22573 T712 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=89316063872155651-127.0.0.1:47673_c%2Fb-n_0000000000) am no longer a leader.
[junit4:junit4] 2> 22574 T694 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d507ca3d60003
[junit4:junit4] 2> 22575 T711 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 22575 T763 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 22575 T747 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 22575 T711 oascc.ZkStateReader$2.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 22575 T689 oaz.ZooKeeper.close Session: 0x13d507ca3d60003 closed
[junit4:junit4] 2> 22575 T691 oazs.NIOServerCnxn.doIO WARNING caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x13d507ca3d60003, likely client has closed socket
[junit4:junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
[junit4:junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:722)
[junit4:junit4] 2>
[junit4:junit4] 2> 22575 T711 oascc.ZkStateReader$3.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 22576 T691 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:57010 which had sessionid 0x13d507ca3d60003
[junit4:junit4] 2> 22576 T711 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 22576 T731 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 22576 T731 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 22577 T747 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 22577 T763 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 22578 T694 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d507ca3d60006 type:delete cxid:0xbc zxid:0xce txntype:-1 reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4] 2> 22578 T731 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 22580 T731 oasc.Overseer.start Overseer (id=89316063872155654-127.0.0.1:49575_c%2Fb-n_0000000001) starting
[junit4:junit4] 2> 22581 T694 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d507ca3d60006 type:create cxid:0xc1 zxid:0xd0 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 22581 T694 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d507ca3d60006 type:create cxid:0xc2 zxid:0xd1 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 22582 T694 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d507ca3d60006 type:create cxid:0xc3 zxid:0xd2 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 22582 T694 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d507ca3d60006 type:create cxid:0xc4 zxid:0xd3 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 22583 T774 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 22584 T773 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 22585 T773 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 22586 T773 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:47673/c/b",
[junit4:junit4] 2> "node_name":"127.0.0.1:47673_c%2Fb",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "roles":null}
[junit4:junit4] 2> 22589 T731 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 22589 T763 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 22589 T747 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 22597 T689 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/c/b,null}
[junit4:junit4] 2> 22650 T689 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 49575
[junit4:junit4] 2> 22650 T689 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=22673042
[junit4:junit4] 2> 24092 T773 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 24093 T773 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:49575/c/b",
[junit4:junit4] 2> "node_name":"127.0.0.1:49575_c%2Fb",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:49575_c%2Fb_collection1",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "roles":null}
[junit4:junit4] 2> 24095 T773 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "collection":"onenodecollection",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:49575/c/b",
[junit4:junit4] 2> "node_name":"127.0.0.1:49575_c%2Fb",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:49575_c%2Fb_onenodecollectioncore",
[junit4:junit4] 2> "core":"onenodecollectioncore",
[junit4:junit4] 2> "roles":"none"}
[junit4:junit4] 2> 24099 T747 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 24099 T731 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 24099 T763 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 25652 T689 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 25653 T689 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 25655 T689 oasc.ZkController.publish publishing core=onenodecollectioncore state=down
[junit4:junit4] 2> 25656 T689 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@fb2072
[junit4:junit4] 2> 25659 T689 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 25659 T689 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 25659 T689 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 25659 T689 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 25661 T689 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 25662 T689 oasc.CachingDirectoryFactory.close Closing directory when closing factory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-1362854978515/jetty1
[junit4:junit4] 2> 25662 T689 oasc.CachingDirectoryFactory.closeDirectory Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-1362854978515/jetty1
[junit4:junit4] 2> 25662 T689 oasc.CachingDirectoryFactory.close Closing directory when closing factory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-1362854978515/jetty1/index
[junit4:junit4] 2> 25662 T689 oasc.CachingDirectoryFactory.closeDirectory Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-1362854978515/jetty1/index
[junit4:junit4] 2> 25663 T689 oasc.SolrCore.close [onenodecollectioncore] CLOSING SolrCore org.apache.solr.core.SolrCore@f94bdc
[junit4:junit4] 2> 25666 T689 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 25666 T689 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 25666 T689 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 25667 T689 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 25667 T689 oasc.SolrCore.closeSearcher [onenodecollectioncore] Closing main searcher on request.
[junit4:junit4] 2> 25668 T689 oasc.CachingDirectoryFactory.close Closing directory when closing factory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1362854978515/onenodecollection/index
[junit4:junit4] 2> 25668 T689 oasc.CachingDirectoryFactory.closeDirectory Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1362854978515/onenodecollection/index
[junit4:junit4] 2> 25668 T689 oasc.CachingDirectoryFactory.close Closing directory when closing factory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1362854978515/onenodecollection
[junit4:junit4] 2> 25668 T689 oasc.CachingDirectoryFactory.closeDirectory Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1362854978515/onenodecollection
[junit4:junit4] 2> 25669 T773 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=89316063872155654-127.0.0.1:49575_c%2Fb-n_0000000001) am no longer a leader.
[junit4:junit4] 2> 25670 T694 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d507ca3d60006
[junit4:junit4] 2> 25671 T691 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:57014 which had sessionid 0x13d507ca3d60006
[junit4:junit4] 2> 25671 T689 oaz.ZooKeeper.close Session: 0x13d507ca3d60006 closed
[junit4:junit4] 2> 25671 T694 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d507ca3d6000a type:delete cxid:0x55 zxid:0xe4 txntype:-1 reqpath:n/a Error Path:/solr/collections/collection1/leaders/shard2 Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders/shard2
[junit4:junit4] 2> 25672 T694 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d507ca3d60008 type:delete cxid:0x71 zxid:0xe5 txntype:-1 reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4] 2> 25672 T763 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 25672 T694 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d507ca3d6000a type:create cxid:0x56 zxid:0xe6 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 25672 T747 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 25673 T763 oasc.ShardLeaderElectionContext.shouldIBeLeader Checking if I should try and be the leader.
[junit4:junit4] 2> 25673 T763 oasc.ShardLeaderElectionContext.shouldIBeLeader My last published State was Active, it's okay to be the leader.
[junit4:junit4] 2> 25673 T763 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 25674 T763 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:50207/c/b/collection1/
[junit4:junit4] 2> 25674 T763 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:50207/c/b START replicas=[http://127.0.0.1:49575/c/b/collection1/] nUpdates=100
[junit4:junit4] 2> 25674 T763 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:50207/c/b DONE. We have no versions. sync failed.
[junit4:junit4] 2> 25674 T763 oasc.SyncStrategy.syncReplicas Leader's attempt to sync with shard failed, moving to the next canidate
[junit4:junit4] 2> 25674 T763 oasc.ShardLeaderElectionContext.runLeaderProcess We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
[junit4:junit4] 2> 25674 T763 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:50207/c/b/collection1/
[junit4:junit4] 2> 25674 T747 oasc.Overseer.start Overseer (id=89316063872155656-127.0.0.1:45621_c%2Fb-n_0000000002) starting
[junit4:junit4] 2> 25675 T763 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4] 2> 25675 T694 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d507ca3d60008 type:create cxid:0x76 zxid:0xe9 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 25676 T694 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d507ca3d60008 type:create cxid:0x77 zxid:0xea txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 25676 T694 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d507ca3d60008 type:create cxid:0x78 zxid:0xeb txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 25677 T694 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d507ca3d60008 type:create cxid:0x79 zxid:0xed txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 25679 T747 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 25679 T777 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 25679 T694 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d507ca3d6000a type:create cxid:0x5e zxid:0xee txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 25681 T763 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 25682 T747 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 25683 T776 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 25683 T763 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 25685 T776 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 25686 T776 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:49575/c/b",
[junit4:junit4] 2> "node_name":"127.0.0.1:49575_c%2Fb",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "roles":null}
[junit4:junit4] 2> 25689 T776 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "collection":"onenodecollection",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:49575/c/b",
[junit4:junit4] 2> "node_name":"127.0.0.1:49575_c%2Fb",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "core":"onenodecollectioncore",
[junit4:junit4] 2> "roles":"none"}
[junit4:junit4] 2> 25693 T689 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/c/b,null}
[junit4:junit4] 2> 25698 T747 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 25698 T763 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 25744 T689 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 45621
[junit4:junit4] 2> 25745 T689 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=9207106
[junit4:junit4] 2> 26746 T689 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 26746 T689 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 26750 T689 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@1fadc6b
[junit4:junit4] 2> 26752 T689 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 26752 T689 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 26753 T689 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 26753 T689 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 26754 T689 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 26754 T689 oasc.CachingDirectoryFactory.close Closing directory when closing factory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-1362854978515/jetty2
[junit4:junit4] 2> 26754 T689 oasc.CachingDirectoryFactory.closeDirectory Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-1362854978515/jetty2
[junit4:junit4] 2> 26755 T689 oasc.CachingDirectoryFactory.close Closing directory when closing factory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-1362854978515/jetty2/index
[junit4:junit4] 2> 26755 T689 oasc.CachingDirectoryFactory.closeDirectory Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-1362854978515/jetty2/index
[junit4:junit4] 2> 26756 T776 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=89316063872155656-127.0.0.1:45621_c%2Fb-n_0000000002) am no longer a leader.
[junit4:junit4] 2> 26756 T694 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d507ca3d60008
[junit4:junit4] 2> 26757 T691 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:57027 which had sessionid 0x13d507ca3d60008
[junit4:junit4] 2> 26757 T689 oaz.ZooKeeper.close Session: 0x13d507ca3d60008 closed
[junit4:junit4] 2> 26758 T694 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d507ca3d6000a type:delete cxid:0x66 zxid:0x100 txntype:-1 reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4] 2> 26758 T763 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 26760 T763 oasc.Overseer.start Overseer (id=89316063872155658-127.0.0.1:50207_c%2Fb-n_0000000003) starting
[junit4:junit4] 2> 26760 T694 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d507ca3d6000a type:create cxid:0x6b zxid:0x102 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 26761 T694 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d507ca3d6000a type:create cxid:0x6c zxid:0x103 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 26762 T694 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d507ca3d6000a type:create cxid:0x6d zxid:0x104 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 26762 T694 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d507ca3d6000a type:create cxid:0x6e zxid:0x105 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 26763 T779 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 26763 T763 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4:junit4] 2> 26764 T763 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 26764 T778 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 26766 T778 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 26766 T778 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:45621/c/b",
[junit4:junit4] 2> "node_name":"127.0.0.1:45621_c%2Fb",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:45621_c%2Fb_collection1",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "roles":null}
[junit4:junit4] 2> 26769 T778 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:45621/c/b",
[junit4:junit4] 2> "node_name":"127.0.0.1:45621_c%2Fb",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "roles":null}
[junit4:junit4] 2> 26772 T763 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 26779 T689 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/c/b,null}
[junit4:junit4] 2> 26830 T689 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 50207
[junit4:junit4] 2> 26831 T689 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=22768413
[junit4:junit4] 2> 26971 T731 oascc.ZkStateReader$3.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 26971 T731 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 26971 T731 oascc.ZkStateReader$2.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 26971 T731 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 27832 T689 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 27832 T689 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 27834 T689 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@2d4265
[junit4:junit4] 2> 27838 T689 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 27839 T689 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 27839 T689 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 27839 T689 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 27841 T689 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 27841 T689 oasc.CachingDirectoryFactory.close Closing directory when closing factory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-1362854978515/jetty3/index
[junit4:junit4] 2> 27841 T689 oasc.CachingDirectoryFactory.closeDirectory Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-1362854978515/jetty3/index
[junit4:junit4] 2> 27841 T689 oasc.CachingDirectoryFactory.close Closing directory when closing factory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-1362854978515/jetty3
[junit4:junit4] 2> 27842 T689 oasc.CachingDirectoryFactory.closeDirectory Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-1362854978515/jetty3
[junit4:junit4] 2> 27842 T689 oasc.CachingDirectoryFactory.close Closing directory when closing factory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-1362854978515/jetty3/index.20130310064957618
[junit4:junit4] 2> 27842 T689 oasc.CachingDirectoryFactory.closeDirectory Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-1362854978515/jetty3/index.20130310064957618
[junit4:junit4] 2> 27843 T778 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=89316063872155658-127.0.0.1:50207_c%2Fb-n_0000000003) am no longer a leader.
[junit4:junit4] 2> 27843 T694 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d507ca3d6000a
[junit4:junit4] 2> 27844 T763 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 27844 T763 oascc.ZkStateReader$2.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 27844 T689 oaz.ZooKeeper.close Session: 0x13d507ca3d6000a closed
[junit4:junit4] 2> 27844 T691 oazs.NIOServerCnxn.doIO WARNING caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x13d507ca3d6000a, likely client has closed socket
[junit4:junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
[junit4:junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:722)
[junit4:junit4] 2>
[junit4:junit4] 2> 27844 T763 oascc.ZkStateReader$3.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 27844 T763 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 27844 T691 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:57035 which had sessionid 0x13d507ca3d6000a
[junit4:junit4] 2> 27865 T689 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/c/b,null}
[junit4:junit4] 2> 27920 T689 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
[junit4:junit4] 2> 27920 T689 oazs.ZooKeeperServer.shutdown shutting down
[junit4:junit4] 2> 27920 T689 oazs.SessionTrackerImpl.shutdown Shutting down
[junit4:junit4] 2> 27920 T689 oazs.PrepRequestProcessor.shutdown Shutting down
[junit4:junit4] 2> 27921 T694 oazs.PrepRequestProcessor.run PrepRequestProcessor exited loop!
[junit4:junit4] 2> 27921 T689 oazs.SyncRequestProcessor.shutdown Shutting down
[junit4:junit4] 2> 27921 T693 oazs.SyncRequestProcessor.run SyncRequestProcessor exited!
[junit4:junit4] 2> 27921 T689 oazs.FinalRequestProcessor.shutdown shutdown of request processor complete
[junit4:junit4] 2> 27922 T689 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:51553 51553
[junit4:junit4] 2> 27924 T691 oazs.NIOServerCnxnFactory.run NIOServerCnxn factory exited run method
[junit4:junit4] 2> 27924 T689 oazs.ZooKeeperServer.shutdown shutting down
[junit4:junit4] 2> 27924 T689 oazs.SessionTrackerImpl.shutdown Shutting down
[junit4:junit4] 2> 27925 T689 oazs.PrepRequestProcessor.shutdown Shutting down
[junit4:junit4] 2> 27925 T689 oazs.SyncRequestProcessor.shutdown Shutting down
[junit4:junit4] 2> 27925 T689 oazs.FinalRequestProcessor.shutdown shutdown of request processor complete
[junit4:junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4:junit4] 1> / (2)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1>
[junit4:junit4] 1> /zookeeper (1)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1>
[junit4:junit4] 1> /solr (7)
[junit4:junit4] 1> /solr/collections (3)
[junit4:junit4] 1> /solr/collections/collection1 (3)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"configName":"conf1"}
[junit4:junit4] 1> /solr/collections/collection1/leaders (2)
[junit4:junit4] 1> /solr/collections/collection1/leaders/shard1 (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "base_url":"http://127.0.0.1:45621/c/b",
[junit4:junit4] 1> "node_name":"127.0.0.1:45621_c%2Fb",
[junit4:junit4] 1> "core":"collection1"}
[junit4:junit4] 1> /solr/collections/collection1/leaders/shard2 (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "base_url":"http://127.0.0.1:49575/c/b",
[junit4:junit4] 1> "node_name":"127.0.0.1:49575_c%2Fb",
[junit4:junit4] 1> "core":"collection1"}
[junit4:junit4] 1> /solr/collections/collection1/leader_elect (2)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1 (1)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election (1)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89316063872155656-127.0.0.1:45621_c%2Fb_collection1-n_0000000000 (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard2 (1)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard2/election (2)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard2/election/89316063872155654-127.0.0.1:49575_c%2Fb_collection1-n_0000000000 (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard2/election/89316063872155658-127.0.0.1:50207_c%2Fb_collection1-n_0000000001 (0)
[junit4:junit4] 1> /solr/collections/collection1/shards (0)
[junit4:junit4] 1> /solr/collections/control_collection (3)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"configName":"conf1"}
[junit4:junit4] 1> /solr/collections/control_collection/leaders (1)
[junit4:junit4] 1> /solr/collections/control_collection/leaders/shard1 (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "base_url":"http://127.0.0.1:47673/c/b",
[junit4:junit4] 1> "node_name":"127.0.0.1:47673_c%2Fb",
[junit4:junit4] 1> "core":"collection1"}
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect (1)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect/shard1 (1)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect/shard1/election (1)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect/shard1/election/89316063872155651-127.0.0.1:47673_c%2Fb_collection1-n_0000000000 (0)
[junit4:junit4] 1> /solr/collections/control_collection/shards (0)
[junit4:junit4] 1> /solr/collections/onenodecollection (2)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "router":"compositeId",
[junit4:junit4] 1> "configName":"conf1"}
[junit4:junit4] 1> /solr/collections/onenodecollection/leaders (1)
[junit4:junit4] 1> /solr/collections/onenodecollection/leaders/shard1 (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "base_url":"http://127.0.0.1:49575/c/b",
[junit4:junit4] 1> "node_name":"127.0.0.1:49575_c%2Fb",
[junit4:junit4] 1> "core":"onenodecollectioncore"}
[junit4:junit4] 1> /solr/collections/onenodecollection/leader_elect (1)
[junit4:junit4] 1> /solr/collections/onenodecollection/leader_elect/shard1 (1)
[junit4:junit4] 1> /solr/collections/onenodecollection/leader_elect/shard1/election (1)
[junit4:junit4] 1> /solr/collections/onenodecollection/leader_elect/shard1/election/89316063872155654-127.0.0.1:49575_c%2Fb_onenodecollectioncore-n_0000000000 (0)
[junit4:junit4] 1> /solr/clusterstate.json (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "collection1":{
[junit4:junit4] 1> "shards":{
[junit4:junit4] 1> "shard2":{
[junit4:junit4] 1> "range":"0-7fffffff",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "replicas":{
[junit4:junit4] 1> "127.0.0.1:49575_c%2Fb_collection1":{
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "shard":"shard2",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:49575/c/b",
[junit4:junit4] 1> "node_name":"127.0.0.1:49575_c%2Fb",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "leader":"true"},
[junit4:junit4] 1> "127.0.0.1:50207_c%2Fb_collection1":{
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "shard":"shard2",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:50207/c/b",
[junit4:junit4] 1> "node_name":"127.0.0.1:50207_c%2Fb",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "core":"collection1"}}},
[junit4:junit4] 1> "shard1":{
[junit4:junit4] 1> "range":"80000000-ffffffff",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "replicas":{"127.0.0.1:45621_c%2Fb_collection1":{
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:45621/c/b",
[junit4:junit4] 1> "node_name":"127.0.0.1:45621_c%2Fb",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "leader":"true"}}}},
[junit4:junit4] 1> "router":"compositeId"},
[junit4:junit4] 1> "control_collection":{
[junit4:junit4] 1> "shards":{"shard1":{
[junit4:junit4] 1> "range":"80000000-7fffffff",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "replicas":{"127.0.0.1:47673_c%2Fb_collection1":{
[junit4:junit4] 1> "collection":"control_collection",
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:47673/c/b",
[junit4:junit4] 1> "node_name":"127.0.0.1:47673_c%2Fb",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "leader":"true"}}}},
[junit4:junit4] 1> "router":"compositeId"},
[junit4:junit4] 1> "onenodecollection":{
[junit4:junit4] 1> "shards":{"shard1":{
[junit4:junit4] 1> "range":"80000000-7fffffff",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "replicas":{"127.0.0.1:49575_c%2Fb_onenodecollectioncore":{
[junit4:junit4] 1> "collection":"onenodecollection",
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:49575/c/b",
[junit4:junit4] 1> "node_name":"127.0.0.1:49575_c%2Fb",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "core":"onenodecollectioncore",
[junit4:junit4] 1> "roles":"none",
[...truncated too long message...]
4] 1> DATA:
[junit4:junit4] 1>
[junit4:junit4] 1> /solr/overseer/collection-queue-work (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1>
[junit4:junit4] 1> /solr/overseer/queue (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1>
[junit4:junit4] 1> /solr/overseer/queue-work (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1>
[junit4:junit4] 1> /solr/aliases.json (0)
[junit4:junit4] 1> /solr/live_nodes (5)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:59877_ (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:32998_ (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:46407_ (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:40227_ (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:42929_ (0)
[junit4:junit4] 1>
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=UnloadDistributedZkTest -Dtests.method=testDistribSearch -Dtests.seed=24CF929680B023C9 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=en_CA -Dtests.timezone=Europe/San_Marino -Dtests.file.encoding=US-ASCII
[junit4:junit4] FAILURE 137s J1 | UnloadDistributedZkTest.testDistribSearch <<<
[junit4:junit4] > Throwable #1: java.lang.AssertionError: Still found shard
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([24CF929680B023C9:A5291C8EF7EF43F5]:0)
[junit4:junit4] > at org.junit.Assert.fail(Assert.java:93)
[junit4:junit4] > at org.apache.solr.cloud.UnloadDistributedZkTest.testUnloadShardAndCollection(UnloadDistributedZkTest.java:124)
[junit4:junit4] > at org.apache.solr.cloud.UnloadDistributedZkTest.doTest(UnloadDistributedZkTest.java:77)
[junit4:junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:806)
[junit4:junit4] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit4:junit4] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
[junit4:junit4] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit4:junit4] > at java.lang.reflect.Method.invoke(Method.java:487)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
[junit4:junit4] > at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
[junit4:junit4] > at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
[junit4:junit4] > at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4:junit4] > at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
[junit4:junit4] > at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
[junit4:junit4] > at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
[junit4:junit4] > at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
[junit4:junit4] > at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4] > at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
[junit4:junit4] > at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4:junit4] > at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
[junit4:junit4] > at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:722)
[junit4:junit4] 2> 137024 T1421 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4] 2> 137035 T1420 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 2 leaked thread(s).
[junit4:junit4] 2> 137876 T1424 oazs.SessionTrackerImpl.run SessionTrackerImpl exited loop!
[junit4:junit4] 2> NOTE: test params are: codec=Lucene40, sim=DefaultSimilarity, locale=en_CA, timezone=Europe/San_Marino
[junit4:junit4] 2> NOTE: Linux 3.2.0-38-generic i386/Oracle Corporation 1.8.0-ea (32-bit)/cpus=8,threads=1,free=82802008,total=195317760
[junit4:junit4] 2> NOTE: All tests run in this JVM: [TestAnalyzedSuggestions, TestCollationField, RequiredFieldsTest, CoreContainerCoreInitFailuresTest, TestZkChroot, SolrCoreCheckLockOnStartupTest, HighlighterConfigTest, TestMergePolicyConfig, BadIndexSchemaTest, DebugComponentTest, NumericFieldsTest, NotRequiredUniqueKeyTest, SolrIndexConfigTest, HardAutoCommitTest, TestFiltering, OpenExchangeRatesOrgProviderTest, TestStressReorder, StatsComponentTest, StandardRequestHandlerTest, SortByFunctionTest, DirectSolrSpellCheckerTest, CurrencyFieldOpenExchangeTest, SpellCheckCollatorTest, DOMUtilTest, TestLuceneMatchVersion, TestLMJelinekMercerSimilarityFactory, TestPostingsSolrHighlighter, DisMaxRequestHandlerTest, DistanceFunctionTest, FieldAnalysisRequestHandlerTest, TestElisionMultitermQuery, TestCSVResponseWriter, DirectUpdateHandlerTest, TestSolrDiscoveryProperties, TestCSVLoader, TestUpdate, EchoParamsTest, TestSolrXMLSerializer, URLClassifyProcessorTest, TestCopyFieldCollectionResource, SolrRequestParserTest, SampleTest, TestDistributedGrouping, AliasIntegrationTest, MultiTermTest, SignatureUpdateProcessorFactoryTest, SimplePostToolTest, IndexBasedSpellCheckerTest, CurrencyFieldXmlFileTest, TestSystemIdResolver, TestDFRSimilarityFactory, TestUtils, TestSuggestSpellingConverter, BasicDistributedZk2Test, ConvertedLegacyTest, ClusterStateTest, TestDistributedSearch, TestReload, TestQuerySenderNoQuery, SpellCheckComponentTest, TimeZoneUtilsTest, XsltUpdateRequestHandlerTest, TestRangeQuery, ChaosMonkeySafeLeaderTest, UnloadDistributedZkTest]
[junit4:junit4] Completed on J1 in 138.06s, 1 test, 1 failure <<< FAILURES!
[...truncated 423 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:381: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:361: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:183: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:449: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1213: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:877: There were test failures: 269 suites, 1137 tests, 1 error, 1 failure, 13 ignored (7 assumptions)
Total time: 39 minutes 17 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Description set: Java: 32bit/jdk1.8.0-ea-b79 -server -XX:+UseSerialGC
Email was triggered for: Failure
Sending email for trigger: Failure