You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/03/29 21:45:18 UTC

[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_51) - Build # 9941 - Still Failing!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/9941/
Java: 32bit/jdk1.7.0_51 -server -XX:+UseParallelGC

1 tests failed.
FAILED:  org.apache.solr.client.solrj.impl.CloudSolrServerTest.testDistribSearch

Error Message:


Stack Trace:
java.lang.AssertionError
	at __randomizedtesting.SeedInfo.seed([F84D670205A65415:79ABE91A72F93429]:0)
	at org.junit.Assert.fail(Assert.java:92)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertTrue(Assert.java:54)
	at org.apache.solr.client.solrj.impl.CloudSolrServerTest.doTest(CloudSolrServerTest.java:256)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:867)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1617)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:826)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:862)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:876)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:783)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:443)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:835)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:771)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:782)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
	at java.lang.Thread.run(Thread.java:744)




Build Log:
[...truncated 12028 lines...]
   [junit4] Suite: org.apache.solr.client.solrj.impl.CloudSolrServerTest
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1396125852407
   [junit4]   2> 22977 T99 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
   [junit4]   2> 22977 T99 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 22979 T99 oasc.AbstractZkTestCase.<clinit> WARN TEST_HOME() does not exist - solrj test?
   [junit4]   2> 22983 T99 oas.SolrTestCaseJ4.setUp ###Starting testShutdown
   [junit4]   2> 22985 T99 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 22987 T100 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 23087 T99 oasc.ZkTestServer.run start zk server on port:60470
   [junit4]   2> 23126 T99 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 23233 T106 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@60461 name:ZooKeeperConnection Watcher:127.0.0.1:60470 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 23234 T99 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 23237 T99 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 23267 T99 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 23269 T108 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1f2507f name:ZooKeeperConnection Watcher:127.0.0.1:60470/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 23269 T99 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 23274 T99 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 23278 T99 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 23282 T99 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 23284 T99 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 23288 T99 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/solrconfig.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 23288 T99 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 23293 T99 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 23293 T99 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 23297 T99 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml because it doesn't exist
   [junit4]   2> 23297 T99 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/stopwords.txt because it doesn't exist
   [junit4]   2> 23297 T99 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/protwords.txt because it doesn't exist
   [junit4]   2> 23298 T99 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/currency.xml because it doesn't exist
   [junit4]   2> 23298 T99 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/enumsConfig.xml because it doesn't exist
   [junit4]   2> 23298 T99 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/open-exchange-rates.json because it doesn't exist
   [junit4]   2> 23299 T99 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/mapping-ISOLatin1Accent.txt because it doesn't exist
   [junit4]   2> 23299 T99 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/old_synonyms.txt because it doesn't exist
   [junit4]   2> 23299 T99 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/synonyms.txt because it doesn't exist
   [junit4]   2> 23303 T99 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 23306 T110 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@9714e6 name:ZooKeeperConnection Watcher:127.0.0.1:60470/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 23306 T99 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 23316 T99 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 23317 T111 oaz.ClientCnxnSocketNIO.connect ERROR Unable to open socket to ff01::114/ff01:0:0:0:0:0:0:114:33332
   [junit4]   2> 23318 T111 oaz.ClientCnxn$SendThread.run WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.SocketException: Network is unreachable
   [junit4]   2> 	at sun.nio.ch.Net.connect0(Native Method)
   [junit4]   2> 	at sun.nio.ch.Net.connect(Net.java:465)
   [junit4]   2> 	at sun.nio.ch.Net.connect(Net.java:457)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:277)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:287)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:967)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
   [junit4]   2> 
   [junit4]   2> 24420 T111 oaz.ClientCnxnSocketNIO.connect ERROR Unable to open socket to ff01::114/ff01:0:0:0:0:0:0:114:33332
   [junit4]   2> 24522 T99 oas.SolrTestCaseJ4.tearDown ###Ending testShutdown
   [junit4]   2> 24524 T99 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:60470 60470
   [junit4]   2> 24589 T100 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:60470 60470
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 24597 T99 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 24598 T99 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 24598 T113 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 24698 T99 oasc.ZkTestServer.run start zk server on port:46939
   [junit4]   2> 24699 T99 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 24703 T119 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e7453c name:ZooKeeperConnection Watcher:127.0.0.1:46939 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 24703 T99 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 24704 T99 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 24708 T99 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 24709 T121 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@944f1 name:ZooKeeperConnection Watcher:127.0.0.1:46939/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 24709 T99 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 24709 T99 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 24713 T99 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 24716 T99 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 24718 T99 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 24721 T99 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/solrconfig.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 24722 T99 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 24726 T99 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 24727 T99 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 24730 T99 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml because it doesn't exist
   [junit4]   2> 24730 T99 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/stopwords.txt because it doesn't exist
   [junit4]   2> 24730 T99 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/protwords.txt because it doesn't exist
   [junit4]   2> 24731 T99 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/currency.xml because it doesn't exist
   [junit4]   2> 24731 T99 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/enumsConfig.xml because it doesn't exist
   [junit4]   2> 24731 T99 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/open-exchange-rates.json because it doesn't exist
   [junit4]   2> 24732 T99 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/mapping-ISOLatin1Accent.txt because it doesn't exist
   [junit4]   2> 24732 T99 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/old_synonyms.txt because it doesn't exist
   [junit4]   2> 24732 T99 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/synonyms.txt because it doesn't exist
   [junit4]   2> 24734 T99 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 24735 T123 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5186b6 name:ZooKeeperConnection Watcher:127.0.0.1:46939/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 24735 T99 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 24822 T99 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 24975 T99 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 24984 T99 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:35500
   [junit4]   2> 24988 T99 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 24988 T99 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 24988 T99 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1396125852407/org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1396125854170
   [junit4]   2> 24989 T99 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1396125852407/org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1396125854170/'
   [junit4]   2> 25000 T99 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1396125852407/org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1396125854170/solr.xml
   [junit4]   2> 25050 T99 oasc.CoreContainer.<init> New CoreContainer 8660371
   [junit4]   2> 25050 T99 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1396125852407/org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1396125854170/]
   [junit4]   2> 25051 T99 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 25052 T99 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 25052 T99 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 25053 T99 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 25053 T99 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 25053 T99 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 25054 T99 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 25054 T99 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 25055 T99 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 25058 T99 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 25058 T99 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 25059 T99 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 25059 T99 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:46939/solr
   [junit4]   2> 25068 T99 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 25069 T99 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 25072 T134 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e83c1e name:ZooKeeperConnection Watcher:127.0.0.1:46939 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 25073 T99 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 25079 T99 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 25080 T136 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@184425c name:ZooKeeperConnection Watcher:127.0.0.1:46939/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 25081 T99 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 25087 T99 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 25094 T99 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 25100 T99 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 25105 T99 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 25109 T99 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 25119 T99 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 25122 T99 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35500_
   [junit4]   2> 25124 T99 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:35500_
   [junit4]   2> 25129 T99 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 25134 T99 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 25144 T99 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:35500_
   [junit4]   2> 25145 T99 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 25148 T99 oasc.Overseer.start Overseer (id=91496503969775620-127.0.0.1:35500_-n_0000000000) starting
   [junit4]   2> 25158 T99 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 25180 T138 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 25181 T99 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 25182 T138 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 25184 T99 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 25186 T99 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 25194 T137 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 25201 T139 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 25201 T139 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 25205 T136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 25205 T139 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 25209 T137 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 25210 T137 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:35500",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:35500_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 25211 T137 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 25218 T137 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 25223 T136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 25227 T136 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 26206 T139 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 26206 T139 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 26207 T139 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 26208 T139 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 26210 T139 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 26211 T139 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1396125852407/org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1396125854170/collection1/'
   [junit4]   2> 26251 T139 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 26283 T139 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 26285 T139 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 26296 T139 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 26587 T139 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 26592 T139 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 26596 T139 oass.IndexSchema.readSchema WARN Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
   [junit4]   2> 26596 T139 oass.IndexSchema.readSchema WARN Field text is not multivalued and destination for multiple copyFields (3)
   [junit4]   2> 26597 T139 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 26598 T139 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 26598 T139 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1396125852407/org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1396125854170/collection1/, dataDir=./solrtest-CloudSolrServerTest-1396125852407/org.apache.solr.client.solrj.impl.CloudSolrServerTest-1396125854029/control/data/
   [junit4]   2> 26599 T139 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 26599 T139 oasc.CachingDirectoryFactory.get return new directory for ./solrtest-CloudSolrServerTest-1396125852407/org.apache.solr.client.solrj.impl.CloudSolrServerTest-1396125854029/control/data
   [junit4]   2> 26600 T139 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solrtest-CloudSolrServerTest-1396125852407/org.apache.solr.client.solrj.impl.CloudSolrServerTest-1396125854029/control/data/index/
   [junit4]   2> 26600 T139 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solrtest-CloudSolrServerTest-1396125852407/org.apache.solr.client.solrj.impl.CloudSolrServerTest-1396125854029/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 26601 T139 oasc.CachingDirectoryFactory.get return new directory for ./solrtest-CloudSolrServerTest-1396125852407/org.apache.solr.client.solrj.impl.CloudSolrServerTest-1396125854029/control/data/index
   [junit4]   2> 26603 T139 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1957aab lockFactory=NativeFSLockFactory@./solrtest-CloudSolrServerTest-1396125852407/org.apache.solr.client.solrj.impl.CloudSolrServerTest-1396125854029/control/data/index),segFN=segments_1,generation=1}
   [junit4]   2> 26604 T139 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 26608 T139 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 26609 T139 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 26609 T139 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 26610 T139 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 26610 T139 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 26611 T139 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 26612 T139 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 26612 T139 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: solr.PingRequestHandler
   [junit4]   2> 26615 T139 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 26618 T139 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 26618 T139 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 26620 T139 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1957aab lockFactory=NativeFSLockFactory@./solrtest-CloudSolrServerTest-1396125852407/org.apache.solr.client.solrj.impl.CloudSolrServerTest-1396125854029/control/data/index),segFN=segments_1,generation=1}
   [junit4]   2> 26620 T139 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 26621 T139 oass.SolrIndexSearcher.<init> Opening Searcher@9f809d[collection1] main
   [junit4]   2> 26621 T139 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 26624 T139 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 26624 T139 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 26629 T139 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 26629 T139 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 26630 T139 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 26632 T139 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 26632 T139 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 26633 T139 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 26633 T139 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 26636 T140 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@9f809d[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 26636 T139 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 26638 T143 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:35500 collection:control_collection shard:shard1
   [junit4]   2> 26638 T99 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0
   [junit4]   2> 26639 T99 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 26650 T99 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 26653 T145 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@796bb2 name:ZooKeeperConnection Watcher:127.0.0.1:46939/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 26653 T143 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 26654 T99 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 26659 T99 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 26670 T99 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 26672 T143 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 26678 T136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 26680 T143 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 26680 T143 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 26682 T137 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> ASYNC  NEW_CORE C11 name=collection1 org.apache.solr.core.SolrCore@48d43e url=https://127.0.0.1:35500/collection1 node=127.0.0.1:35500_ C11_STATE=coll:control_collection core:collection1 props:{state=down, base_url=https://127.0.0.1:35500, core=collection1, node_name=127.0.0.1:35500_}
   [junit4]   2> 26683 T143 C11 P35500 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:35500/collection1/
   [junit4]   2> 26685 T143 C11 P35500 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 26685 T143 C11 P35500 oasc.SyncStrategy.syncToMe https://127.0.0.1:35500/collection1/ has no replicas
   [junit4]   2> 26686 T143 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:35500/collection1/ shard1
   [junit4]   2> 26688 T143 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 26697 T136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 26706 T136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 26714 T136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 26737 T99 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 26738 T99 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 26742 T99 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 26746 T99 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:42972
   [junit4]   2> 26749 T99 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 26749 T99 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 26750 T99 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1396125852407/org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1396125856101
   [junit4]   2> 26751 T99 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1396125852407/org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1396125856101/'
   [junit4]   2> 26763 T99 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1396125852407/org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1396125856101/solr.xml
   [junit4]   2> 26802 T99 oasc.CoreContainer.<init> New CoreContainer 10864742
   [junit4]   2> 26802 T99 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1396125852407/org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1396125856101/]
   [junit4]   2> 26803 T99 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 26804 T99 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 26804 T99 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 26804 T99 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 26805 T99 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 26805 T99 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 26805 T99 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 26806 T99 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 26806 T99 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 26809 T99 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 26809 T99 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 26810 T99 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 26810 T99 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:46939/solr
   [junit4]   2> 26811 T99 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 26811 T99 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 26814 T156 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@181b8af name:ZooKeeperConnection Watcher:127.0.0.1:46939 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 26814 T99 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 26818 T145 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 26818 T136 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 26820 T99 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 26824 T158 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1041757 name:ZooKeeperConnection Watcher:127.0.0.1:46939/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 26824 T99 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 26838 T99 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 26857 T143 oasc.ZkController.register We are https://127.0.0.1:35500/collection1/ and leader is https://127.0.0.1:35500/collection1/
   [junit4]   2> 26857 T143 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:35500
   [junit4]   2> 26858 T143 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 26858 T143 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 26858 T143 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 26861 T136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 26861 T136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 26862 T136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 26862 T143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 26865 T137 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 26867 T137 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:35500",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:35500_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 26872 T136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 26978 T136 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 26979 T145 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 26978 T158 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 27845 T99 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42972_
   [junit4]   2> 27848 T99 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:42972_
   [junit4]   2> 27853 T158 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 27853 T145 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 27854 T136 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 27872 T159 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 27873 T159 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 27876 T136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 27876 T136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 27876 T136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 27876 T159 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 27878 T137 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 27879 T137 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:42972",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:42972_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 27880 T137 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 27880 T137 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 27885 T136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 27991 T158 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 27991 T136 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 27991 T145 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 28877 T159 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 28877 T159 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 28878 T159 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 28879 T159 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 28880 T159 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 28880 T159 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1396125852407/org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1396125856101/collection1/'
   [junit4]   2> 28929 T159 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 28981 T159 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 28983 T159 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 28999 T159 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 29317 T159 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 29320 T159 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 29321 T159 oass.IndexSchema.readSchema WARN Field text is not multivalued and destination for multiple copyFields (3)
   [junit4]   2> 29322 T159 oass.IndexSchema.readSchema WARN Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
   [junit4]   2> 29322 T159 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 29322 T159 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 29323 T159 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1396125852407/org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1396125856101/collection1/, dataDir=./solrtest-CloudSolrServerTest-1396125852407/org.apache.solr.client.solrj.impl.CloudSolrServerTest-1396125854029/jetty1/
   [junit4]   2> 29323 T159 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 29323 T159 oasc.CachingDirectoryFactory.get return new directory for ./solrtest-CloudSolrServerTest-1396125852407/org.apache.solr.client.solrj.impl.CloudSolrServerTest-1396125854029/jetty1
   [junit4]   2> 29324 T159 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solrtest-CloudSolrServerTest-1396125852407/org.apache.solr.client.solrj.impl.CloudSolrServerTest-1396125854029/jetty1/index/
   [junit4]   2> 29324 T159 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solrtest-CloudSolrServerTest-1396125852407/org.apache.solr.client.solrj.impl.CloudSolrServerTest-1396125854029/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 29324 T159 oasc.CachingDirectoryFactory.get return new directory for ./solrtest-CloudSolrServerTest-1396125852407/org.apache.solr.client.solrj.impl.CloudSolrServerTest-1396125854029/jetty1/index
   [junit4]   2> 29326 T159 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@14c1b45 lockFactory=NativeFSLockFactory@./solrtest-CloudSolrServerTest-1396125852407/org.apache.solr.client.solrj.impl.CloudSolrServerTest-1396125854029/jetty1/index),segFN=segments_1,generation=1}
   [junit4]   2> 29326 T159 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 29331 T159 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 29331 T159 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 29331 T159 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 29332 T159 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 29332 T159 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 29332 T159 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 29332 T159 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 29333 T159 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: solr.PingRequestHandler
   [junit4]   2> 29335 T159 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 29337 T159 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 29338 T159 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 29338 T159 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@14c1b45 lockFactory=NativeFSLockFactory@./solrtest-CloudSolrServerTest-1396125852407/org.apache.solr.client.solrj.impl.CloudSolrServerTest-1396125854029/jetty1/index),segFN=segments_1,generation=1}
   [junit4]   2> 29339 T159 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 29339 T159 oass.SolrIndexSearcher.<init> Opening Searcher@1f50fec[collection1] main
   [junit4]   2> 29339 T159 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 29341 T159 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 29341 T159 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 29341 T159 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 29342 T159 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 29342 T159 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 29343 T159 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 29343 T159 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 29343 T159 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 29343 T159 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 29344 T159 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 29344 T160 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1f50fec[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 29345 T163 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:42972 collection:collection1 shard:shard1
   [junit4]   2> 29345 T99 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0
   [junit4]   2> 29346 T99 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 29346 T163 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 29356 T163 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 29358 T136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 29359 T136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 29359 T136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 29359 T163 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 29360 T163 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C12 name=collection1 org.apache.solr.core.SolrCore@4721b6 url=https://127.0.0.1:42972/collection1 node=127.0.0.1:42972_ C12_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:42972, core=collection1, node_name=127.0.0.1:42972_}
   [junit4]   2> 29360 T163 C12 P42972 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:42972/collection1/
   [junit4]   2> 29361 T137 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 29362 T163 C12 P42972 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 29362 T163 C12 P42972 oasc.SyncStrategy.syncToMe https://127.0.0.1:42972/collection1/ has no replicas
   [junit4]   2> 29362 T163 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:42972/collection1/ shard1
   [junit4]   2> 29363 T163 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 29367 T136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 29376 T136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 29381 T136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 29404 T99 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 29405 T99 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 29409 T99 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 29411 T99 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:38370
   [junit4]   2> 29414 T99 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 29415 T99 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 29415 T99 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solrtest-CloudSolrServerTest-1396125852407/org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1396125858779
   [junit4]   2> 29415 T99 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solrtest-CloudSolrServerTest-1396125852407/org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1396125858779/'
   [junit4]   2> 29428 T99 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1396125852407/org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1396125858779/solr.xml
   [junit4]   2> 29464 T99 oasc.CoreContainer.<init> New CoreContainer 20491738
   [junit4]   2> 29464 T99 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solrtest-CloudSolrServerTest-1396125852407/org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1396125858779/]
   [junit4]   2> 29465 T99 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 29466 T99 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 29466 T99 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 29466 T99 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 29467 T99 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 29467 T99 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 29467 T99 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 29467 T99 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 29468 T99 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 29471 T99 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 29471 T99 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 29471 T99 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 29472 T99 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:46939/solr
   [junit4]   2> 29472 T99 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 29473 T99 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 29474 T174 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@62a98a name:ZooKeeperConnection Watcher:127.0.0.1:46939 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 29475 T99 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 29477 T99 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 29481 T176 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1f5f8a1 name:ZooKeeperConnection Watcher:127.0.0.1:46939/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 29481 T99 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 29485 T158 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 29486 T145 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 29485 T136 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 29494 T99 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 29528 T163 oasc.ZkController.register We are https://127.0.0.1:42972/collection1/ and leader is https://127.0.0.1:42972/collection1/
   [junit4]   2> 29528 T163 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:42972
   [junit4]   2> 29528 T163 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 29528 T163 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 29528 T163 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 29530 T136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 29530 T136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 29530 T136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 29531 T163 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 29532 T137 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 29534 T137 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:42972",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:42972_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 29538 T136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 29642 T136 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 29643 T176 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 29643 T158 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 29643 T145 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 30499 T99 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38370_
   [junit4]   2> 30503 T99 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:38370_
   [junit4]   2> 30508 T136 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 30508 T145 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 30509 T158 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 30509 T176 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 30525 T177 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 30526 T177 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 30528 T136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 30528 T177 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 30529 T136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 30529 T136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 30533 T137 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 30535 T137 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:38370",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:38370_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 30536 T137 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 30536 T137 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 30542 T136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 30648 T176 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 30648 T136 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 30648 T158 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 30648 T145 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 31529 T177 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 31530 T177 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 31532 T177 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 31532 T177 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 31535 T177 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 31535 T177 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solrtest-CloudSolrServerTest-1396125852407/org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1396125858779/collection1/'
   [junit4]   2> 31579 T177 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 31616 T177 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 31619 T177 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 31628 T177 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 31899 T177 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 31901 T177 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 31903 T177 oass.IndexSchema.readSchema WARN Field text is not multivalued and destination for multiple copyFields (3)
   [junit4]   2> 31903 T177 oass.IndexSchema.readSchema WARN Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
   [junit4]   2> 31904 T177 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 31904 T177 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 31904 T177 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solrtest-CloudSolrServerTest-1396125852407/org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1396125858779/collection1/, dataDir=./solrtest-CloudSolrServerTest-1396125852407/org.apache.solr.client.solrj.impl.CloudSolrServerTest-1396125854029/jetty2/
   [junit4]   2> 31904 T177 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 31905 T177 oasc.CachingDirectoryFactory.get return new directory for ./solrtest-CloudSolrServerTest-1396125852407/org.apache.solr.client.solrj.impl.CloudSolrServerTest-1396125854029/jetty2
   [junit4]   2> 31905 T177 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solrtest-CloudSolrServerTest-1396125852407/org.apache.solr.client.solrj.impl.CloudSolrServerTest-1396125854029/jetty2/index/
   [junit4]   2> 31905 T177 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solrtest-CloudSolrServerTest-1396125852407/org.apache.solr.client.solrj.impl.CloudSolrServerTest-1396125854029/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 31906 T177 oasc.CachingDirectoryFactory.get return new directory for ./solrtest-CloudSolrServerTest-1396125852407/org.apache.solr.client.solrj.impl.CloudSolrServerTest-1396125854029/jetty2/index
   [junit4]   2> 31907 T177 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@5e618 lockFactory=NativeFSLockFactory@./solrtest-CloudSolrServerTest-1396125852407/org.apache.solr.client.solrj.impl.CloudSolrServerTest-1396125854029/jetty2/index),segFN=segments_1,generation=1}
   [junit4]   2> 31907 T177 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 31908 T177 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 31908 T177 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 31908 T177 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 31909 T177 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 31909 T177 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 31909 T177 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 31909 T177 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 31910 T177 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: solr.PingRequestHandler
   [junit4]   2> 31911 T177 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 31914 T177 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 31914 T177 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 31915 T177 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@5e618 lockFactory=NativeFSLockFactory@./solrtest-CloudSolrServerTest-1396125852407/org.apache.solr.client.solrj.impl.CloudSolrServerTest-1396125854029/jetty2/index),segFN=segments_1,generation=1}
   [junit4]   2> 31915 T177 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 31915 T177 oass.SolrIndexSearcher.<init> Opening Searcher@132f845[collection1] main
   [junit4]   2> 31916 T177 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 31917 T177 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 31917 T177 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 31919 T177 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 31919 T177 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 31919 T177 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 31921 T177 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 31921 T177 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 31921 T177 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 31921 T177 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 31923 T178 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@132f845[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 31923 T177 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 31924 T181 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:38370 collection:collection1 shard:shard2
   [junit4]   2> 31925 T99 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0
   [junit4]   2> 31926 T99 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 31927 T181 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 31935 T181 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 31937 T136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 31937 T136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 31937 T136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 31938 T181 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 31938 T181 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C13 name=collection1 org.apache.solr.core.SolrCore@503e0e url=https://127.0.0.1:38370/collection1 node=127.0.0.1:38370_ C13_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:38370, core=collection1, node_name=127.0.0.1:38370_}
   [junit4]   2> 31938 T181 C13 P38370 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:38370/collection1/
   [junit4]   2> 31938 T181 C13 P38370 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 31938 T181 C13 P38370 oasc.SyncStrategy.syncToMe https://127.0.0.1:38370/collection1/ has no replicas
   [junit4]   2> 31938 T181 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:38370/collection1/ shard2
   [junit4]   2> 31939 T181 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 31939 T137 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 31942 T136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 31944 T136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 31948 T136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 31974 T99 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 31975 T99 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 31983 T99 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 31991 T99 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:45443
   [junit4]   2> 31995 T99 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 31995 T99 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 31996 T99 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1396125852407/org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1396125861360
   [junit4]   2> 31996 T99 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1396125852407/org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1396125861360/'
   [junit4]   2> 32012 T99 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1396125852407/org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1396125861360/solr.xml
   [junit4]   2> 32055 T145 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 32056 T176 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 32056 T136 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 32055 T158 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 32062 T99 oasc.CoreContainer.<init> New CoreContainer 23683737
   [junit4]   2> 32062 T99 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1396125852407/org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1396125861360/]
   [junit4]   2> 32063 T99 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 32064 T99 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 32064 T99 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 32064 T99 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 32065 T99 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 32065 T99 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 32065 T99 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 32065 T99 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 32066 T99 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 32069 T99 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 32069 T99 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 32069 T99 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 32070 T99 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:46939/solr
   [junit4]   2> 32070 T99 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 32071 T99 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 32073 T192 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1004776 name:ZooKeeperConnection Watcher:127.0.0.1:46939 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 32073 T99 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 32076 T99 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 32077 T194 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4b529f name:ZooKeeperConnection Watcher:127.0.0.1:46939/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 32078 T99 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 32092 T99 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 32095 T181 oasc.ZkController.register We are https://127.0.0.1:38370/collection1/ and leader is https://127.0.0.1:38370/collection1/
   [junit4]   2> 32096 T181 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:38370
   [junit4]   2> 32096 T181 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 32096 T181 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 32097 T181 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 32099 T136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 32099 T136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 32099 T136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 32100 T181 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 32102 T137 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 32103 T137 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:38370",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:38370_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 32109 T136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 32215 T136 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 32215 T145 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 32216 T158 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 32215 T194 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 32215 T176 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 33098 T99 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45443_
   [junit4]   2> 33101 T99 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:45443_
   [junit4]   2> 33106 T145 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 33106 T158 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 33107 T176 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 33107 T194 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 33107 T136 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 33127 T195 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 33127 T195 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 33130 T136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 33130 T195 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 33130 T136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 33131 T136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 33134 T137 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 33137 T137 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:45443",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:45443_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 33137 T137 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 33137 T137 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 33143 T136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 33248 T136 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 33249 T194 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 33249 T158 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 33248 T145 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 33248 T176 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 34131 T195 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 34131 T195 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 34133 T195 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 34133 T195 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 34136 T195 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 34136 T195 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1396125852407/org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1396125861360/collection1/'
   [junit4]   2> 34173 T195 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 34204 T195 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 34207 T195 oass.IndexSchema.readS

[...truncated too long message...]

irectory: ./solrtest-CloudSolrServerTest-1396125852407/org.apache.solr.client.solrj.impl.CloudSolrServerTest-1396125854029/jetty2/index
   [junit4]   2> 42011 T99 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./solrtest-CloudSolrServerTest-1396125852407/org.apache.solr.client.solrj.impl.CloudSolrServerTest-1396125854029/jetty2 [CachedDir<<refCount=0;path=./solrtest-CloudSolrServerTest-1396125852407/org.apache.solr.client.solrj.impl.CloudSolrServerTest-1396125854029/jetty2;done=false>>]
   [junit4]   2> 42011 T99 oasc.CachingDirectoryFactory.close Closing directory: ./solrtest-CloudSolrServerTest-1396125852407/org.apache.solr.client.solrj.impl.CloudSolrServerTest-1396125854029/jetty2
   [junit4]   2> 42011 T176 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 42036 T99 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
   [junit4]   2> 42088 T99 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 45443
   [junit4]   2> 42088 T99 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=23683737
   [junit4]   2> 42089 T99 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 42089 T99 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 42113 T99 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
   [junit4]   2> 42166 T99 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
   [junit4]   2> 42167 T99 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:46939 46939
   [junit4]   2> 42210 T113 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:46939 46939
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=CloudSolrServerTest -Dtests.method=testDistribSearch -Dtests.seed=F84D670205A65415 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=in -Dtests.timezone=America/Goose_Bay -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 17.7s J0 | CloudSolrServerTest.testDistribSearch <<<
   [junit4]    > Throwable #1: java.lang.AssertionError
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([F84D670205A65415:79ABE91A72F93429]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServerTest.doTest(CloudSolrServerTest.java:256)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:867)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 42250 T99 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 43330 T136 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 43427 T158 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 43515 T176 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 44434 T194 C14 P45443 oasc.SyncStrategy.sync WARN Closed, skipping sync up.
   [junit4]   2> 44437 T194 oasc.ShardLeaderElectionContext.rejoinLeaderElection Not rejoining election because CoreContainer is shutdown
   [junit4]   2> 44438 T194 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@1b6f095
   [junit4]   2> 44438 T194 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=5,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=2,cumulative_deletesById=1,cumulative_deletesByQuery=1,cumulative_errors=0,transaction_logs_total_size=354,transaction_logs_total_number=4}
   [junit4]   2> 44438 T194 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 44438 T194 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 44438 T194 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 44440 T194 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 44440 T194 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 44441 T194 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./solrtest-CloudSolrServerTest-1396125852407/org.apache.solr.client.solrj.impl.CloudSolrServerTest-1396125854029/jetty3/index [CachedDir<<refCount=0;path=./solrtest-CloudSolrServerTest-1396125852407/org.apache.solr.client.solrj.impl.CloudSolrServerTest-1396125854029/jetty3/index;done=false>>]
   [junit4]   2> 44441 T194 oasc.CachingDirectoryFactory.close Closing directory: ./solrtest-CloudSolrServerTest-1396125852407/org.apache.solr.client.solrj.impl.CloudSolrServerTest-1396125854029/jetty3/index
   [junit4]   2> 44441 T194 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./solrtest-CloudSolrServerTest-1396125852407/org.apache.solr.client.solrj.impl.CloudSolrServerTest-1396125854029/jetty3 [CachedDir<<refCount=0;path=./solrtest-CloudSolrServerTest-1396125852407/org.apache.solr.client.solrj.impl.CloudSolrServerTest-1396125854029/jetty3;done=false>>]
   [junit4]   2> 44441 T194 oasc.CachingDirectoryFactory.close Closing directory: ./solrtest-CloudSolrServerTest-1396125852407/org.apache.solr.client.solrj.impl.CloudSolrServerTest-1396125854029/jetty3
   [junit4]   2> 44441 T194 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 44442 T194 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 44442 T194 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:259)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> 44442 T194 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> NOTE: test params are: codec=Lucene46: {timestamp=TestBloomFilteredLucene41Postings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), range_facet_sl=FSTOrd41, multiDefault=Pulsing41(freqCutoff=12 minBlockSize=23 maxBlockSize=87), a_t=TestBloomFilteredLucene41Postings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), text=FSTOrd41, range_facet_si=Pulsing41(freqCutoff=12 minBlockSize=23 maxBlockSize=87), intDefault=TestBloomFilteredLucene41Postings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), id=PostingsFormat(name=FSTOrdPulsing41), _version_=TestBloomFilteredLucene41Postings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), range_facet_l=TestBloomFilteredLucene41Postings(BloomFilteringPostingsFormat(Lucene41(blocksize=128)))}, docValues:{}, sim=DefaultSimilarity, locale=in, timezone=America/Goose_Bay
   [junit4]   2> NOTE: Linux 3.8.0-37-generic i386/Oracle Corporation 1.7.0_51 (32-bit)/cpus=8,threads=1,free=132957616,total=209977344
   [junit4]   2> NOTE: All tests run in this JVM: [TestCoreAdmin, ClientUtilsTest, ModifiableSolrParamsTest, TestXMLEscaping, SolrExampleEmbeddedTest, TermsResponseTest, LargeVolumeEmbeddedTest, TestRetryUtil, JettyWebappTest, ContentStreamTest, SolrPingTest, TestUpdateRequestCodec, SolrExceptionTest, CloudSolrServerTest]
   [junit4] Completed on J0 in 22.38s, 2 tests, 1 failure <<< FAILURES!

[...truncated 91 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:202: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1275: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:907: There were test failures: 49 suites, 284 tests, 1 failure

Total time: 55 minutes 25 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_51 -server -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure