You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/03/30 10:25:52 UTC

[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0) - Build # 9945 - Failure!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/9945/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseSerialGC

1 tests failed.
FAILED:  org.apache.solr.client.solrj.impl.CloudSolrServerTest.testDistribSearch

Error Message:


Stack Trace:
java.lang.AssertionError
	at __randomizedtesting.SeedInfo.seed([6B1058D0602F7DCA:EAF6D6C817701DF6]:0)
	at org.junit.Assert.fail(Assert.java:92)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertTrue(Assert.java:54)
	at org.apache.solr.client.solrj.impl.CloudSolrServerTest.doTest(CloudSolrServerTest.java:256)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:867)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1617)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:826)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:862)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:876)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:783)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:443)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:835)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:771)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:782)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
	at java.lang.Thread.run(Thread.java:744)




Build Log:
[...truncated 12102 lines...]
   [junit4] Suite: org.apache.solr.client.solrj.impl.CloudSolrServerTest
   [junit4]   2> log4j:WARN No such property [conversionPattern] in org.apache.solr.util.SolrLogLayout.
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1396167849801
   [junit4]   2> 1070 T12 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (true)
   [junit4]   2> 1094 T12 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /o_xjf/
   [junit4]   2> 1101 T12 oasc.AbstractZkTestCase.<clinit> WARN TEST_HOME() does not exist - solrj test?
   [junit4]   2> 1137 T12 oas.SolrTestCaseJ4.setUp ###Starting testShutdown
   [junit4]   2> 1140 T12 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 1143 T13 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1242 T12 oasc.ZkTestServer.run start zk server on port:48400
   [junit4]   2> 1302 T12 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 20813 T16 oazsp.FileTxnLog.commit WARN fsync-ing the write ahead log in SyncThread:0 took 19474ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
   [junit4]   2> 20826 T14 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x1451216b8930000, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 20829 T19 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@74674e04 name:ZooKeeperConnection Watcher:127.0.0.1:48400 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 20830 T12 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 20832 T12 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 20867 T12 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 20883 T21 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@39bf85c8 name:ZooKeeperConnection Watcher:127.0.0.1:48400/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 20887 T12 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 20897 T12 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 20903 T12 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 20909 T12 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 20916 T12 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 20928 T12 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/solrconfig.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 20946 T12 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 20953 T12 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 20956 T12 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 20964 T12 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml because it doesn't exist
   [junit4]   2> 20965 T12 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/stopwords.txt because it doesn't exist
   [junit4]   2> 20966 T12 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/protwords.txt because it doesn't exist
   [junit4]   2> 20968 T12 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/currency.xml because it doesn't exist
   [junit4]   2> 20973 T12 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/enumsConfig.xml because it doesn't exist
   [junit4]   2> 20974 T12 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/open-exchange-rates.json because it doesn't exist
   [junit4]   2> 20975 T12 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/mapping-ISOLatin1Accent.txt because it doesn't exist
   [junit4]   2> 20975 T12 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/old_synonyms.txt because it doesn't exist
   [junit4]   2> 20976 T12 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/synonyms.txt because it doesn't exist
   [junit4]   2> 20987 T12 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 20992 T23 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@67b5c7f6 name:ZooKeeperConnection Watcher:127.0.0.1:48400/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 20992 T12 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 21333 T12 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 21335 T24 oaz.ClientCnxnSocketNIO.connect ERROR Unable to open socket to ff01::114/ff01:0:0:0:0:0:0:114:33332
   [junit4]   2> 21336 T24 oaz.ClientCnxn$SendThread.run WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.SocketException: Network is unreachable
   [junit4]   2> 	at sun.nio.ch.Net.connect0(Native Method)
   [junit4]   2> 	at sun.nio.ch.Net.connect(Net.java:435)
   [junit4]   2> 	at sun.nio.ch.Net.connect(Net.java:427)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:643)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:277)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:287)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:967)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
   [junit4]   2> 
   [junit4]   2> 22439 T24 oaz.ClientCnxnSocketNIO.connect ERROR Unable to open socket to ff01::114/ff01:0:0:0:0:0:0:114:33332
   [junit4]   2> 22613 T12 oas.SolrTestCaseJ4.tearDown ###Ending testShutdown
   [junit4]   2> 22616 T12 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:48400 48400
   [junit4]   2> 23101 T13 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:48400 48400
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 23142 T12 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 23143 T12 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 23146 T26 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 23246 T12 oasc.ZkTestServer.run start zk server on port:51581
   [junit4]   2> 23255 T12 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 23334 T32 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4ecd933d name:ZooKeeperConnection Watcher:127.0.0.1:51581 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 23335 T12 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 23336 T12 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 23355 T12 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 23373 T34 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@79f42143 name:ZooKeeperConnection Watcher:127.0.0.1:51581/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 23374 T12 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 23374 T12 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 23386 T12 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 23398 T12 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 23401 T12 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 23406 T12 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/solrconfig.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 23407 T12 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 23413 T12 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 23414 T12 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 23419 T12 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml because it doesn't exist
   [junit4]   2> 23422 T12 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/stopwords.txt because it doesn't exist
   [junit4]   2> 23423 T12 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/protwords.txt because it doesn't exist
   [junit4]   2> 23423 T12 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/currency.xml because it doesn't exist
   [junit4]   2> 23424 T12 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/enumsConfig.xml because it doesn't exist
   [junit4]   2> 23424 T12 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/open-exchange-rates.json because it doesn't exist
   [junit4]   2> 23425 T12 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/mapping-ISOLatin1Accent.txt because it doesn't exist
   [junit4]   2> 23426 T12 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/old_synonyms.txt because it doesn't exist
   [junit4]   2> 23426 T12 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/synonyms.txt because it doesn't exist
   [junit4]   2> 23428 T12 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 23431 T36 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d6ef7e8 name:ZooKeeperConnection Watcher:127.0.0.1:51581/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 23431 T12 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 23840 T12 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 24292 T12 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 24368 T12 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:55818
   [junit4]   2> 24402 T12 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 24408 T12 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 24408 T12 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1396167849801/org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1396167872364
   [junit4]   2> 24409 T12 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1396167849801/org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1396167872364/'
   [junit4]   2> 24591 T12 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1396167849801/org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1396167872364/solr.xml
   [junit4]   2> 24837 T12 oasc.CoreContainer.<init> New CoreContainer 1632214732
   [junit4]   2> 24838 T12 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1396167849801/org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1396167872364/]
   [junit4]   2> 24871 T12 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 24872 T12 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 24877 T12 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 24877 T12 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 24878 T12 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 24878 T12 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 24879 T12 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 24879 T12 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 24880 T12 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 24905 T12 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 24907 T12 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 24909 T12 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 24909 T12 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51581/solr
   [junit4]   2> 24918 T12 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 24920 T12 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 24925 T48 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6c30423b name:ZooKeeperConnection Watcher:127.0.0.1:51581 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 24925 T12 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 24931 T12 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 24933 T50 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4e873931 name:ZooKeeperConnection Watcher:127.0.0.1:51581/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 24934 T12 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 24940 T12 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 24947 T12 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 24953 T12 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 24957 T12 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 24961 T12 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 24976 T12 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 24979 T12 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55818_o_xjf
   [junit4]   2> 24982 T12 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:55818_o_xjf
   [junit4]   2> 24988 T12 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 24994 T12 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 25005 T12 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:55818_o_xjf
   [junit4]   2> 25006 T12 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 25008 T12 oasc.Overseer.start Overseer (id=91499257664503812-127.0.0.1:55818_o_xjf-n_0000000000) starting
   [junit4]   2> 25018 T12 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 25051 T52 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 25052 T12 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 25054 T52 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 25056 T12 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 25059 T12 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 25081 T51 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 25173 T53 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 25174 T53 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 25178 T50 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 25178 T53 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 25181 T51 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 25183 T51 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:55818/o_xjf",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:55818_o_xjf",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 25184 T51 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 25194 T51 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 25198 T50 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 25203 T50 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 26179 T53 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 26180 T53 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 26181 T53 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 26182 T53 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 26185 T53 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 26186 T53 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1396167849801/org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1396167872364/collection1/'
   [junit4]   2> 26242 T53 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 26313 T53 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 26327 T53 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 26405 T53 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 26509 T53 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [solr.SortableIntField]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 26515 T53 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [solr.SortableLongField]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 26521 T53 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [solr.SortableFloatField]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 26526 T53 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [solr.SortableDoubleField]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 26775 T53 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 26776 T53 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 26780 T53 oass.IndexSchema.readSchema WARN Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
   [junit4]   2> 26781 T53 oass.IndexSchema.readSchema WARN Field text is not multivalued and destination for multiple copyFields (3)
   [junit4]   2> 26797 T53 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 26822 T53 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 26831 T53 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1396167849801/org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1396167872364/collection1/, dataDir=./solrtest-CloudSolrServerTest-1396167849801/org.apache.solr.client.solrj.impl.CloudSolrServerTest-1396167872072/control/data/
   [junit4]   2> 26833 T53 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 26860 T53 oasc.CachingDirectoryFactory.get return new directory for ./solrtest-CloudSolrServerTest-1396167849801/org.apache.solr.client.solrj.impl.CloudSolrServerTest-1396167872072/control/data
   [junit4]   2> 26861 T53 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solrtest-CloudSolrServerTest-1396167849801/org.apache.solr.client.solrj.impl.CloudSolrServerTest-1396167872072/control/data/index/
   [junit4]   2> 26861 T53 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solrtest-CloudSolrServerTest-1396167849801/org.apache.solr.client.solrj.impl.CloudSolrServerTest-1396167872072/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 26875 T53 oasc.CachingDirectoryFactory.get return new directory for ./solrtest-CloudSolrServerTest-1396167849801/org.apache.solr.client.solrj.impl.CloudSolrServerTest-1396167872072/control/data/index
   [junit4]   2> 26915 T53 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@9024a6c lockFactory=NativeFSLockFactory@./solrtest-CloudSolrServerTest-1396167849801/org.apache.solr.client.solrj.impl.CloudSolrServerTest-1396167872072/control/data/index),segFN=segments_1,generation=1}
   [junit4]   2> 26916 T53 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 27062 T53 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 27075 T53 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 27075 T53 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 27077 T53 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 27081 T53 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 27087 T53 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 27089 T53 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 27093 T53 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: solr.PingRequestHandler
   [junit4]   2> 27119 T53 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 27151 T53 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 27152 T53 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 27154 T53 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@9024a6c lockFactory=NativeFSLockFactory@./solrtest-CloudSolrServerTest-1396167849801/org.apache.solr.client.solrj.impl.CloudSolrServerTest-1396167872072/control/data/index),segFN=segments_1,generation=1}
   [junit4]   2> 27155 T53 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 27178 T53 oass.SolrIndexSearcher.<init> Opening Searcher@695009b6[collection1] main
   [junit4]   2> 27192 T53 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 27194 T53 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 27194 T53 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 27197 T53 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 27197 T53 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 27216 T53 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 27217 T53 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 27218 T53 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 27218 T53 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 27219 T53 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 27242 T53 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 27243 T54 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@695009b6[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 27245 T57 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:55818/o_xjf collection:control_collection shard:shard1
   [junit4]   2> 27246 T12 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0
   [junit4]   2> 27246 T12 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 27252 T57 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 27263 T12 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 27266 T59 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@93f6513 name:ZooKeeperConnection Watcher:127.0.0.1:51581/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 27269 T12 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 27268 T57 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 27272 T12 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 27273 T50 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 27275 T57 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 27275 T57 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 27276 T51 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> ASYNC  NEW_CORE C0 name=collection1 org.apache.solr.core.SolrCore@6eaa5bea url=https://127.0.0.1:55818/o_xjf/collection1 node=127.0.0.1:55818_o_xjf C0_STATE=coll:control_collection core:collection1 props:{state=down, base_url=https://127.0.0.1:55818/o_xjf, core=collection1, node_name=127.0.0.1:55818_o_xjf}
   [junit4]   2> 27278 T57 C0 P55818 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:55818/o_xjf/collection1/
   [junit4]   2> 27279 T12 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 27280 T57 C0 P55818 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 27281 T57 C0 P55818 oasc.SyncStrategy.syncToMe https://127.0.0.1:55818/o_xjf/collection1/ has no replicas
   [junit4]   2> 27282 T57 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:55818/o_xjf/collection1/ shard1
   [junit4]   2> 27282 T50 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 27285 T50 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 27285 T57 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 27287 T59 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 27291 T50 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 27293 T51 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 27297 T50 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 27342 T12 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 27343 T12 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 27350 T12 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 27353 T12 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:53393
   [junit4]   2> 27355 T12 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 27356 T12 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 27356 T12 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solrtest-CloudSolrServerTest-1396167849801/org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1396167876210
   [junit4]   2> 27356 T12 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solrtest-CloudSolrServerTest-1396167849801/org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1396167876210/'
   [junit4]   2> 27377 T12 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1396167849801/org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1396167876210/solr.xml
   [junit4]   2> 27396 T12 oasc.CoreContainer.<init> New CoreContainer 427175319
   [junit4]   2> 27397 T12 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solrtest-CloudSolrServerTest-1396167849801/org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1396167876210/]
   [junit4]   2> 27398 T12 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 27398 T12 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 27399 T12 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 27399 T12 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 27400 T12 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 27400 T12 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 27401 T59 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 27401 T50 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 27401 T12 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 27405 T12 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 27405 T12 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 27409 T12 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 27410 T12 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 27410 T12 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 27411 T12 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51581/solr
   [junit4]   2> 27411 T12 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 27412 T12 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 27416 T70 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@26d41e1f name:ZooKeeperConnection Watcher:127.0.0.1:51581 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 27416 T12 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 27427 T12 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 27429 T72 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@57360751 name:ZooKeeperConnection Watcher:127.0.0.1:51581/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 27430 T12 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 27438 T12 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 27443 T57 oasc.ZkController.register We are https://127.0.0.1:55818/o_xjf/collection1/ and leader is https://127.0.0.1:55818/o_xjf/collection1/
   [junit4]   2> 27445 T57 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:55818/o_xjf
   [junit4]   2> 27445 T57 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 27447 T57 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 27448 T57 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 27450 T50 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 27450 T50 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 27450 T50 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 27452 T57 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 27453 T51 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 27454 T51 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:55818/o_xjf",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:55818_o_xjf",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 27458 T50 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 27560 T59 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 27560 T72 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 27560 T50 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 28442 T12 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53393_o_xjf
   [junit4]   2> 28444 T12 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53393_o_xjf
   [junit4]   2> 28446 T72 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 28446 T50 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 28446 T59 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 28454 T73 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 28454 T73 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 28456 T50 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 28457 T50 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 28457 T50 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 28459 T73 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 28459 T51 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 28460 T51 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:53393/o_xjf",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53393_o_xjf",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 28461 T51 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 28461 T51 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 28467 T50 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 28571 T72 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 28572 T50 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 28572 T59 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 29459 T73 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 29460 T73 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 29461 T73 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 29461 T73 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 29462 T73 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 29463 T73 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solrtest-CloudSolrServerTest-1396167849801/org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1396167876210/collection1/'
   [junit4]   2> 29490 T73 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 29501 T73 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 29503 T73 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 29542 T73 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 29704 T73 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 29706 T73 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 29708 T73 oass.IndexSchema.readSchema WARN Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
   [junit4]   2> 29712 T73 oass.IndexSchema.readSchema WARN Field text is not multivalued and destination for multiple copyFields (3)
   [junit4]   2> 29714 T73 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 29714 T73 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 29714 T73 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solrtest-CloudSolrServerTest-1396167849801/org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1396167876210/collection1/, dataDir=./solrtest-CloudSolrServerTest-1396167849801/org.apache.solr.client.solrj.impl.CloudSolrServerTest-1396167872072/jetty1/
   [junit4]   2> 29715 T73 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 29717 T73 oasc.CachingDirectoryFactory.get return new directory for ./solrtest-CloudSolrServerTest-1396167849801/org.apache.solr.client.solrj.impl.CloudSolrServerTest-1396167872072/jetty1
   [junit4]   2> 29717 T73 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solrtest-CloudSolrServerTest-1396167849801/org.apache.solr.client.solrj.impl.CloudSolrServerTest-1396167872072/jetty1/index/
   [junit4]   2> 29717 T73 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solrtest-CloudSolrServerTest-1396167849801/org.apache.solr.client.solrj.impl.CloudSolrServerTest-1396167872072/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 29718 T73 oasc.CachingDirectoryFactory.get return new directory for ./solrtest-CloudSolrServerTest-1396167849801/org.apache.solr.client.solrj.impl.CloudSolrServerTest-1396167872072/jetty1/index
   [junit4]   2> 29719 T73 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@135a6c51 lockFactory=NativeFSLockFactory@./solrtest-CloudSolrServerTest-1396167849801/org.apache.solr.client.solrj.impl.CloudSolrServerTest-1396167872072/jetty1/index),segFN=segments_1,generation=1}
   [junit4]   2> 29720 T73 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 29721 T73 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 29721 T73 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 29722 T73 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 29722 T73 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 29722 T73 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 29722 T73 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 29723 T73 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 29723 T73 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: solr.PingRequestHandler
   [junit4]   2> 29725 T73 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 29727 T73 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 29727 T73 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 29729 T73 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@135a6c51 lockFactory=NativeFSLockFactory@./solrtest-CloudSolrServerTest-1396167849801/org.apache.solr.client.solrj.impl.CloudSolrServerTest-1396167872072/jetty1/index),segFN=segments_1,generation=1}
   [junit4]   2> 29729 T73 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 29730 T73 oass.SolrIndexSearcher.<init> Opening Searcher@70e336bf[collection1] main
   [junit4]   2> 29730 T73 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 29731 T73 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 29732 T73 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 29733 T73 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 29733 T73 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 29734 T73 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 29735 T73 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 29736 T73 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 29736 T73 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 29736 T73 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 29738 T74 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@70e336bf[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 29738 T73 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 29741 T77 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:53393/o_xjf collection:collection1 shard:shard2
   [junit4]   2> 29741 T12 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0
   [junit4]   2> 29742 T12 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 29754 T77 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 29804 T77 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 29809 T50 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 29809 T50 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 29810 T50 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 29810 T77 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 29810 T77 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C1 name=collection1 org.apache.solr.core.SolrCore@6d67da08 url=https://127.0.0.1:53393/o_xjf/collection1 node=127.0.0.1:53393_o_xjf C1_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:53393/o_xjf, core=collection1, node_name=127.0.0.1:53393_o_xjf}
   [junit4]   2> 29811 T77 C1 P53393 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:53393/o_xjf/collection1/
   [junit4]   2> 29815 T77 C1 P53393 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 29816 T51 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 29816 T77 C1 P53393 oasc.SyncStrategy.syncToMe https://127.0.0.1:53393/o_xjf/collection1/ has no replicas
   [junit4]   2> 29816 T77 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:53393/o_xjf/collection1/ shard2
   [junit4]   2> 29817 T77 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 29825 T50 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 29830 T12 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 29831 T12 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 29833 T50 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 29849 T50 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 29851 T12 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 29854 T12 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:37506
   [junit4]   2> 29867 T12 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 29868 T12 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 29868 T12 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1396167849801/org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1396167878675
   [junit4]   2> 29869 T12 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1396167849801/org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1396167878675/'
   [junit4]   2> 29897 T12 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1396167849801/org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1396167878675/solr.xml
   [junit4]   2> 29924 T12 oasc.CoreContainer.<init> New CoreContainer 663944090
   [junit4]   2> 29925 T12 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1396167849801/org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1396167878675/]
   [junit4]   2> 29925 T12 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 29926 T12 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 29927 T12 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 29927 T12 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 29927 T12 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 29928 T12 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 29928 T12 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 29929 T12 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 29929 T12 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 29939 T12 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 29940 T12 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 29940 T12 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 29941 T12 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51581/solr
   [junit4]   2> 29941 T12 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 29947 T12 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 29953 T88 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4645f4aa name:ZooKeeperConnection Watcher:127.0.0.1:51581 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 29953 T50 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 29953 T59 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 29953 T72 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 29958 T12 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 29971 T12 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 29981 T90 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@28cf4558 name:ZooKeeperConnection Watcher:127.0.0.1:51581/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 29997 T12 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 29984 T77 oasc.ZkController.register We are https://127.0.0.1:53393/o_xjf/collection1/ and leader is https://127.0.0.1:53393/o_xjf/collection1/
   [junit4]   2> 29998 T77 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:53393/o_xjf
   [junit4]   2> 29999 T77 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 29999 T77 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 30000 T77 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 30001 T50 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 30001 T50 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 30001 T50 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 30002 T77 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 30003 T51 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 30004 T51 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:53393/o_xjf",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53393_o_xjf",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 30007 T50 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 30008 T12 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 30111 T59 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 30111 T50 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 30111 T72 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 30111 T90 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 31012 T12 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37506_o_xjf
   [junit4]   2> 31014 T12 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:37506_o_xjf
   [junit4]   2> 31017 T72 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 31017 T90 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 31017 T50 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 31018 T59 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 31026 T91 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 31027 T91 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 31028 T50 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 31028 T50 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 31029 T50 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 31030 T51 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 31031 T51 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:37506/o_xjf",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:37506_o_xjf",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 31032 T51 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 31032 T51 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 31035 T91 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 31039 T50 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 31142 T90 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 31142 T50 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 31142 T59 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 31143 T72 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 32038 T91 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 32039 T91 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 32040 T91 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 32040 T91 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 32041 T91 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 32041 T91 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1396167849801/org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1396167878675/collection1/'
   [junit4]   2> 32069 T91 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 32090 T91 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 32092 T91 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 32134 T91 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 32292 T91 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 32294 T91 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 32296 T91 oass.IndexSchema.readSchema WARN Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
   [junit4]   2> 32297 T91 oass.IndexSchema.readSchema WARN Field text is not multivalued and destination for multiple copyFields (3)
   [junit4]   2> 32299 T91 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 32299 T91 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 32300 T91 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1396167849801/org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1396167878675/collection1/, dataDir=./solrtest-CloudSolrServerTest-1396167849801/org.apache.solr.client.solrj.impl.CloudSolrServerTest-1396167872072/jetty2/
   [junit4]   2> 32300 T91 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 32301 T91 oasc.CachingDirectoryFactory.get return new directory for ./solrtest-CloudSolrServerTest-1396167849801/org.apache.solr.client.solrj.impl.CloudSolrServerTest-1396167872072/jetty2
   [junit4]   2> 32302 T91 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solrtest-CloudSolrServerTest-1396167849801/org.apache.solr.client.solrj.impl.CloudSolrServerTest-1396167872072/jetty2/index/
   [junit4]   2> 32302 T91 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solrtest-CloudSolrServerTest-1396167849801/org.apache.solr.client.solrj.impl.CloudSolrServerTest-1396167872072/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 32303 T91 oasc.CachingDirectoryFactory.get return new directory for ./solrtest-CloudSolrServerTest-1396167849801/org.apache.solr.client.solrj.impl.CloudSolrServerTest-1396167872072/jetty2/index
   [junit4]   2> 32305 T91 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@6c353399 lockFactory=NativeFSLockFactory@./solrtest-CloudSolrServerTest-1396167849801/org.apache.solr.client.solrj.impl.CloudSolrServerTest-1396167872072/jetty2/index),segFN=segments_1,generation=1}
   [junit4]   2> 32305 T91 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 32306 T91 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 32307 T91 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 32307 T91 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 32307 T91 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 32308 T91 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 32308 T91 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 32309 T91 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 32309 T91 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: solr.PingRequestHandler
   [junit4]   2> 32311 T91 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 32313 T91 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 32313 T91 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 32315 T91 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@6c353399 lockFactory=NativeFSLockFactory@./solrtest-CloudSolrServerTest-1396167849801/org.apache.solr.client.solrj.impl.CloudSolrServerTest-1396167872072/jetty2/index),segFN=segments_1,generation=1}
   [junit4]   2> 32316 T91 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 32316 T91 oass.SolrIndexSearcher.<init> Opening Searcher@2dc6bc9a[collection1] main
   [junit4]   2> 32316 T91 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 32318 T91 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 32318 T91 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 32319 T91 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 32319 T91 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 32320 T91 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 32321 T91 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 32321 T91 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 32322 T91 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 32322 T91 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 32323 T92 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2dc6bc9a[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 32323 T91 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 32324 T95 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:37506/o_xjf collection:collection1 shard:shard1
   [junit4]   2> 32325 T12 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0
   [junit4]   2> 32325 T12 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 32327 T95 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 32332 T95 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 32334 T50 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 32335 T50 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 32335 T50 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 32336 T95 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 32337 T51 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 32337 T95 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C2 name=collection1 org.apache.solr.core.SolrCore@1ffca5f5 url=https://127.0.0.1:37506/o_xjf/collection1 node=127.0.0.1:37506_o_xjf C2_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:37506/o_xjf, core=collection1, node_name=127.0.0.1:37506_o_xjf}
   [junit4]   2> 32338 T95 C2 P37506 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:37506/o_xjf/collection1/
   [junit4]   2> 32338 T95 C2 P37506 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 32338 T95 C2 P37506 oasc.SyncStrategy.syncToMe https://127.0.0.1:37506/o_xjf/collection1/ has no replicas
   [junit4]   2> 32339 T95 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:37506/o_xjf/collection1/ shard1
   [junit4]   2> 32339 T95 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 32353 T50 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 32357 T50 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 32406 T12 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 32409 T12 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 32420 T12 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 32424 T12 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:50632
   [junit4]   2> 32427 T12 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 32428 T12 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 32429 T12 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solrtest-CloudSolrServerTest-1396167849801/org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1396167881257
   [junit4]   2> 32429 T12 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solrtest-CloudSolrServerTest-1396167849801/org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1396167881257/'
   [junit4]   2> 32461 T59 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 32462 T72 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 32462 T90 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 32462 T50 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 32479 T12 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1396167849801/org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1396167881257/solr.xml
   [junit4]   2> 32498 T12 oasc.CoreContainer.<init> New CoreContainer 985731990
   [junit4]   2> 32498 T12 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solrtest-CloudSolrServerTest-1396167849801/org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1396167881257/]
   [junit4]   2> 32499 T12 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 32500 T12 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 32500 T12 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 32500 T12 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 32501 T12 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 32501 T12 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 32502 T12 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 32505 T95 oasc.ZkController.register We are https://127.0.0.1:37506/o_xjf/collection1/ and leader is https://127.0.0.1:37506/o_xjf/collection1/
   [junit4]   2> 32508 T95 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:37506/o_xjf
   [junit4]   2> 32508 T95 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 32508 T95 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 32509 T95 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 32507 T12 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 32510 T12 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 32510 T50 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 32511 T95 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 32511 T50 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 32511 T50 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 32513 T51 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 32514 T51 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:37506/o_xjf",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:37506_o_xjf",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 32515 T12 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 32516 T12 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 32516 T12 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 32516 T50 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 32517 T12 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51581/solr
   [junit4]   2> 32518 T12 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 32523 T12 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 32526 T106 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2e462d5b name:ZooKeeperConnection Watcher:127.0.0.1:51581 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 32526 T12 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 32531 T12 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 32537 T108 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@64cb7b05 name:ZooKeeperConnection Watcher:127.0.0.1:51581/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 32538 T12 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 32548 T12 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 32621 T59 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 32621 T108 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 32621 T72 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 32621 T50 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 32621 T90 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 33551 T12 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50632_o_xjf
   [junit4]   2> 33553 T12 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50632_o_xjf
   [junit4]   2> 33555 T59 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 33555 T90 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 33555 T50 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 33555 T108 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 33555 T72 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 33561 T109 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 33561 T109 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 33562 T50 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 33562 T109 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 33562 T50 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 33563 T50 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 33564 T51 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 33565 T51 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:50632/o_xjf",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50632_o_xjf",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 33565 T51 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 33565 T51 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 33568 T50 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 33670 T59 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 33670 T50 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 33670 T108 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 33670 T72 oascc.ZkSt

[...truncated too long message...]

 Closing directory: ./solrtest-CloudSolrServerTest-1396167849801/org.apache.solr.client.solrj.impl.CloudSolrServerTest-1396167872072/jetty2
   [junit4]   2> 42658 T12 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./solrtest-CloudSolrServerTest-1396167849801/org.apache.solr.client.solrj.impl.CloudSolrServerTest-1396167872072/jetty2/index [CachedDir<<refCount=0;path=./solrtest-CloudSolrServerTest-1396167849801/org.apache.solr.client.solrj.impl.CloudSolrServerTest-1396167872072/jetty2/index;done=false>>]
   [junit4]   2> 42658 T12 oasc.CachingDirectoryFactory.close Closing directory: ./solrtest-CloudSolrServerTest-1396167849801/org.apache.solr.client.solrj.impl.CloudSolrServerTest-1396167872072/jetty2/index
   [junit4]   2> 42659 T90 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 42687 T12 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/o_xjf,null}
   [junit4]   2> 42739 T12 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 50632
   [junit4]   2> 42739 T12 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=985731990
   [junit4]   2> 42740 T12 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 42740 T12 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 42765 T12 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/o_xjf,null}
   [junit4]   2> 42817 T12 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
   [junit4]   2> 42818 T12 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:51581 51581
   [junit4]   2> 42947 T26 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:51581 51581
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=CloudSolrServerTest -Dtests.method=testDistribSearch -Dtests.seed=6B1058D0602F7DCA -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=zh_HK -Dtests.timezone=Canada/Yukon -Dtests.file.encoding=ISO-8859-1
   [junit4] FAILURE 19.9s J0 | CloudSolrServerTest.testDistribSearch <<<
   [junit4]    > Throwable #1: java.lang.AssertionError
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([6B1058D0602F7DCA:EAF6D6C817701DF6]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServerTest.doTest(CloudSolrServerTest.java:256)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:867)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 43040 T12 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 43957 T50 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 43958 T50 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 44064 T72 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 44064 T72 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 44163 T90 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> ASYNC  NEW_CORE C13 name=collection1 org.apache.solr.core.SolrCore@915ced3 url=https://127.0.0.1:50632/o_xjf/collection1 node=127.0.0.1:50632_o_xjf C13_STATE=coll:collection1 core:collection1 props:{state=active, base_url=https://127.0.0.1:50632/o_xjf, core=collection1, node_name=127.0.0.1:50632_o_xjf}
   [junit4]   2> 45069 T108 C13 P50632 oasc.SyncStrategy.sync WARN Closed, skipping sync up.
   [junit4]   2> 45075 T108 oasc.ShardLeaderElectionContext.rejoinLeaderElection Not rejoining election because CoreContainer is shutdown
   [junit4]   2> 45076 T108 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@915ced3
   [junit4]   2> 45076 T108 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=5,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=2,cumulative_deletesById=1,cumulative_deletesByQuery=1,cumulative_errors=0,transaction_logs_total_size=354,transaction_logs_total_number=4}
   [junit4]   2> 45077 T108 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 45077 T108 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 45078 T108 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 45080 T108 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 45080 T108 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 45081 T108 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./solrtest-CloudSolrServerTest-1396167849801/org.apache.solr.client.solrj.impl.CloudSolrServerTest-1396167872072/jetty3/index [CachedDir<<refCount=0;path=./solrtest-CloudSolrServerTest-1396167849801/org.apache.solr.client.solrj.impl.CloudSolrServerTest-1396167872072/jetty3/index;done=false>>]
   [junit4]   2> 45081 T108 oasc.CachingDirectoryFactory.close Closing directory: ./solrtest-CloudSolrServerTest-1396167849801/org.apache.solr.client.solrj.impl.CloudSolrServerTest-1396167872072/jetty3/index
   [junit4]   2> 45081 T108 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./solrtest-CloudSolrServerTest-1396167849801/org.apache.solr.client.solrj.impl.CloudSolrServerTest-1396167872072/jetty3 [CachedDir<<refCount=0;path=./solrtest-CloudSolrServerTest-1396167849801/org.apache.solr.client.solrj.impl.CloudSolrServerTest-1396167872072/jetty3;done=false>>]
   [junit4]   2> 45081 T108 oasc.CachingDirectoryFactory.close Closing directory: ./solrtest-CloudSolrServerTest-1396167849801/org.apache.solr.client.solrj.impl.CloudSolrServerTest-1396167872072/jetty3
   [junit4]   2> 45082 T108 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 45082 T108 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 45082 T108 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:259)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> 45083 T108 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> NOTE: test params are: codec=Lucene46: {_version_=PostingsFormat(name=FSTPulsing41), multiDefault=MockFixedIntBlock(blockSize=835), a_t=MockFixedIntBlock(blockSize=835), intDefault=PostingsFormat(name=FSTPulsing41), range_facet_sl=FSTOrd41, range_facet_si=MockFixedIntBlock(blockSize=835), id=FSTOrd41, text=Lucene41(blocksize=128), range_facet_l=MockFixedIntBlock(blockSize=835), timestamp=MockFixedIntBlock(blockSize=835)}, docValues:{}, sim=DefaultSimilarity, locale=zh_HK, timezone=Canada/Yukon
   [junit4]   2> NOTE: Linux 3.8.0-37-generic amd64/Oracle Corporation 1.8.0 (64-bit)/cpus=8,threads=1,free=133682480,total=189792256
   [junit4]   2> NOTE: All tests run in this JVM: [CloudSolrServerTest]
   [junit4] Completed on J0 in 46.24s, 2 tests, 1 failure <<< FAILURES!

[...truncated 121 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:202: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1275: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:907: There were test failures: 49 suites, 284 tests, 1 failure

Total time: 54 minutes 46 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure