You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/02/25 18:13:43 UTC

[JENKINS] Lucene-Solr-4.x-Linux (64bit/jdk1.7.0_60-ea-b04) - Build # 9486 - Still Failing!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/9486/
Java: 64bit/jdk1.7.0_60-ea-b04 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC

2 tests failed.
REGRESSION:  org.apache.solr.client.solrj.SolrExampleXMLTest.testGetEmptyResults

Error Message:
IOException occured when talking to server at: https://127.0.0.1:38308/solr/collection1

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: https://127.0.0.1:38308/solr/collection1
	at __randomizedtesting.SeedInfo.seed([9FF96A0C461266DA:C4387B12729502D0]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:509)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:199)
	at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:117)
	at org.apache.solr.client.solrj.SolrServer.commit(SolrServer.java:168)
	at org.apache.solr.client.solrj.SolrServer.commit(SolrServer.java:146)
	at org.apache.solr.client.solrj.SolrExampleTests.testGetEmptyResults(SolrExampleTests.java:286)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.http.NoHttpResponseException: The target server failed to respond
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:260)
	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:271)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:682)
	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:486)
	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:395)
	... 45 more


REGRESSION:  org.apache.solr.client.solrj.impl.CloudSolrServerTest.testShutdown

Error Message:
java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:43677 within 30000 ms

Stack Trace:
org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:43677 within 30000 ms
	at __randomizedtesting.SeedInfo.seed([9FF96A0C461266DA:7C8F6399A168F3A8]:0)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:148)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:99)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:94)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:85)
	at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:89)
	at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:83)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.setUp(AbstractDistribZkTestBase.java:70)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.setUp(AbstractFullDistribZkTestBase.java:200)
	at org.apache.solr.client.solrj.impl.CloudSolrServerTest.setUp(CloudSolrServerTest.java:77)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:771)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)
Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:43677 within 30000 ms
	at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:222)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:140)
	... 47 more




Build Log:
[...truncated 11613 lines...]
   [junit4] Suite: org.apache.solr.client.solrj.impl.CloudSolrServerTest
   [junit4]   2> log4j:WARN No such property [conversionPattern] in org.apache.solr.util.SolrLogLayout.
   [junit4]   2> 1274 T11 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (true)
   [junit4]   2> 1308 T11 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 1311 T11 oasc.AbstractZkTestCase.<clinit> WARN TEST_HOME() does not exist - solrj test?
   [junit4]   2> 1337 T11 oas.SolrTestCaseJ4.setUp ###Starting testShutdown
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1393348071207
   [junit4]   2> 1341 T11 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 1344 T12 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1444 T11 oasc.ZkTestServer.run start zk server on port:43677
   [junit4]   2> 1486 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1510 T13 oazs.NIOServerCnxn.doIO WARN Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
   [junit4]   2> 69135 T15 oazsp.FileTxnLog.commit WARN fsync-ing the write ahead log in SyncThread:0 took 66489ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
   [junit4]   2> 69144 T13 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x1446a044f840000, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 69148 T13 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x1446a044f840001, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 69149 T13 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x1446a044f840002, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 69197 T11 oas.SolrTestCaseJ4.tearDown ###Ending testShutdown
   [junit4]   2> 69199 T11 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:43677 43677
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=CloudSolrServerTest -Dtests.method=testShutdown -Dtests.seed=9FF96A0C461266DA -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=es_MX -Dtests.timezone=Australia/West -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   68.1s J0 | CloudSolrServerTest.testShutdown <<<
   [junit4]    > Throwable #1: org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:43677 within 30000 ms
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([9FF96A0C461266DA:7C8F6399A168F3A8]:0)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:148)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:99)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:94)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:85)
   [junit4]    > 	at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:89)
   [junit4]    > 	at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:83)
   [junit4]    > 	at org.apache.solr.cloud.AbstractDistribZkTestBase.setUp(AbstractDistribZkTestBase.java:70)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.setUp(AbstractFullDistribZkTestBase.java:200)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServerTest.setUp(CloudSolrServerTest.java:77)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:744)
   [junit4]    > Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:43677 within 30000 ms
   [junit4]    > 	at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:222)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:140)
   [junit4]    > 	... 47 more
   [junit4]   2> 69451 T11 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1393348139322
   [junit4]   2> 69453 T11 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 69454 T19 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 69554 T11 oasc.ZkTestServer.run start zk server on port:43137
   [junit4]   2> 69556 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 69628 T25 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@40185808 name:ZooKeeperConnection Watcher:127.0.0.1:43137 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 69629 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 69631 T11 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 69679 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 69697 T27 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@48bc6410 name:ZooKeeperConnection Watcher:127.0.0.1:43137/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 69698 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 69710 T11 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 69722 T11 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 69732 T11 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 69740 T11 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 69759 T11 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/solrconfig.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 69780 T11 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 69794 T11 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 69795 T11 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 69817 T11 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml because it doesn't exist
   [junit4]   2> 69818 T11 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/stopwords.txt because it doesn't exist
   [junit4]   2> 69818 T11 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/protwords.txt because it doesn't exist
   [junit4]   2> 69819 T11 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/currency.xml because it doesn't exist
   [junit4]   2> 69820 T11 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/enumsConfig.xml because it doesn't exist
   [junit4]   2> 69820 T11 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/open-exchange-rates.json because it doesn't exist
   [junit4]   2> 69821 T11 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/mapping-ISOLatin1Accent.txt because it doesn't exist
   [junit4]   2> 69821 T11 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/old_synonyms.txt because it doesn't exist
   [junit4]   2> 69822 T11 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/synonyms.txt because it doesn't exist
   [junit4]   2> 69832 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 69836 T29 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@a52c93d name:ZooKeeperConnection Watcher:127.0.0.1:43137/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 69837 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 70104 T11 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 70467 T11 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 70488 T11 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:49720
   [junit4]   2> 70667 T11 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 70674 T11 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 70675 T11 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1393348139711
   [junit4]   2> 70676 T11 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1393348139711/'
   [junit4]   2> 70828 T11 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1393348139711/solr.xml
   [junit4]   2> 71016 T11 oasc.CoreContainer.<init> New CoreContainer 815342107
   [junit4]   2> 71017 T11 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1393348139711/]
   [junit4]   2> 71036 T11 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
   [junit4]   2> 71037 T11 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 71043 T11 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
   [junit4]   2> 71044 T11 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 71044 T11 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 71045 T11 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 71046 T11 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 71046 T11 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 71047 T11 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 71086 T11 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 71088 T11 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 71090 T11 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 71091 T11 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:43137/solr
   [junit4]   2> 71099 T11 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 71101 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 71117 T41 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@87d66d5 name:ZooKeeperConnection Watcher:127.0.0.1:43137 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 71118 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 71140 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 71157 T43 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@41fdd158 name:ZooKeeperConnection Watcher:127.0.0.1:43137/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 71158 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 71164 T11 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 71176 T11 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 71204 T11 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 71209 T11 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49720_
   [junit4]   2> 71212 T11 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49720_
   [junit4]   2> 71223 T11 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 71228 T11 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 71259 T11 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:49720_
   [junit4]   2> 71260 T11 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 71265 T11 oasc.Overseer.start Overseer (id=91314463659065348-127.0.0.1:49720_-n_0000000000) starting
   [junit4]   2> 71290 T11 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 71319 T45 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 71320 T11 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 71322 T45 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 71325 T11 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 71328 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 71336 T44 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 71434 T46 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 71435 T46 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 71439 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 71439 T46 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 71443 T44 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 71445 T44 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:49720",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:49720_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 71446 T44 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 71457 T44 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 71463 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 71468 T43 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 72440 T46 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 72441 T46 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1393348139711/collection1
   [junit4]   2> 72441 T46 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 72443 T46 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 72443 T46 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 72445 T46 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 72446 T46 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1393348139711/collection1/'
   [junit4]   2> 72536 T46 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
   [junit4]   2> 72662 T46 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 72673 T46 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 72782 T46 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 72935 T46 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [solr.SortableIntField]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 72942 T46 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [solr.SortableLongField]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 72950 T46 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [solr.SortableFloatField]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 72957 T46 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [solr.SortableDoubleField]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 73379 T46 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 73381 T46 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 73387 T46 oass.IndexSchema.readSchema WARN Field text is not multivalued and destination for multiple copyFields (3)
   [junit4]   2> 73388 T46 oass.IndexSchema.readSchema WARN Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
   [junit4]   2> 73427 T46 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 73436 T46 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1393348139711/collection1/, dataDir=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393348139321/control/data/
   [junit4]   2> 73438 T46 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 73464 T46 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393348139321/control/data
   [junit4]   2> 73464 T46 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393348139321/control/data/index/
   [junit4]   2> 73465 T46 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393348139321/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 73473 T46 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393348139321/control/data/index
   [junit4]   2> 73514 T46 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@31c4b348 lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393348139321/control/data/index),segFN=segments_1,generation=1}
   [junit4]   2> 73515 T46 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 73688 T46 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 73703 T46 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 73704 T46 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 73705 T46 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 73709 T46 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 73716 T46 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 73717 T46 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 73721 T46 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: solr.PingRequestHandler
   [junit4]   2> 73749 T46 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 73772 T46 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 73773 T46 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 73776 T46 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@31c4b348 lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393348139321/control/data/index),segFN=segments_1,generation=1}
   [junit4]   2> 73776 T46 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 73788 T46 oass.SolrIndexSearcher.<init> Opening Searcher@580f62a6[collection1] main
   [junit4]   2> 73821 T46 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 73821 T47 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@580f62a6[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 73825 T11 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0
   [junit4]   2> 73825 T50 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:49720 collection:control_collection shard:shard1
   [junit4]   2> 73825 T11 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 73832 T50 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 73845 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 73849 T52 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2dff2589 name:ZooKeeperConnection Watcher:127.0.0.1:43137/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 73850 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 73850 T50 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 73857 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 73858 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 73859 T50 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 73860 T50 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 73861 T44 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 73863 T11 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> ASYNC  NEW_CORE C0 name=collection1 org.apache.solr.core.SolrCore@2204d434 url=https://127.0.0.1:49720/collection1 node=127.0.0.1:49720_ C0_STATE=coll:control_collection core:collection1 props:{state=down, base_url=https://127.0.0.1:49720, core=collection1, node_name=127.0.0.1:49720_}
   [junit4]   2> 73863 T50 C0 P49720 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:49720/collection1/
   [junit4]   2> 73865 T50 C0 P49720 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 73866 T50 C0 P49720 oasc.SyncStrategy.syncToMe https://127.0.0.1:49720/collection1/ has no replicas
   [junit4]   2> 73866 T50 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:49720/collection1/ shard1
   [junit4]   2> 73867 T50 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 73869 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 73872 T43 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 73878 T52 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 73918 T11 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 73920 T11 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 73931 T11 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 73937 T11 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:39216
   [junit4]   2> 73940 T11 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 73941 T11 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 73941 T11 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1393348143732
   [junit4]   2> 73942 T11 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1393348143732/'
   [junit4]   2> 73959 T11 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1393348143732/solr.xml
   [junit4]   2> 73981 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 73984 T44 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 73994 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 74027 T11 oasc.CoreContainer.<init> New CoreContainer 2139452586
   [junit4]   2> 74028 T11 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1393348143732/]
   [junit4]   2> 74034 T11 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
   [junit4]   2> 74034 T11 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 74035 T11 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
   [junit4]   2> 74035 T11 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 74036 T11 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 74036 T11 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 74037 T11 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 74037 T11 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 74038 T11 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 74047 T11 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 74048 T11 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 74048 T11 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 74049 T11 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:43137/solr
   [junit4]   2> 74049 T11 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 74051 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 74053 T63 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@68a53237 name:ZooKeeperConnection Watcher:127.0.0.1:43137 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 74054 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 74059 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 74061 T65 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@37c22f06 name:ZooKeeperConnection Watcher:127.0.0.1:43137/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 74062 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 74070 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 74098 T43 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 74099 T65 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 74098 T52 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 74133 T50 oasc.ZkController.register We are https://127.0.0.1:49720/collection1/ and leader is https://127.0.0.1:49720/collection1/
   [junit4]   2> 74134 T50 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:49720
   [junit4]   2> 74134 T50 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 74135 T50 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 74135 T50 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 74138 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 74139 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 74139 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 74139 T50 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 74142 T44 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 74144 T44 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:49720",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:49720_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 74149 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 74253 T43 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 74253 T52 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 74253 T65 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 75076 T11 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39216_
   [junit4]   2> 75078 T11 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:39216_
   [junit4]   2> 75084 T52 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 75085 T43 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 75085 T65 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 75103 T66 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 75104 T66 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 75106 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 75106 T66 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 75107 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 75108 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 75110 T44 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 75112 T44 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:39216",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:39216_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 75113 T44 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 75114 T44 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 75120 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 75227 T43 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 75227 T65 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 75227 T52 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 76107 T66 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 76108 T66 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1393348143732/collection1
   [junit4]   2> 76108 T66 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 76110 T66 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 76110 T66 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 76112 T66 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 76112 T66 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1393348143732/collection1/'
   [junit4]   2> 76166 T66 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
   [junit4]   2> 76222 T66 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 76224 T66 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 76322 T66 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 76728 T66 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 76733 T66 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 76739 T66 oass.IndexSchema.readSchema WARN Field text is not multivalued and destination for multiple copyFields (3)
   [junit4]   2> 76740 T66 oass.IndexSchema.readSchema WARN Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
   [junit4]   2> 76743 T66 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 76744 T66 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1393348143732/collection1/, dataDir=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393348139321/jetty1/
   [junit4]   2> 76744 T66 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 76745 T66 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393348139321/jetty1
   [junit4]   2> 76746 T66 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393348139321/jetty1/index/
   [junit4]   2> 76747 T66 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393348139321/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 76747 T66 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393348139321/jetty1/index
   [junit4]   2> 76749 T66 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1ed408f0 lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393348139321/jetty1/index),segFN=segments_1,generation=1}
   [junit4]   2> 76750 T66 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 76751 T66 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 76752 T66 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 76752 T66 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 76753 T66 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 76753 T66 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 76753 T66 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 76754 T66 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 76755 T66 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: solr.PingRequestHandler
   [junit4]   2> 76758 T66 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 76761 T66 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 76761 T66 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 76763 T66 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1ed408f0 lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393348139321/jetty1/index),segFN=segments_1,generation=1}
   [junit4]   2> 76763 T66 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 76764 T66 oass.SolrIndexSearcher.<init> Opening Searcher@72b368f3[collection1] main
   [junit4]   2> 76766 T67 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@72b368f3[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 76766 T66 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 76767 T11 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0
   [junit4]   2> 76767 T70 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:39216 collection:collection1 shard:shard1
   [junit4]   2> 76768 T11 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 76770 T70 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 76787 T70 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 76794 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 76794 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 76795 T70 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 76795 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 76795 T70 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C1 name=collection1 org.apache.solr.core.SolrCore@6def748b url=https://127.0.0.1:39216/collection1 node=127.0.0.1:39216_ C1_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:39216, core=collection1, node_name=127.0.0.1:39216_}
   [junit4]   2> 76796 T70 C1 P39216 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:39216/collection1/
   [junit4]   2> 76798 T70 C1 P39216 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 76798 T44 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 76798 T70 C1 P39216 oasc.SyncStrategy.syncToMe https://127.0.0.1:39216/collection1/ has no replicas
   [junit4]   2> 76799 T70 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:39216/collection1/ shard1
   [junit4]   2> 76800 T70 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 76806 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 76809 T43 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 76809 T52 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 76811 T65 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 76816 T11 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 76817 T11 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 76824 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 76827 T44 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 76828 T11 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 76836 T11 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:32949
   [junit4]   2> 76838 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 76842 T11 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 76843 T11 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 76844 T11 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1393348146641
   [junit4]   2> 76844 T11 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1393348146641/'
   [junit4]   2> 76864 T11 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1393348146641/solr.xml
   [junit4]   2> 76911 T11 oasc.CoreContainer.<init> New CoreContainer 980497961
   [junit4]   2> 76912 T11 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1393348146641/]
   [junit4]   2> 76913 T11 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
   [junit4]   2> 76914 T11 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 76914 T11 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
   [junit4]   2> 76914 T11 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 76915 T11 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 76915 T11 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 76916 T11 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 76916 T11 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 76917 T11 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 76925 T11 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 76925 T11 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 76926 T11 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 76927 T11 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:43137/solr
   [junit4]   2> 76928 T11 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 76929 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 76932 T81 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2502835 name:ZooKeeperConnection Watcher:127.0.0.1:43137 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 76933 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 76936 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 76938 T83 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@32e95912 name:ZooKeeperConnection Watcher:127.0.0.1:43137/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 76939 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 76942 T65 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 76942 T52 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 76944 T43 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 76953 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 76975 T70 oasc.ZkController.register We are https://127.0.0.1:39216/collection1/ and leader is https://127.0.0.1:39216/collection1/
   [junit4]   2> 76975 T70 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:39216
   [junit4]   2> 76976 T70 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 76976 T70 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 76976 T70 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 76978 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 76978 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 76979 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 76979 T70 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 76980 T44 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 76982 T44 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:39216",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:39216_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 76986 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 77090 T52 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 77090 T83 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 77090 T43 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 77090 T65 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 77962 T11 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:32949_
   [junit4]   2> 77964 T11 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:32949_
   [junit4]   2> 77970 T43 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 77970 T83 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 77971 T52 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 77971 T65 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 77984 T84 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 77984 T84 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 77986 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 77986 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 77987 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 77987 T84 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 77989 T44 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 77992 T44 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:32949",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:32949_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 77992 T44 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 77993 T44 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 77998 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 78102 T65 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 78102 T83 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 78102 T52 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 78102 T43 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 78988 T84 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 78988 T84 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1393348146641/collection1
   [junit4]   2> 78989 T84 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 78990 T84 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 78990 T84 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 78993 T84 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 78993 T84 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1393348146641/collection1/'
   [junit4]   2> 79030 T84 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
   [junit4]   2> 79069 T84 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 79073 T84 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 79161 T84 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 79478 T84 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 79481 T84 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 79484 T84 oass.IndexSchema.readSchema WARN Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
   [junit4]   2> 79485 T84 oass.IndexSchema.readSchema WARN Field text is not multivalued and destination for multiple copyFields (3)
   [junit4]   2> 79486 T84 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 79487 T84 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1393348146641/collection1/, dataDir=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393348139321/jetty2/
   [junit4]   2> 79487 T84 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 79489 T84 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393348139321/jetty2
   [junit4]   2> 79489 T84 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393348139321/jetty2/index/
   [junit4]   2> 79489 T84 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393348139321/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 79490 T84 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393348139321/jetty2/index
   [junit4]   2> 79492 T84 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1a8c85c4 lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393348139321/jetty2/index),segFN=segments_1,generation=1}
   [junit4]   2> 79492 T84 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 79494 T84 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 79495 T84 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 79495 T84 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 79496 T84 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 79496 T84 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 79497 T84 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 79497 T84 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 79497 T84 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: solr.PingRequestHandler
   [junit4]   2> 79500 T84 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 79503 T84 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 79503 T84 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 79504 T84 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1a8c85c4 lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393348139321/jetty2/index),segFN=segments_1,generation=1}
   [junit4]   2> 79505 T84 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 79505 T84 oass.SolrIndexSearcher.<init> Opening Searcher@4668b788[collection1] main
   [junit4]   2> 79507 T85 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4668b788[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 79507 T84 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 79508 T11 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0
   [junit4]   2> 79508 T88 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:32949 collection:collection1 shard:shard2
   [junit4]   2> 79509 T11 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 79510 T88 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 79521 T88 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 79525 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 79526 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 79526 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 79527 T88 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 79527 T88 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C2 name=collection1 org.apache.solr.core.SolrCore@16ce3917 url=https://127.0.0.1:32949/collection1 node=127.0.0.1:32949_ C2_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:32949, core=collection1, node_name=127.0.0.1:32949_}
   [junit4]   2> 79528 T88 C2 P32949 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:32949/collection1/
   [junit4]   2> 79528 T88 C2 P32949 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 79529 T44 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 79529 T88 C2 P32949 oasc.SyncStrategy.syncToMe https://127.0.0.1:32949/collection1/ has no replicas
   [junit4]   2> 79530 T88 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:32949/collection1/ shard2
   [junit4]   2> 79530 T88 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 79536 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 79539 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 79540 T11 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 79541 T11 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 79546 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 79548 T11 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 79552 T11 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:58232
   [junit4]   2> 79556 T11 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 79556 T11 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 79557 T11 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1393348149380
   [junit4]   2> 79557 T11 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1393348149380/'
   [junit4]   2> 79580 T11 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1393348149380/solr.xml
   [junit4]   2> 79622 T11 oasc.CoreContainer.<init> New CoreContainer 289356862
   [junit4]   2> 79623 T11 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1393348149380/]
   [junit4]   2> 79624 T11 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
   [junit4]   2> 79625 T11 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 79626 T11 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
   [junit4]   2> 79626 T11 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 79627 T11 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 79627 T11 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 79627 T11 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 79628 T11 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 79628 T11 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 79638 T11 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 79639 T11 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 79639 T11 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 79640 T11 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:43137/solr
   [junit4]   2> 79641 T11 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 79642 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 79646 T99 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7f3b346b name:ZooKeeperConnection Watcher:127.0.0.1:43137 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 79647 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 79650 T65 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 79650 T43 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 79650 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 79652 T83 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 79650 T52 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 79655 T101 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5adb7fe1 name:ZooKeeperConnection Watcher:127.0.0.1:43137/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 79656 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 79668 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 79690 T88 oasc.ZkController.register We are https://127.0.0.1:32949/collection1/ and leader is https://127.0.0.1:32949/collection1/
   [junit4]   2> 79691 T88 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:32949
   [junit4]   2> 79691 T88 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 79692 T88 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 79692 T88 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 79694 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 79694 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 79695 T88 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 79695 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 79699 T44 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 79701 T44 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:32949",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:32949_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 79705 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 79810 T83 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 79810 T43 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 79810 T65 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 79810 T52 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 79810 T101 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 80673 T11 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:58232_
   [junit4]   2> 80675 T11 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:58232_
   [junit4]   2> 80679 T83 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 80679 T101 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 80680 T65 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 80679 T43 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 80679 T52 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 80692 T102 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 80693 T102 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 80694 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 80694 T102 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 80695 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 80695 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 80697 T44 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 80699 T44 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:58232",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:58232_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 80699 T44 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 80700 T44 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 80704 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 80808 T83 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 80808 T43 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 80808 T52 oascc.ZkStateReade

[...truncated too long message...]

unit4]   2> 336339 T295 C170 oasu.LoggingInfoStream.message [IFD][SUITE-SolrExampleXMLTest-seed#[9FF96A0C461266DA]-worker]: delete "_1a_Lucene41_0.tip"
   [junit4]   2> 336339 T295 C170 oasu.LoggingInfoStream.message [IFD][SUITE-SolrExampleXMLTest-seed#[9FF96A0C461266DA]-worker]: delete "_1a.nvd"
   [junit4]   2> 336339 T295 C170 oasu.LoggingInfoStream.message [IFD][SUITE-SolrExampleXMLTest-seed#[9FF96A0C461266DA]-worker]: delete "_1a_Lucene41_0.doc"
   [junit4]   2> 336340 T295 C170 oasu.LoggingInfoStream.message [IFD][SUITE-SolrExampleXMLTest-seed#[9FF96A0C461266DA]-worker]: delete "segments_1q"
   [junit4]   2> 336340 T295 C170 oasu.LoggingInfoStream.message [IFD][SUITE-SolrExampleXMLTest-seed#[9FF96A0C461266DA]-worker]: delete "_1a.fdx"
   [junit4]   2> 336340 T295 C170 oasu.LoggingInfoStream.message [IFD][SUITE-SolrExampleXMLTest-seed#[9FF96A0C461266DA]-worker]: delete "_1a_Lucene41_0.tim"
   [junit4]   2> 336340 T295 C170 oasu.LoggingInfoStream.message [IFD][SUITE-SolrExampleXMLTest-seed#[9FF96A0C461266DA]-worker]: delete "_1a.fnm"
   [junit4]   2> 336340 T295 C170 oasu.LoggingInfoStream.message [IFD][SUITE-SolrExampleXMLTest-seed#[9FF96A0C461266DA]-worker]: delete "_1a.nvm"
   [junit4]   2> 336341 T295 C170 oasu.LoggingInfoStream.message [IFD][SUITE-SolrExampleXMLTest-seed#[9FF96A0C461266DA]-worker]: 3 msec to checkpoint
   [junit4]   2> 336341 T295 C170 oasu.LoggingInfoStream.message [IW][SUITE-SolrExampleXMLTest-seed#[9FF96A0C461266DA]-worker]: commit: done
   [junit4]   2> 336341 T295 C170 oasu.LoggingInfoStream.message [IW][SUITE-SolrExampleXMLTest-seed#[9FF96A0C461266DA]-worker]: now flush at close waitForMerges=true
   [junit4]   2> 336342 T295 C170 oasu.LoggingInfoStream.message [IW][SUITE-SolrExampleXMLTest-seed#[9FF96A0C461266DA]-worker]:   start flush: applyAllDeletes=true
   [junit4]   2> 336342 T295 C170 oasu.LoggingInfoStream.message [IW][SUITE-SolrExampleXMLTest-seed#[9FF96A0C461266DA]-worker]:   index before flush 
   [junit4]   2> 336342 T295 C170 oasu.LoggingInfoStream.message [DW][SUITE-SolrExampleXMLTest-seed#[9FF96A0C461266DA]-worker]: SUITE-SolrExampleXMLTest-seed#[9FF96A0C461266DA]-worker startFullFlush
   [junit4]   2> 336342 T295 C170 oasu.LoggingInfoStream.message [DW][SUITE-SolrExampleXMLTest-seed#[9FF96A0C461266DA]-worker]: anyChanges? numDocsInRam=0 deletes=false hasTickets:false pendingChangesInFullFlush: false
   [junit4]   2> 336342 T295 C170 oasu.LoggingInfoStream.message [DW][SUITE-SolrExampleXMLTest-seed#[9FF96A0C461266DA]-worker]: SUITE-SolrExampleXMLTest-seed#[9FF96A0C461266DA]-worker finishFullFlush success=true
   [junit4]   2> 336343 T295 C170 oasu.LoggingInfoStream.message [IW][SUITE-SolrExampleXMLTest-seed#[9FF96A0C461266DA]-worker]: apply all deletes during flush
   [junit4]   2> 336343 T295 C170 oasu.LoggingInfoStream.message [BD][SUITE-SolrExampleXMLTest-seed#[9FF96A0C461266DA]-worker]: prune sis=org.apache.lucene.index.SegmentInfos@1b2bd92c minGen=9223372036854775807 packetCount=0
   [junit4]   2> 336343 T295 C170 oasu.LoggingInfoStream.message [CMS][SUITE-SolrExampleXMLTest-seed#[9FF96A0C461266DA]-worker]: now merge
   [junit4]   2> 336343 T295 C170 oasu.LoggingInfoStream.message [CMS][SUITE-SolrExampleXMLTest-seed#[9FF96A0C461266DA]-worker]:   index: 
   [junit4]   2> 336343 T295 C170 oasu.LoggingInfoStream.message [CMS][SUITE-SolrExampleXMLTest-seed#[9FF96A0C461266DA]-worker]:   no more merges pending; now return
   [junit4]   2> 336344 T295 C170 oasu.LoggingInfoStream.message [IW][SUITE-SolrExampleXMLTest-seed#[9FF96A0C461266DA]-worker]: waitForMerges
   [junit4]   2> 336344 T295 C170 oasu.LoggingInfoStream.message [IW][SUITE-SolrExampleXMLTest-seed#[9FF96A0C461266DA]-worker]: waitForMerges done
   [junit4]   2> 336344 T295 C170 oasu.LoggingInfoStream.message [IW][SUITE-SolrExampleXMLTest-seed#[9FF96A0C461266DA]-worker]: now call final commit()
   [junit4]   2> 336344 T295 C170 oasu.LoggingInfoStream.message [IW][SUITE-SolrExampleXMLTest-seed#[9FF96A0C461266DA]-worker]: commit: start
   [junit4]   2> 336345 T295 C170 oasu.LoggingInfoStream.message [IW][SUITE-SolrExampleXMLTest-seed#[9FF96A0C461266DA]-worker]: commit: enter lock
   [junit4]   2> 336345 T295 C170 oasu.LoggingInfoStream.message [IW][SUITE-SolrExampleXMLTest-seed#[9FF96A0C461266DA]-worker]: commit: now prepare
   [junit4]   2> 336346 T295 C170 oasu.LoggingInfoStream.message [IW][SUITE-SolrExampleXMLTest-seed#[9FF96A0C461266DA]-worker]: prepareCommit: flush
   [junit4]   2> 336346 T295 C170 oasu.LoggingInfoStream.message [IW][SUITE-SolrExampleXMLTest-seed#[9FF96A0C461266DA]-worker]:   index before flush 
   [junit4]   2> 336346 T295 C170 oasu.LoggingInfoStream.message [DW][SUITE-SolrExampleXMLTest-seed#[9FF96A0C461266DA]-worker]: SUITE-SolrExampleXMLTest-seed#[9FF96A0C461266DA]-worker startFullFlush
   [junit4]   2> 336347 T295 C170 oasu.LoggingInfoStream.message [DW][SUITE-SolrExampleXMLTest-seed#[9FF96A0C461266DA]-worker]: anyChanges? numDocsInRam=0 deletes=false hasTickets:false pendingChangesInFullFlush: false
   [junit4]   2> 336347 T295 C170 oasu.LoggingInfoStream.message [IW][SUITE-SolrExampleXMLTest-seed#[9FF96A0C461266DA]-worker]: apply all deletes during flush
   [junit4]   2> 336348 T295 C170 oasu.LoggingInfoStream.message [BD][SUITE-SolrExampleXMLTest-seed#[9FF96A0C461266DA]-worker]: prune sis=org.apache.lucene.index.SegmentInfos@1b2bd92c minGen=9223372036854775807 packetCount=0
   [junit4]   2> 336348 T295 C170 oasu.LoggingInfoStream.message [DW][SUITE-SolrExampleXMLTest-seed#[9FF96A0C461266DA]-worker]: SUITE-SolrExampleXMLTest-seed#[9FF96A0C461266DA]-worker finishFullFlush success=true
   [junit4]   2> 336348 T295 C170 oasu.LoggingInfoStream.message [IW][SUITE-SolrExampleXMLTest-seed#[9FF96A0C461266DA]-worker]: startCommit(): start
   [junit4]   2> 336348 T295 C170 oasu.LoggingInfoStream.message [IW][SUITE-SolrExampleXMLTest-seed#[9FF96A0C461266DA]-worker]:   skip startCommit(): no changes pending
   [junit4]   2> 336348 T295 C170 oasu.LoggingInfoStream.message [IW][SUITE-SolrExampleXMLTest-seed#[9FF96A0C461266DA]-worker]: commit: pendingCommit == null; skip
   [junit4]   2> 336349 T295 C170 oasu.LoggingInfoStream.message [IW][SUITE-SolrExampleXMLTest-seed#[9FF96A0C461266DA]-worker]: commit: done
   [junit4]   2> 336349 T295 C170 oasu.LoggingInfoStream.message [IW][SUITE-SolrExampleXMLTest-seed#[9FF96A0C461266DA]-worker]: at close: 
   [junit4]   2> 336350 T295 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 336350 T295 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 336350 T295 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J1/solrtest-SolrExampleXMLTest-1393348198616/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J1/solrtest-SolrExampleXMLTest-1393348198616/index;done=false>>]
   [junit4]   2> 336350 T295 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J1/solrtest-SolrExampleXMLTest-1393348198616/index
   [junit4]   2> 336351 T295 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J1/solrtest-SolrExampleXMLTest-1393348198616 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J1/solrtest-SolrExampleXMLTest-1393348198616;done=false>>]
   [junit4]   2> 336351 T295 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J1/solrtest-SolrExampleXMLTest-1393348198616
   [junit4]   2> 336351 T295 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
   [junit4]   2> 336403 T295 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: test params are: codec=Lucene42, sim=DefaultSimilarity, locale=sr_RS, timezone=America/Pangnirtung
   [junit4]   2> NOTE: Linux 3.8.0-36-generic amd64/Oracle Corporation 1.7.0_60-ea (64-bit)/cpus=8,threads=1,free=55794392,total=126418944
   [junit4]   2> NOTE: All tests run in this JVM: [TestSolrProperties, BasicHttpSolrServerTest, FieldAnalysisResponseTest, TestHash, MultiCoreExampleJettyTest, TestFastInputStream, IteratorChainTest, QueryResponseTest, MultiCoreEmbeddedTest, SolrQueryTest, TestNamedListCodec, TestLBHttpSolrServer, SolrExampleStreamingTest, SolrExampleXMLTest]
   [junit4] Completed on J1 in 207.61s, 26 tests, 1 error <<< FAILURES!

[...truncated 11 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:471: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:451: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:202: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:490: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1278: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:911: There were test failures: 48 suites, 280 tests, 2 errors

Total time: 50 minutes 11 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_60-ea-b04 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure