You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2014/02/01 09:59:06 UTC

[JENKINS] Lucene-Solr-NightlyTests-4.x - Build # 502 - Still Failing

Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-4.x/502/

2 tests failed.
FAILED:  org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testDistribSearch

Error Message:
Error executing query

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Error executing query
	at __randomizedtesting.SeedInfo.seed([5F00138BEE1F8861:DEE69D939940E85D]:0)
	at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:98)
	at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.compareResults(AbstractFullDistribZkTestBase.java:1254)
	at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testIndexingBatchPerRequestWithHttpSolrServer(FullSolrCloudDistribCmdsTest.java:413)
	at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.doTest(FullSolrCloudDistribCmdsTest.java:143)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:867)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:679)
Caused by: org.apache.http.ParseException: Invalid content type: 
	at org.apache.http.entity.ContentType.parse(ContentType.java:233)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:443)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:199)
	at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:90)
	... 45 more


FAILED:  org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload

Error Message:
expected:<[{indexVersion=1391244943152,generation=2,filelist=[_0.fdt, _0.fdx, _0.fnm, _0.frq, _0.nrm, _0.si, _0.tii, _0.tis, _1.fdt, _1.fdx, _1.fnm, _1.frq, _1.nrm, _1.si, _1.tii, _1.tis, _2.fdt, _2.fdx, _2.fnm, _2.frq, _2.nrm, _2.si, _2.tii, _2.tis, _3.fdt, _3.fdx, _3.fnm, _3.frq, _3.nrm, _3.si, _3.tii, _3.tis, _4.fdt, _4.fdx, _4.fnm, _4.frq, _4.nrm, _4.si, _4.tii, _4.tis, _5.fdt, _5.fdx, _5.fnm, _5.frq, _5.nrm, _5.si, _5.tii, _5.tis, segments_2]}]> but was:<[{indexVersion=1391244943152,generation=2,filelist=[_0.fdt, _0.fdx, _0.fnm, _0.frq, _0.nrm, _0.si, _0.tii, _0.tis, _1.fdt, _1.fdx, _1.fnm, _1.frq, _1.nrm, _1.si, _1.tii, _1.tis, _2.fdt, _2.fdx, _2.fnm, _2.frq, _2.nrm, _2.si, _2.tii, _2.tis, _3.fdt, _3.fdx, _3.fnm, _3.frq, _3.nrm, _3.si, _3.tii, _3.tis, _4.fdt, _4.fdx, _4.fnm, _4.frq, _4.nrm, _4.si, _4.tii, _4.tis, _5.fdt, _5.fdx, _5.fnm, _5.frq, _5.nrm, _5.si, _5.tii, _5.tis, segments_2]}, {indexVersion=1391244943152,generation=3,filelist=[_0.fdt, _0.fdx, _0.fnm, _0.frq, _0.nrm, _0.si, _0.tii, _0.tis, _2.fdt, _2.fdx, _2.fnm, _2.frq, _2.nrm, _2.si, _2.tii, _2.tis, _3.fdt, _3.fdx, _3.fnm, _3.frq, _3.nrm, _3.si, _3.tii, _3.tis, _6.cfe, _6.cfs, _6.si, segments_3]}]>

Stack Trace:
java.lang.AssertionError: expected:<[{indexVersion=1391244943152,generation=2,filelist=[_0.fdt, _0.fdx, _0.fnm, _0.frq, _0.nrm, _0.si, _0.tii, _0.tis, _1.fdt, _1.fdx, _1.fnm, _1.frq, _1.nrm, _1.si, _1.tii, _1.tis, _2.fdt, _2.fdx, _2.fnm, _2.frq, _2.nrm, _2.si, _2.tii, _2.tis, _3.fdt, _3.fdx, _3.fnm, _3.frq, _3.nrm, _3.si, _3.tii, _3.tis, _4.fdt, _4.fdx, _4.fnm, _4.frq, _4.nrm, _4.si, _4.tii, _4.tis, _5.fdt, _5.fdx, _5.fnm, _5.frq, _5.nrm, _5.si, _5.tii, _5.tis, segments_2]}]> but was:<[{indexVersion=1391244943152,generation=2,filelist=[_0.fdt, _0.fdx, _0.fnm, _0.frq, _0.nrm, _0.si, _0.tii, _0.tis, _1.fdt, _1.fdx, _1.fnm, _1.frq, _1.nrm, _1.si, _1.tii, _1.tis, _2.fdt, _2.fdx, _2.fnm, _2.frq, _2.nrm, _2.si, _2.tii, _2.tis, _3.fdt, _3.fdx, _3.fnm, _3.frq, _3.nrm, _3.si, _3.tii, _3.tis, _4.fdt, _4.fdx, _4.fnm, _4.frq, _4.nrm, _4.si, _4.tii, _4.tis, _5.fdt, _5.fdx, _5.fnm, _5.frq, _5.nrm, _5.si, _5.tii, _5.tis, segments_2]}, {indexVersion=1391244943152,generation=3,filelist=[_0.fdt, _0.fdx, _0.fnm, _0.frq, _0.nrm, _0.si, _0.tii, _0.tis, _2.fdt, _2.fdx, _2.fnm, _2.frq, _2.nrm, _2.si, _2.tii, _2.tis, _3.fdt, _3.fdx, _3.fnm, _3.frq, _3.nrm, _3.si, _3.tii, _3.tis, _6.cfe, _6.cfs, _6.si, segments_3]}]>
	at __randomizedtesting.SeedInfo.seed([5F00138BEE1F8861:7AD708BB9E578662]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:147)
	at org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload(TestReplicationHandler.java:1179)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:679)




Build Log:
[...truncated 9735 lines...]
   [junit4] Suite: org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
   [junit4]   2> 2380756 T5841 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 2380762 T5841 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./solrtest-FullSolrCloudDistribCmdsTest-1391244082938
   [junit4]   2> 2380763 T5841 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 2380764 T5842 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 2380865 T5841 oasc.ZkTestServer.run start zk server on port:29581
   [junit4]   2> 2380867 T5841 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2380907 T5848 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@eab2482 name:ZooKeeperConnection Watcher:127.0.0.1:29581 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2380908 T5841 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2380908 T5841 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 2380924 T5841 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2380926 T5850 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@31937fcc name:ZooKeeperConnection Watcher:127.0.0.1:29581/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2380926 T5841 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2380926 T5841 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 2380937 T5841 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 2380939 T5841 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 2380944 T5841 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 2380947 T5841 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 2380948 T5841 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 2380957 T5841 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 2380958 T5841 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 2381061 T5841 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2381062 T5841 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2381074 T5841 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 2381075 T5841 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 2381078 T5841 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 2381078 T5841 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 2381081 T5841 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 2381081 T5841 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 2381089 T5841 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 2381089 T5841 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 2381092 T5841 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2381093 T5841 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2381095 T5841 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2381096 T5841 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2381099 T5841 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 2381099 T5841 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 2381106 T5841 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 2381107 T5841 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 2381394 T5841 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2381399 T5841 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:29584
   [junit4]   2> 2381400 T5841 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2381401 T5841 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2381401 T5841 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1391244083286
   [junit4]   2> 2381402 T5841 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1391244083286/'
   [junit4]   2> 2381442 T5841 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1391244083286/solr.xml
   [junit4]   2> 2381521 T5841 oasc.CoreContainer.<init> New CoreContainer 748118626
   [junit4]   2> 2381522 T5841 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1391244083286/]
   [junit4]   2> 2381523 T5841 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2381524 T5841 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 2381524 T5841 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2381525 T5841 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2381525 T5841 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2381526 T5841 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2381526 T5841 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2381527 T5841 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2381527 T5841 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2381539 T5841 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2381539 T5841 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2381540 T5841 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2381540 T5841 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:29581/solr
   [junit4]   2> 2381541 T5841 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2381542 T5841 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2381544 T5861 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6bc26cd name:ZooKeeperConnection Watcher:127.0.0.1:29581 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2381546 T5841 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2381548 T5841 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2381557 T5863 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5880067 name:ZooKeeperConnection Watcher:127.0.0.1:29581/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2381557 T5841 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2381559 T5841 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 2381570 T5841 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 2381573 T5841 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 2381576 T5841 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:29584_
   [junit4]   2> 2381583 T5841 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:29584_
   [junit4]   2> 2381586 T5841 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 2381589 T5841 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 2381600 T5841 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:29584_
   [junit4]   2> 2381601 T5841 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 2381603 T5841 oasc.Overseer.start Overseer (id=91176572219949059-127.0.0.1:29584_-n_0000000000) starting
   [junit4]   2> 2381614 T5841 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 2381625 T5865 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 2381626 T5841 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 2381626 T5865 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 2381628 T5841 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 2381636 T5841 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2381639 T5864 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 2381645 T5866 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 2381645 T5866 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2381646 T5863 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2381646 T5866 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2381647 T5864 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2381648 T5864 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:29584",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:29584_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 2381648 T5864 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 2381649 T5864 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 2381651 T5863 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2381660 T5863 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2382648 T5866 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 2382648 T5866 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1391244083286/collection1
   [junit4]   2> 2382648 T5866 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 2382649 T5866 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2382649 T5866 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 2382650 T5866 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1391244083286/collection1/'
   [junit4]   2> 2382652 T5866 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1391244083286/collection1/lib/README' to classloader
   [junit4]   2> 2382652 T5866 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1391244083286/collection1/lib/classes/' to classloader
   [junit4]   2> 2382717 T5866 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 2382786 T5866 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2382888 T5866 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2382898 T5866 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 2383479 T5866 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2383483 T5866 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2383486 T5866 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2383491 T5866 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2383512 T5866 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 2383512 T5866 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1391244083286/collection1/, dataDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1391244082937/control/data/
   [junit4]   2> 2383513 T5866 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3b961a84
   [junit4]   2> 2383513 T5866 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1391244082937/control/data
   [junit4]   2> 2383514 T5866 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1391244082937/control/data/index/
   [junit4]   2> 2383514 T5866 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1391244082937/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 2383514 T5866 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1391244082937/control/data/index
   [junit4]   2> 2383515 T5866 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=47, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 2383516 T5866 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@58217f7e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@5bb75053),segFN=segments_1,generation=1}
   [junit4]   2> 2383516 T5866 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2383519 T5866 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 2383519 T5866 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 2383520 T5866 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 2383520 T5866 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2383520 T5866 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2383520 T5866 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 2383521 T5866 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2383521 T5866 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2383522 T5866 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2383522 T5866 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 2383523 T5866 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 2383523 T5866 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 2383523 T5866 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 2383524 T5866 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 2383524 T5866 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 2383525 T5866 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 2383534 T5866 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2383540 T5866 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 2383541 T5866 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 2383541 T5866 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=33, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=1.89453125, noCFSRatio=0.0]
   [junit4]   2> 2383542 T5866 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@58217f7e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@5bb75053),segFN=segments_1,generation=1}
   [junit4]   2> 2383543 T5866 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2383543 T5866 oass.SolrIndexSearcher.<init> Opening Searcher@2620d2f6 main
   [junit4]   2> 2383545 T5867 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2620d2f6 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 2383547 T5866 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 2383548 T5841 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0
   [junit4]   2> 2383548 T5841 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2383548 T5870 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:29584 collection:control_collection shard:shard1
   [junit4]   2> 2383550 T5841 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2383550 T5870 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 2383552 T5872 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@66829dea name:ZooKeeperConnection Watcher:127.0.0.1:29581/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2383552 T5841 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2383571 T5841 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2383586 T5841 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 2383587 T5870 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 2383595 T5863 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2383595 T5870 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 2383596 T5870 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C4410 name=collection1 org.apache.solr.core.SolrCore@154d5417 url=http://127.0.0.1:29584/collection1 node=127.0.0.1:29584_ C4410_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:29584, core=collection1, node_name=127.0.0.1:29584_}
   [junit4]   2> 2383596 T5870 C4410 P29584 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:29584/collection1/
   [junit4]   2> 2383596 T5864 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2383596 T5870 C4410 P29584 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 2383597 T5870 C4410 P29584 oasc.SyncStrategy.syncToMe http://127.0.0.1:29584/collection1/ has no replicas
   [junit4]   2> 2383597 T5870 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:29584/collection1/ shard1
   [junit4]   2> 2383597 T5870 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 2383605 T5863 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2383613 T5872 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2383613 T5863 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2383630 T5864 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2383633 T5863 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2383736 T5863 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2383736 T5872 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2383783 T5870 oasc.ZkController.register We are http://127.0.0.1:29584/collection1/ and leader is http://127.0.0.1:29584/collection1/
   [junit4]   2> 2383783 T5870 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:29584
   [junit4]   2> 2383783 T5870 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 2383784 T5870 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 2383784 T5870 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2383785 T5863 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2383786 T5863 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2383786 T5863 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2383786 T5870 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2383787 T5864 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2383788 T5864 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:29584",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:29584_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 2383791 T5863 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2383874 T5841 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 2383875 T5841 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2383879 T5841 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:29588
   [junit4]   2> 2383880 T5841 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2383880 T5841 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2383881 T5841 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1391244085761
   [junit4]   2> 2383881 T5841 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1391244085761/'
   [junit4]   2> 2383894 T5872 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2383894 T5863 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2383921 T5841 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1391244085761/solr.xml
   [junit4]   2> 2384010 T5841 oasc.CoreContainer.<init> New CoreContainer 1671840285
   [junit4]   2> 2384010 T5841 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1391244085761/]
   [junit4]   2> 2384012 T5841 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2384013 T5841 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 2384013 T5841 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2384014 T5841 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2384014 T5841 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2384015 T5841 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2384015 T5841 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2384016 T5841 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2384016 T5841 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2384027 T5841 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2384028 T5841 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2384029 T5841 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2384029 T5841 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:29581/solr
   [junit4]   2> 2384030 T5841 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2384031 T5841 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2384034 T5883 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@383eb107 name:ZooKeeperConnection Watcher:127.0.0.1:29581 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2384035 T5841 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2384038 T5841 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2384045 T5885 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@44c2988e name:ZooKeeperConnection Watcher:127.0.0.1:29581/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2384046 T5841 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2384050 T5841 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2385053 T5841 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:29588_
   [junit4]   2> 2385055 T5841 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:29588_
   [junit4]   2> 2385058 T5872 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 2385058 T5863 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 2385058 T5885 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 2385080 T5886 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 2385081 T5886 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2385082 T5863 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2385082 T5886 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2385082 T5863 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2385083 T5863 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2385084 T5864 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2385084 T5864 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:29588",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:29588_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 2385085 T5864 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
   [junit4]   2> 2385085 T5864 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 2385088 T5863 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2385194 T5872 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2385194 T5863 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2385194 T5885 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2386083 T5886 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 2386083 T5886 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1391244085761/collection1
   [junit4]   2> 2386083 T5886 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 2386084 T5886 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2386084 T5886 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 2386085 T5886 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1391244085761/collection1/'
   [junit4]   2> 2386087 T5886 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1391244085761/collection1/lib/README' to classloader
   [junit4]   2> 2386087 T5886 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1391244085761/collection1/lib/classes/' to classloader
   [junit4]   2> 2386152 T5886 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 2386220 T5886 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2386322 T5886 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2386332 T5886 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 2386909 T5886 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2386914 T5886 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2386916 T5886 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2386921 T5886 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2386942 T5886 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 2386942 T5886 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1391244085761/collection1/, dataDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1391244082937/jetty1/
   [junit4]   2> 2386942 T5886 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3b961a84
   [junit4]   2> 2386943 T5886 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1391244082937/jetty1
   [junit4]   2> 2386944 T5886 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1391244082937/jetty1/index/
   [junit4]   2> 2386944 T5886 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1391244082937/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 2386945 T5886 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1391244082937/jetty1/index
   [junit4]   2> 2386946 T5886 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=47, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 2386947 T5886 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@72a0b344 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1ee2a7a2),segFN=segments_1,generation=1}
   [junit4]   2> 2386947 T5886 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2386950 T5886 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 2386950 T5886 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 2386951 T5886 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 2386951 T5886 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2386951 T5886 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2386952 T5886 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 2386952 T5886 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2386953 T5886 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2386953 T5886 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2386954 T5886 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 2386954 T5886 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 2386954 T5886 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 2386955 T5886 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 2386955 T5886 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 2386956 T5886 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 2386956 T5886 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 2386966 T5886 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2386971 T5886 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 2386972 T5886 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 2386973 T5886 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=33, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=1.89453125, noCFSRatio=0.0]
   [junit4]   2> 2386973 T5886 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@72a0b344 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1ee2a7a2),segFN=segments_1,generation=1}
   [junit4]   2> 2386974 T5886 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2386974 T5886 oass.SolrIndexSearcher.<init> Opening Searcher@73aad76 main
   [junit4]   2> 2386978 T5887 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@73aad76 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 2386980 T5886 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 2386981 T5841 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0
   [junit4]   2> 2386981 T5841 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2386981 T5890 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:29588 collection:collection1 shard:shard1
   [junit4]   2> 2386983 T5890 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 2387025 T5890 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 2387035 T5863 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2387035 T5863 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2387035 T5890 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 2387035 T5863 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2387036 T5890 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C4411 name=collection1 org.apache.solr.core.SolrCore@59715e10 url=http://127.0.0.1:29588/collection1 node=127.0.0.1:29588_ C4411_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:29588, core=collection1, node_name=127.0.0.1:29588_}
   [junit4]   2> 2387036 T5890 C4411 P29588 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:29588/collection1/
   [junit4]   2> 2387037 T5890 C4411 P29588 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 2387037 T5890 C4411 P29588 oasc.SyncStrategy.syncToMe http://127.0.0.1:29588/collection1/ has no replicas
   [junit4]   2> 2387037 T5864 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2387037 T5890 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:29588/collection1/ shard1
   [junit4]   2> 2387038 T5890 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 2387041 T5863 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2387049 T5863 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2387049 T5885 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2387049 T5872 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2387053 T5863 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2387054 T5864 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2387057 T5863 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2387160 T5863 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2387160 T5872 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2387160 T5885 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2387206 T5890 oasc.ZkController.register We are http://127.0.0.1:29588/collection1/ and leader is http://127.0.0.1:29588/collection1/
   [junit4]   2> 2387206 T5890 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:29588
   [junit4]   2> 2387206 T5890 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 2387206 T5890 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 2387207 T5890 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2387208 T5863 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2387208 T5863 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2387209 T5890 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2387209 T5863 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2387210 T5864 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2387210 T5864 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:29588",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:29588_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 2387219 T5863 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2387264 T5841 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 2387265 T5841 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2387269 T5841 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:12596
   [junit4]   2> 2387270 T5841 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2387270 T5841 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2387271 T5841 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1391244089157
   [junit4]   2> 2387271 T5841 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1391244089157/'
   [junit4]   2> 2387312 T5841 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1391244089157/solr.xml
   [junit4]   2> 2387321 T5863 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2387321 T5885 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2387321 T5872 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2387391 T5841 oasc.CoreContainer.<init> New CoreContainer 686798752
   [junit4]   2> 2387392 T5841 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1391244089157/]
   [junit4]   2> 2387394 T5841 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2387395 T5841 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 2387395 T5841 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2387396 T5841 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2387396 T5841 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2387397 T5841 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2387397 T5841 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2387397 T5841 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2387398 T5841 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2387409 T5841 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2387410 T5841 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2387411 T5841 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2387411 T5841 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:29581/solr
   [junit4]   2> 2387412 T5841 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2387413 T5841 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2387415 T5901 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@12007e5f name:ZooKeeperConnection Watcher:127.0.0.1:29581 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2387416 T5841 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2387419 T5841 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2387426 T5903 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@37bc0c5a name:ZooKeeperConnection Watcher:127.0.0.1:29581/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2387427 T5841 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2387431 T5841 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2388434 T5841 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:12596_
   [junit4]   2> 2388436 T5841 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:12596_
   [junit4]   2> 2388439 T5872 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 2388439 T5863 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 2388439 T5903 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 2388439 T5885 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 2388458 T5904 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 2388458 T5904 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2388460 T5863 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2388460 T5904 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2388460 T5863 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2388460 T5863 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2388461 T5864 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2388462 T5864 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:12596",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:12596_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 2388462 T5864 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 2388463 T5864 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 2388465 T5863 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2388567 T5863 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2388567 T5885 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2388567 T5872 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2388567 T5903 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2389461 T5904 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 2389461 T5904 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1391244089157/collection1
   [junit4]   2> 2389461 T5904 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 2389462 T5904 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2389462 T5904 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 2389463 T5904 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1391244089157/collection1/'
   [junit4]   2> 2389465 T5904 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1391244089157/collection1/lib/classes/' to classloader
   [junit4]   2> 2389466 T5904 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1391244089157/collection1/lib/README' to classloader
   [junit4]   2> 2389531 T5904 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 2389601 T5904 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2389702 T5904 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2389722 T5904 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 2390317 T5904 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2390322 T5904 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2390324 T5904 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2390329 T5904 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2390349 T5904 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 2390350 T5904 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1391244089157/collection1/, dataDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1391244082937/jetty2/
   [junit4]   2> 2390350 T5904 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3b961a84
   [junit4]   2> 2390351 T5904 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1391244082937/jetty2
   [junit4]   2> 2390351 T5904 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1391244082937/jetty2/index/
   [junit4]   2> 2390352 T5904 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1391244082937/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 2390353 T5904 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1391244082937/jetty2/index
   [junit4]   2> 2390353 T5904 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=47, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 2390354 T5904 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@42358cad lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@3b9aa61e),segFN=segments_1,generation=1}
   [junit4]   2> 2390355 T5904 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2390358 T5904 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 2390358 T5904 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 2390359 T5904 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 2390359 T5904 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2390359 T5904 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2390359 T5904 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 2390360 T5904 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2390360 T5904 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2390361 T5904 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2390361 T5904 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 2390362 T5904 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 2390362 T5904 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 2390363 T5904 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 2390363 T5904 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 2390364 T5904 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 2390364 T5904 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 2390374 T5904 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2390380 T5904 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 2390380 T5904 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 2390381 T5904 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=33, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=1.89453125, noCFSRatio=0.0]
   [junit4]   2> 2390381 T5904 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@42358cad lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@3b9aa61e),segFN=segments_1,generation=1}
   [junit4]   2> 2390382 T5904 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2390382 T5904 oass.SolrIndexSearcher.<init> Opening Searcher@73126fe4 main
   [junit4]   2> 2390386 T5905 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@73126fe4 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 2390388 T5904 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 2390388 T5841 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0
   [junit4]   2> 2390389 T5841 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2390388 T5908 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:12596 collection:collection1 shard:shard2
   [junit4]   2> 2390391 T5908 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 2390424 T5908 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 2390426 T5863 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2390427 T5908 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 2390427 T5908 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 2390427 T5863 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> ASYNC  NEW_CORE C4412 name=collection1 org.apache.solr.core.SolrCore@2770fd90 url=http://127.0.0.1:12596/collection1 node=127.0.0.1:12596_ C4412_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:12596, core=collection1, node_name=127.0.0.1:12596_}
   [junit4]   2> 2390427 T5908 C4412 P12596 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:12596/collection1/
   [junit4]   2> 2390428 T5863 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2390428 T5908 C4412 P12596 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 2390429 T5908 C4412 P12596 oasc.SyncStrategy.syncToMe http://127.0.0.1:12596/collection1/ has no replicas
   [junit4]   2> 2390429 T5908 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:12596/collection1/ shard2
   [junit4]   2> 2390430 T5908 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 2390431 T5864 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2390440 T5863 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2390448 T5872 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2390448 T5885 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2390448 T5903 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2390448 T5863 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2390452 T5864 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2390460 T5863 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2390563 T5863 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2390563 T5885 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2390563 T5872 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2390563 T5903 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2390603 T5908 oasc.ZkController.register We are http://127.0.0.1:12596/collection1/ and leader is http://127.0.0.1:12596/collection1/
   [junit4]   2> 2390603 T5908 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:12596
   [junit4]   2> 2390603 T5908 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 2390603 T5908 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 2390604 T5908 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2390605 T5863 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2390605 T5863 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2390606 T5908 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2390606 T5863 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2390607 T5864 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2390608 T5864 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:12596",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:12596_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 2390616 T5863 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2390675 T5841 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 2390676 T5841 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2390679 T5841 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:16350
   [junit4]   2> 2390680 T5841 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2390681 T5841 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2390681 T5841 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1391244092565
   [junit4]   2> 2390682 T5841 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1391244092565/'
   [junit4]   2> 2390719 T5863 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2390719 T5885 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2390719 T5903 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2390719 T5872 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2390721 T5841 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1391244092565/solr.xml
   [junit4]   2> 2390800 T5841 oasc.CoreContainer.<init> New CoreContainer 1204068270
   [junit4]   2> 2390801 T5841 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1391244092565/]
   [junit4]   2> 2390803 T5841 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2390803 T5841 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 2390804 T5841 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2390804 T5841 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2390805 T5841 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2390805 T5841 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2390806 T5841 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2390806 T5841 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2390806 T5841 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2390818 T5841 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2390818 T5841 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2390819 T5841 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2390819 T5841 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:29581/solr
   [junit4]   2> 2390820 T5841 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2390821 T5841 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2390823 T5919 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3dcf7d51 name:ZooKeeperConnection Watcher:127.0.0.1:29581 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2390825 T5841 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2390828 T5841 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2390829 T5921 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@192f8220 name:ZooKeeperConnection Watcher:127.0.0.1:29581/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2390830 T5841 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2390839 T5841 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2391843 T584

[...truncated too long message...]

988095/repeater/collection1/data
   [junit4]   2> 3331604 T6334 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
   [junit4]   2> 3331661 T6334 oas.SolrTestCaseJ4.tearDown ###Ending doTestRepeater
   [junit4]   2> 3331662 T6334 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1495207593
   [junit4]   2> 3331664 T6334 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@3e24502f
   [junit4]   2> 3331664 T6334 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=2,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=6,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
   [junit4]   2> 3331665 T6334 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 3331665 T6334 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 3331666 T6334 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 3331668 T6334 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 3331668 T6334 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 3331669 T6334 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1391244987082/master/collection1/data [CachedDir<<refCount=0;path=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1391244987082/master/collection1/data;done=false>>]
   [junit4]   2> 3331669 T6334 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1391244987082/master/collection1/data
   [junit4]   2> 3331670 T6334 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1391244987082/master/collection1/data/index [CachedDir<<refCount=0;path=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1391244987082/master/collection1/data/index;done=false>>]
   [junit4]   2> 3331670 T6334 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1391244987082/master/collection1/data/index
   [junit4]   2> 3331671 T6334 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
   [junit4]   2> 3331724 T6334 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=880882966
   [junit4]   2> 3331725 T6334 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@6e803071
   [junit4]   2> 3331726 T6334 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
   [junit4]   2> 3331726 T6334 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 3331727 T6334 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 3331727 T6334 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 3331729 T6334 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 3331729 T6334 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 3331730 T6334 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1391244987475/slave/collection1/data/index [CachedDir<<refCount=0;path=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1391244987475/slave/collection1/data/index;done=false>>]
   [junit4]   2> 3331730 T6334 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1391244987475/slave/collection1/data/index
   [junit4]   2> 3331731 T6334 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1391244987475/slave/collection1/data [CachedDir<<refCount=0;path=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1391244987475/slave/collection1/data;done=false>>]
   [junit4]   2> 3331731 T6334 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1391244987475/slave/collection1/data
   [junit4]   2> 3331732 T6334 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
   [junit4]   2> 3331798 T6334 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: test params are: codec=Lucene3x, sim=DefaultSimilarity, locale=tr_TR, timezone=Indian/Christmas
   [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Sun Microsystems Inc. 1.6.0_32 (64-bit)/cpus=16,threads=1,free=238511184,total=498466816
   [junit4]   2> NOTE: All tests run in this JVM: [TestMergePolicyConfig, SyncSliceTest, TestCharFilters, SpellPossibilityIteratorTest, SolrIndexSplitterTest, InfoHandlerTest, TestValueSourceCache, TestSearchPerf, ChangedSchemaMergeTest, TestCoreContainer, TestFaceting, URLClassifyProcessorTest, TestLMJelinekMercerSimilarityFactory, QueryEqualityTest, TestDocBasedVersionConstraints, TestStressVersions, QueryElevationComponentTest, TestCopyFieldCollectionResource, TestHighFrequencyDictionaryFactory, TestFieldTypeCollectionResource, TestElisionMultitermQuery, TestPerFieldSimilarity, FileBasedSpellCheckerTest, UpdateRequestProcessorFactoryTest, TestSort, TestInfoStreamLogging, SampleTest, TestAnalyzeInfixSuggestions, PeerSyncTest, SoftAutoCommitTest, BadCopyFieldTest, IndexSchemaTest, TestRemoteStreaming, DistributedDebugComponentTest, TestSolrDeletionPolicy1, RegexBoostProcessorTest, TestSchemaVersionResource, PolyFieldTest, DirectUpdateHandlerOptimizeTest, DeleteReplicaTest, ReturnFieldsTest, TestFunctionQuery, SliceStateTest, SliceStateUpdateTest, UpdateParamsTest, TestCloudManagedSchema, HdfsLockFactoryTest, MBeansHandlerTest, ShowFileRequestHandlerTest, TestTrie, TestPostingsSolrHighlighter, TestRealTimeGet, QueryParsingTest, AutoCommitTest, TestRandomMergePolicy, TestAddFieldRealTimeGet, BasicDistributedZkTest, TermVectorComponentTest, LegacyHTMLStripCharFilterTest, TestStressUserVersions, TestShardHandlerFactory, DeleteShardTest, TestUniqueKeyFieldResource, DistribCursorPagingTest, AnalysisAfterCoreReloadTest, TestJmxMonitoredMap, BinaryUpdateRequestHandlerTest, SolrCoreTest, DocValuesTest, AssignTest, WordBreakSolrSpellCheckerTest, EnumFieldTest, SimpleFacetsTest, OverseerRolesTest, CSVRequestHandlerTest, TestSuggestSpellingConverter, MoreLikeThisHandlerTest, PrimitiveFieldTypeTest, ClusterStateTest, HdfsBasicDistributedZk2Test, LoggingHandlerTest, FieldMutatingUpdateProcessorTest, ZkControllerTest, TestDistributedMissingSort, TestSchemaNameResource, CollectionsAPIDistributedZkTest, OverseerTest, ShardRoutingTest, RecoveryZkTest, FullSolrCloudDistribCmdsTest, TestReplicationHandler]
   [junit4] Completed on J0 in 816.03s, 14 tests, 1 failure <<< FAILURES!

[...truncated 122 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/build.xml:466: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/build.xml:439: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/extra-targets.xml:37: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build.xml:189: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/common-build.xml:491: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/common-build.xml:1308: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/common-build.xml:941: There were test failures: 355 suites, 1549 tests, 1 error, 1 failure, 34 ignored (6 assumptions)

Total time: 131 minutes 45 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-NightlyTests-4.x - Build # 505 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-4.x/505/

10 tests failed.
REGRESSION:  org.apache.solr.BasicFunctionalityTest.testClientErrorOnMalformedNumbers

Error Message:
not an (update) client error on field: malformed_ti : org.apache.solr.common.SolrException: Exception writing document id 100 to the index; possible analysis error.

Stack Trace:
java.lang.AssertionError: not an (update) client error on field: malformed_ti : org.apache.solr.common.SolrException: Exception writing document id 100 to the index; possible analysis error.
	at __randomizedtesting.SeedInfo.seed([18448792A411A24B:756307404FA810EC]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.BasicFunctionalityTest.testClientErrorOnMalformedNumbers(BasicFunctionalityTest.java:367)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:679)


REGRESSION:  org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.testDistribSearch

Error Message:
Captured an uncaught exception in thread: Thread[id=5684, name=updateExecutor-336-thread-8, state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=5684, name=updateExecutor-336-thread-8, state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeTest]
	at __randomizedtesting.SeedInfo.seed([18448792A411A24B:99A2098AD34EC277]:0)
Caused by: java.util.concurrent.RejectedExecutionException
	at __randomizedtesting.SeedInfo.seed([18448792A411A24B]:0)
	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1992)
	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:822)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1373)
	at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:261)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:679)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest

Error Message:
ERROR: SolrIndexSearcher opens=19 closes=18

Stack Trace:
java.lang.AssertionError: ERROR: SolrIndexSearcher opens=19 closes=18
	at __randomizedtesting.SeedInfo.seed([18448792A411A24B]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.SolrTestCaseJ4.endTrackingSearchers(SolrTestCaseJ4.java:335)
	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:139)
	at sun.reflect.GeneratedMethodAccessor51.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:700)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:679)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest

Error Message:
2 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest:     1) Thread[id=5542, name=searcherExecutor-3054-thread-1, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:386)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1131)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=5536, name=qtp996870534-5536, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:838)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:871)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1201)         at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:214)         at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)         at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer.blockUntilFinished(ConcurrentUpdateSolrServer.java:386)         at org.apache.solr.update.StreamingSolrServers.blockUntilFinished(StreamingSolrServers.java:97)         at org.apache.solr.update.SolrCmdDistributor.finish(SolrCmdDistributor.java:69)         at org.apache.solr.update.processor.DistributedUpdateProcessor.doFinish(DistributedUpdateProcessor.java:608)         at org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1451)         at org.apache.solr.update.processor.LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:179)         at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:83)         at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)         at org.apache.solr.core.SolrCore.execute(SolrCore.java:1913)         at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:764)         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:418)         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:203)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)         at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:136)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)         at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)         at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229)         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)         at org.eclipse.jetty.server.handler.GzipHandler.handle(GzipHandler.java:301)         at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1077)         at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)         at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)         at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)         at org.eclipse.jetty.server.Server.handle(Server.java:368)         at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)         at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)         at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)         at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014)         at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:953)         at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)         at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)         at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest: 
   1) Thread[id=5542, name=searcherExecutor-3054-thread-1, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:386)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1131)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=5536, name=qtp996870534-5536, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:838)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:871)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1201)
        at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:214)
        at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
        at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer.blockUntilFinished(ConcurrentUpdateSolrServer.java:386)
        at org.apache.solr.update.StreamingSolrServers.blockUntilFinished(StreamingSolrServers.java:97)
        at org.apache.solr.update.SolrCmdDistributor.finish(SolrCmdDistributor.java:69)
        at org.apache.solr.update.processor.DistributedUpdateProcessor.doFinish(DistributedUpdateProcessor.java:608)
        at org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1451)
        at org.apache.solr.update.processor.LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:179)
        at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:83)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:1913)
        at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:764)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:418)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:203)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:136)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
        at org.eclipse.jetty.server.handler.GzipHandler.handle(GzipHandler.java:301)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1077)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
        at org.eclipse.jetty.server.Server.handle(Server.java:368)
        at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
        at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
        at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)
        at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014)
        at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:953)
        at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
        at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
        at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([18448792A411A24B]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=5542, name=searcherExecutor-3054-thread-1, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:386)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1131)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=5536, name=qtp996870534-5536, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:838)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:871)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1201)         at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:214)         at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)         at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer.blockUntilFinished(ConcurrentUpdateSolrServer.java:386)         at org.apache.solr.update.StreamingSolrServers.blockUntilFinished(StreamingSolrServers.java:97)         at org.apache.solr.update.SolrCmdDistributor.finish(SolrCmdDistributor.java:69)         at org.apache.solr.update.processor.DistributedUpdateProcessor.doFinish(DistributedUpdateProcessor.java:608)         at org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1451)         at org.apache.solr.update.processor.LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:179)         at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:83)         at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)         at org.apache.solr.core.SolrCore.execute(SolrCore.java:1913)         at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:764)         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:418)         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:203)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)         at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:136)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)         at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)         at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229)         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)         at org.eclipse.jetty.server.handler.GzipHandler.handle(GzipHandler.java:301)         at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1077)         at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)         at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)         at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)         at org.eclipse.jetty.server.Server.handle(Server.java:368)         at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)         at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)         at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)         at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014)         at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:953)         at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)         at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)         at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=5542, name=searcherExecutor-3054-thread-1, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:386)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1131)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=5536, name=qtp996870534-5536, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:838)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:871)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1201)
        at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:214)
        at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
        at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer.blockUntilFinished(ConcurrentUpdateSolrServer.java:386)
        at org.apache.solr.update.StreamingSolrServers.blockUntilFinished(StreamingSolrServers.java:97)
        at org.apache.solr.update.SolrCmdDistributor.finish(SolrCmdDistributor.java:69)
        at org.apache.solr.update.processor.DistributedUpdateProcessor.doFinish(DistributedUpdateProcessor.java:608)
        at org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1451)
        at org.apache.solr.update.processor.LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:179)
        at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:83)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:1913)
        at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:764)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:418)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:203)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:136)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
        at org.eclipse.jetty.server.handler.GzipHandler.handle(GzipHandler.java:301)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1077)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
        at org.eclipse.jetty.server.Server.handle(Server.java:368)
        at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
        at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
        at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)
        at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014)
        at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:953)
        at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
        at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
        at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([18448792A411A24B]:0)


REGRESSION:  org.apache.solr.handler.JsonLoaderTest.testAddBigIntegerValueToTrieField

Error Message:
Exception writing document id 1 to the index; possible analysis error.

Stack Trace:
org.apache.solr.common.SolrException: Exception writing document id 1 to the index; possible analysis error.
	at __randomizedtesting.SeedInfo.seed([18448792A411A24B:FB637CCA4FA97D0B]:0)
	at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:162)
	at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:69)
	at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51)
	at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:683)
	at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:837)
	at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:536)
	at org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:100)
	at org.apache.solr.handler.loader.JsonLoader$SingleThreadedJsonLoader.handleAdds(JsonLoader.java:392)
	at org.apache.solr.handler.loader.JsonLoader$SingleThreadedJsonLoader.processUpdate(JsonLoader.java:117)
	at org.apache.solr.handler.loader.JsonLoader$SingleThreadedJsonLoader.load(JsonLoader.java:101)
	at org.apache.solr.handler.loader.JsonLoader.load(JsonLoader.java:65)
	at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
	at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
	at org.apache.solr.core.SolrCore.execute(SolrCore.java:1913)
	at org.apache.solr.servlet.DirectSolrConnection.request(DirectSolrConnection.java:131)
	at org.apache.solr.SolrTestCaseJ4.updateJ(SolrTestCaseJ4.java:973)
	at org.apache.solr.handler.JsonLoaderTest.testAddBigIntegerValueToTrieField(JsonLoaderTest.java:402)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:679)
Caused by: org.apache.solr.common.SolrException: ERROR: [doc=1] Error adding field 'big_integer_tl'='12345678901234567890' msg=For input string: "12345678901234567890"
	at org.apache.solr.update.DocumentBuilder.toDocument(DocumentBuilder.java:167)
	at org.apache.solr.update.AddUpdateCommand.getLuceneDocument(AddUpdateCommand.java:77)
	at org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:232)
	at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:159)
	... 57 more
Caused by: java.lang.NumberFormatException: For input string: "12345678901234567890"
	at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
	at java.lang.Long.parseLong(Long.java:441)
	at java.lang.Long.parseLong(Long.java:478)
	at org.apache.solr.schema.TrieField.createField(TrieField.java:609)
	at org.apache.solr.schema.TrieField.createFields(TrieField.java:660)
	at org.apache.solr.update.DocumentBuilder.addField(DocumentBuilder.java:47)
	at org.apache.solr.update.DocumentBuilder.toDocument(DocumentBuilder.java:118)
	... 60 more


REGRESSION:  org.apache.solr.search.TestSolr4Spatial.testBadShapeParse400 {p0=srpt_geohash}

Error Message:
expected:<400> but was:<500>

Stack Trace:
java.lang.AssertionError: expected:<400> but was:<500>
	at __randomizedtesting.SeedInfo.seed([18448792A411A24B:BA9811396CF631A4]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.junit.Assert.assertEquals(Assert.java:456)
	at org.apache.solr.search.TestSolr4Spatial.testBadShapeParse400(TestSolr4Spatial.java:91)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:679)


REGRESSION:  org.apache.solr.search.TestSolr4Spatial.testBadShapeParse400 {p0=srpt_quad}

Error Message:
expected:<400> but was:<500>

Stack Trace:
java.lang.AssertionError: expected:<400> but was:<500>
	at __randomizedtesting.SeedInfo.seed([18448792A411A24B:BA9811396CF631A4]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.junit.Assert.assertEquals(Assert.java:456)
	at org.apache.solr.search.TestSolr4Spatial.testBadShapeParse400(TestSolr4Spatial.java:91)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:679)


REGRESSION:  org.apache.solr.search.TestSolr4Spatial.testBadShapeParse400 {p0=stqpt_geohash}

Error Message:
expected:<400> but was:<500>

Stack Trace:
java.lang.AssertionError: expected:<400> but was:<500>
	at __randomizedtesting.SeedInfo.seed([18448792A411A24B:BA9811396CF631A4]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.junit.Assert.assertEquals(Assert.java:456)
	at org.apache.solr.search.TestSolr4Spatial.testBadShapeParse400(TestSolr4Spatial.java:91)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:679)


REGRESSION:  org.apache.solr.search.TestSolr4Spatial.testBadShapeParse400 {p0=pointvector}

Error Message:
expected:<400> but was:<500>

Stack Trace:
java.lang.AssertionError: expected:<400> but was:<500>
	at __randomizedtesting.SeedInfo.seed([18448792A411A24B:BA9811396CF631A4]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.junit.Assert.assertEquals(Assert.java:456)
	at org.apache.solr.search.TestSolr4Spatial.testBadShapeParse400(TestSolr4Spatial.java:91)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:679)




Build Log:
[...truncated 9609 lines...]
   [junit4] Suite: org.apache.solr.handler.JsonLoaderTest
   [junit4]   2> 472050 T2112 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./solrtest-JsonLoaderTest-1391575227645
   [junit4]   2> 472052 T2112 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/'
   [junit4]   2> 472053 T2112 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 472054 T2112 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 472218 T2112 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 472308 T2112 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 472309 T2112 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 472319 T2112 oass.IndexSchema.readSchema [null] Schema name=test
   [junit4]   2> 472951 T2112 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 472952 T2112 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 472953 T2112 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 472965 T2112 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 472970 T2112 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 472982 T2112 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 472987 T2112 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 472991 T2112 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 472992 T2112 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 472993 T2112 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 472993 T2112 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 472994 T2112 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 472994 T2112 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 472995 T2112 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 472995 T2112 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr
   [junit4]   2> 472995 T2112 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/'
   [junit4]   2> 473013 T2112 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 473013 T2112 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr
   [junit4]   2> 473014 T2112 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/'
   [junit4]   2> 473092 T2112 oasc.CoreContainer.<init> New CoreContainer 344446677
   [junit4]   2> 473092 T2112 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/]
   [junit4]   2> 473094 T2112 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
   [junit4]   2> 473094 T2112 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 473095 T2112 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
   [junit4]   2> 473095 T2112 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 473095 T2112 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 473096 T2112 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 473096 T2112 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 473096 T2112 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 473097 T2112 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 473109 T2112 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 473109 T2112 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 473109 T2112 oasc.CoreContainer.load Host Name: 
   [junit4]   2> 473115 T2113 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1
   [junit4]   2> 473116 T2113 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/'
   [junit4]   2> 473117 T2113 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 473117 T2113 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 473184 T2113 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 473274 T2113 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 473275 T2113 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 473285 T2113 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 473903 T2113 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 473904 T2113 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 473904 T2113 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 473913 T2113 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 473917 T2113 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 473927 T2113 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 473932 T2113 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 473936 T2113 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 473937 T2113 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 473937 T2113 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 473937 T2113 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 473938 T2113 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 473938 T2113 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 473939 T2113 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 473939 T2113 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/, dataDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./solrtest-JsonLoaderTest-1391575227645/
   [junit4]   2> 473939 T2113 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6aadae91
   [junit4]   2> 473940 T2113 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./solrtest-JsonLoaderTest-1391575227645
   [junit4]   2> 473940 T2113 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./solrtest-JsonLoaderTest-1391575227645/index/
   [junit4]   2> 473941 T2113 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./solrtest-JsonLoaderTest-1391575227645/index' doesn't exist. Creating new index...
   [junit4]   2> 473941 T2113 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./solrtest-JsonLoaderTest-1391575227645/index
   [junit4]   2> 473941 T2113 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=23, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 473942 T2113 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@8741946 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2d1a8b1)),segFN=segments_1,generation=1}
   [junit4]   2> 473943 T2113 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 473943 T2113 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
   [junit4]   2> 473947 T2113 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 473947 T2113 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 473948 T2113 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 473948 T2113 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 473948 T2113 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 473948 T2113 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 473949 T2113 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 473949 T2113 oasup.UniqFieldsUpdateProcessorFactory.init WARN Use of the 'fields' init param in UniqFieldsUpdateProcessorFactory is deprecated, please use 'fieldName' (or another FieldMutatingUpdateProcessorFactory selector option) instead
   [junit4]   2> 473949 T2113 oasup.UniqFieldsUpdateProcessorFactory.init Replacing 'fields' init param with (individual) 'fieldName' params
   [junit4]   2> 473950 T2113 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 473950 T2113 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 473951 T2113 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 473951 T2113 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 473951 T2113 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 473951 T2113 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 473952 T2113 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 473952 T2113 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 473953 T2113 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 473953 T2113 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
   [junit4]   2> 473954 T2113 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
   [junit4]   2> 473954 T2113 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 473955 T2113 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 473955 T2113 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
   [junit4]   2> 473956 T2113 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
   [junit4]   2> 473956 T2113 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
   [junit4]   2> 473956 T2113 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 473957 T2113 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 473957 T2113 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 473958 T2113 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 473958 T2113 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 473959 T2113 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 473959 T2113 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 473960 T2113 oasc.RequestHandlers.initHandlersFromConfig created mltrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 473960 T2113 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 473961 T2113 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
   [junit4]   2> 473961 T2113 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
   [junit4]   2> 473971 T2113 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 473973 T2113 oasc.SolrCore.initDeprecatedSupport WARN solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
   [junit4]   2> 473977 T2113 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 473977 T2113 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 473978 T2113 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=32, maxMergeAtOnceExplicit=21, maxMergedSegmentMB=14.353515625, floorSegmentMB=2.03125, forceMergeDeletesPctAllowed=18.19916975805477, segmentsPerTier=41.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 473978 T2113 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@8741946 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2d1a8b1)),segFN=segments_1,generation=1}
   [junit4]   2> 473979 T2113 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 473979 T2113 oass.SolrIndexSearcher.<init> Opening Searcher@6fb8e6b9 main
   [junit4]   2> 473979 T2113 oashc.SpellCheckComponent.inform Initializing spell checkers
   [junit4]   2> 473987 T2113 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 474047 T2114 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6fb8e6b9 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 474052 T2113 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 474062 T2112 oas.SolrTestCaseJ4.initCore ####initCore end
   [junit4]   2> 474067 T2112 oas.SolrTestCaseJ4.setUp ###Starting testSimpleFormatInAdd
   [junit4]   2> 474068 T2112 oas.SolrTestCaseJ4.tearDown ###Ending testSimpleFormatInAdd
   [junit4]   2> 474073 T2112 oas.SolrTestCaseJ4.setUp ###Starting testDeleteSyntax
   [junit4]   2> 474073 T2112 oas.SolrTestCaseJ4.tearDown ###Ending testDeleteSyntax
   [junit4]   2> 474078 T2112 oas.SolrTestCaseJ4.setUp ###Starting testAddBigIntegerValueToTrieField
   [junit4]   2> ASYNC  NEW_CORE C445 name=collection1 org.apache.solr.core.SolrCore@3a44ebcb
   [junit4]   2> 474082 T2112 C445 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={wt=json&indent=true} {} 0 1
   [junit4]   2> 474084 T2112 C445 oasc.SolrException.log Ignoring exception matching big_integer_t
   [junit4]   2> 474084 T2112 oas.SolrTestCaseJ4.tearDown ###Ending testAddBigIntegerValueToTrieField
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=JsonLoaderTest -Dtests.method=testAddBigIntegerValueToTrieField -Dtests.seed=18448792A411A24B -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/hudson/lucene-data/enwiki.random.lines.txt -Dtests.locale=bg -Dtests.timezone=Africa/Sao_Tome -Dtests.file.encoding=ISO8859-1
   [junit4] ERROR   0.03s J0 | JsonLoaderTest.testAddBigIntegerValueToTrieField <<<
   [junit4]    > Throwable #1: org.apache.solr.common.SolrException: Exception writing document id 1 to the index; possible analysis error.
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([18448792A411A24B:FB637CCA4FA97D0B]:0)
   [junit4]    > 	at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:162)
   [junit4]    > 	at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:69)
   [junit4]    > 	at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51)
   [junit4]    > 	at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:683)
   [junit4]    > 	at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:837)
   [junit4]    > 	at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:536)
   [junit4]    > 	at org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:100)
   [junit4]    > 	at org.apache.solr.handler.loader.JsonLoader$SingleThreadedJsonLoader.handleAdds(JsonLoader.java:392)
   [junit4]    > 	at org.apache.solr.handler.loader.JsonLoader$SingleThreadedJsonLoader.processUpdate(JsonLoader.java:117)
   [junit4]    > 	at org.apache.solr.handler.loader.JsonLoader$SingleThreadedJsonLoader.load(JsonLoader.java:101)
   [junit4]    > 	at org.apache.solr.handler.loader.JsonLoader.load(JsonLoader.java:65)
   [junit4]    > 	at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
   [junit4]    > 	at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
   [junit4]    > 	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
   [junit4]    > 	at org.apache.solr.core.SolrCore.execute(SolrCore.java:1913)
   [junit4]    > 	at org.apache.solr.servlet.DirectSolrConnection.request(DirectSolrConnection.java:131)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.updateJ(SolrTestCaseJ4.java:973)
   [junit4]    > 	at org.apache.solr.handler.JsonLoaderTest.testAddBigIntegerValueToTrieField(JsonLoaderTest.java:402)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:679)
   [junit4]    > Caused by: org.apache.solr.common.SolrException: ERROR: [doc=1] Error adding field 'big_integer_tl'='12345678901234567890' msg=For input string: "12345678901234567890"
   [junit4]    > 	at org.apache.solr.update.DocumentBuilder.toDocument(DocumentBuilder.java:167)
   [junit4]    > 	at org.apache.solr.update.AddUpdateCommand.getLuceneDocument(AddUpdateCommand.java:77)
   [junit4]    > 	at org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:232)
   [junit4]    > 	at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:159)
   [junit4]    > 	... 57 more
   [junit4]    > Caused by: java.lang.NumberFormatException: For input string: "12345678901234567890"
   [junit4]    > 	at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
   [junit4]    > 	at java.lang.Long.parseLong(Long.java:441)
   [junit4]    > 	at java.lang.Long.parseLong(Long.java:478)
   [junit4]    > 	at org.apache.solr.schema.TrieField.createField(TrieField.java:609)
   [junit4]    > 	at org.apache.solr.schema.TrieField.createFields(TrieField.java:660)
   [junit4]    > 	at org.apache.solr.update.DocumentBuilder.addField(DocumentBuilder.java:47)
   [junit4]    > 	at org.apache.solr.update.DocumentBuilder.toDocument(DocumentBuilder.java:118)
   [junit4]    > 	... 60 more
   [junit4]   2> 474106 T2112 oas.SolrTestCaseJ4.setUp ###Starting testBooleanValuesInAdd
   [junit4]   2> 474107 T2112 oas.SolrTestCaseJ4.tearDown ###Ending testBooleanValuesInAdd
   [junit4]   2> 474112 T2112 oas.SolrTestCaseJ4.setUp ###Starting testNullValues
   [junit4]   2> 474116 T2112 C445 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 474120 T2112 C445 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@8741946 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2d1a8b1)),segFN=segments_1,generation=1}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@8741946 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2d1a8b1)),segFN=segments_2,generation=2}
   [junit4]   2> 474120 T2112 C445 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
   [junit4]   2> 474174 T2112 C445 oass.SolrIndexSearcher.<init> Opening Searcher@531da6d3 main
   [junit4]   2> 474175 T2112 C445 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 474177 T2114 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@531da6d3 main{StandardDirectoryReader(segments_2:3:nrt _0(4.7):c1)}
   [junit4]   2> 474177 T2112 C445 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={commit=true&wt=json&indent=true} {add=[10 (1459172388066295808)],commit=} 0 63
   [junit4]   2> 474178 T2112 C445 oasc.SolrCore.execute [collection1] webapp=null path=null params={fl=foo_s,foo2_s&q=id:10} hits=1 status=0 QTime=0 
   [junit4]   2> 474179 T2112 oas.SolrTestCaseJ4.tearDown ###Ending testNullValues
   [junit4]   2> 474184 T2112 oas.SolrTestCaseJ4.setUp ###Starting testIntegerValuesInAdd
   [junit4]   2> 474185 T2112 oas.SolrTestCaseJ4.tearDown ###Ending testIntegerValuesInAdd
   [junit4]   2> 474190 T2112 oas.SolrTestCaseJ4.setUp ###Starting testSimpleFormat
   [junit4]   2> 474191 T2112 oas.SolrTestCaseJ4.tearDown ###Ending testSimpleFormat
   [junit4]   2> 474196 T2112 oas.SolrTestCaseJ4.setUp ###Starting testBigDecimalValuesInAdd
   [junit4]   2> 474197 T2112 oas.SolrTestCaseJ4.tearDown ###Ending testBigDecimalValuesInAdd
   [junit4]   2> 474202 T2112 oas.SolrTestCaseJ4.setUp ###Starting testParsing
   [junit4]   2> 474208 T2112 oas.SolrTestCaseJ4.tearDown ###Ending testParsing
   [junit4]   2> 474213 T2112 oas.SolrTestCaseJ4.setUp ###Starting testDecimalValuesInAdd
   [junit4]   2> 474214 T2112 oas.SolrTestCaseJ4.tearDown ###Ending testDecimalValuesInAdd
   [junit4]   2> 474222 T2112 oas.SolrTestCaseJ4.setUp ###Starting testBigIntegerValuesInAdd
   [junit4]   2> 474223 T2112 oas.SolrTestCaseJ4.tearDown ###Ending testBigIntegerValuesInAdd
   [junit4]   2> 474228 T2112 oas.SolrTestCaseJ4.setUp ###Starting testExtendedFieldValues
   [junit4]   2> 474229 T2112 oas.SolrTestCaseJ4.tearDown ###Ending testExtendedFieldValues
   [junit4]   2> 474234 T2112 oas.SolrTestCaseJ4.setUp ###Starting testAddNonStringValues
   [junit4]   2> 474239 T2112 C445 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 474243 T2112 C445 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@8741946 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2d1a8b1)),segFN=segments_2,generation=2}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@8741946 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2d1a8b1)),segFN=segments_3,generation=3}
   [junit4]   2> 474243 T2112 C445 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
   [junit4]   2> 474245 T2112 C445 oass.SolrIndexSearcher.<init> Opening Searcher@2d62624e main
   [junit4]   2> 474246 T2112 C445 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 474248 T2114 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2d62624e main{StandardDirectoryReader(segments_3:5:nrt _0(4.7):c1 _1(4.7):c1)}
   [junit4]   2> 474249 T2112 C445 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={commit=true&wt=json&indent=true} {add=[1 (1459172388195270656)],commit=} 0 12
   [junit4]   2> 474250 T2112 C445 oasc.SolrCore.execute [collection1] webapp=null path=null params={fl=boolean_b,long_l,double_d,big_integer_s,big_decimal_s&q=id:1} hits=1 status=0 QTime=1 
   [junit4]   2> 474250 T2112 oas.SolrTestCaseJ4.tearDown ###Ending testAddNonStringValues
   [junit4]   2> 474256 T2112 oas.SolrTestCaseJ4.setUp ###Starting testAddBigDecimalValueToTrieField
   [junit4]   2> 474261 T2112 C445 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 474264 T2112 C445 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@8741946 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2d1a8b1)),segFN=segments_3,generation=3}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@8741946 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2d1a8b1)),segFN=segments_4,generation=4}
   [junit4]   2> 474264 T2112 C445 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
   [junit4]   2> 474266 T2112 C445 oass.SolrIndexSearcher.<init> Opening Searcher@6954dec7 main
   [junit4]   2> 474267 T2112 C445 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 474269 T2114 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6954dec7 main{StandardDirectoryReader(segments_4:9:nrt _0(4.7):c1 _2(4.7):c1)}
   [junit4]   2> 474270 T2112 C445 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={commit=true&wt=json&indent=true} {add=[1 (1459172388218339328)],commit=} 0 12
   [junit4]   2> 474271 T2112 C445 oasc.SolrCore.execute [collection1] webapp=null path=null params={fl=big_decimal_td&q=id:1} hits=1 status=0 QTime=1 
   [junit4]   2> 474275 T2112 C445 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 474278 T2112 C445 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@8741946 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2d1a8b1)),segFN=segments_4,generation=4}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@8741946 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2d1a8b1)),segFN=segments_5,generation=5}
   [junit4]   2> 474279 T2112 C445 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 5
   [junit4]   2> 474280 T2112 C445 oass.SolrIndexSearcher.<init> Opening Searcher@227669e7 main
   [junit4]   2> 474281 T2112 C445 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 474283 T2114 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@227669e7 main{StandardDirectoryReader(segments_5:11:nrt _0(4.7):c1 _2(4.7):c1 _3(4.7):c1)}
   [junit4]   2> 474284 T2112 C445 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={commit=true&wt=json&indent=true} {add=[2 (1459172388233019392)],commit=} 0 11
   [junit4]   2> 474285 T2112 C445 oasc.SolrCore.execute [collection1] webapp=null path=null params={fl=big_decimal_tf&q=id:2} hits=1 status=0 QTime=1 
   [junit4]   2> 474285 T2112 oas.SolrTestCaseJ4.tearDown ###Ending testAddBigDecimalValueToTrieField
   [junit4]   2> 474287 T2112 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 474287 T2112 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=344446677
   [junit4]   2> 474288 T2112 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@3a44ebcb
   [junit4]   2> 474307 T2112 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=4,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=1,cumulative_adds=5,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=1,transaction_logs_total_size=668,transaction_logs_total_number=4}
   [junit4]   2> 474308 T2112 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 474308 T2112 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 474309 T2112 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 474309 T2112 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 474310 T2112 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 474310 T2112 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./solrtest-JsonLoaderTest-1391575227645 [CachedDir<<refCount=0;path=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./solrtest-JsonLoaderTest-1391575227645;done=false>>]
   [junit4]   2> 474311 T2112 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./solrtest-JsonLoaderTest-1391575227645
   [junit4]   2> 474311 T2112 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./solrtest-JsonLoaderTest-1391575227645/index [CachedDir<<refCount=0;path=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./solrtest-JsonLoaderTest-1391575227645/index;done=false>>]
   [junit4]   2> 474311 T2112 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./solrtest-JsonLoaderTest-1391575227645/index
   [junit4]   2> NOTE: test params are: codec=Appending, sim=DefaultSimilarity, locale=bg, timezone=Africa/Sao_Tome
   [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Sun Microsystems Inc. 1.6.0_32 (64-bit)/cpus=16,threads=1,free=305181328,total=462290944
   [junit4]   2> NOTE: All tests run in this JVM: [TriLevelCompositeIdRoutingTest, TestJoin, ReturnFieldsTest, OverseerRolesTest, RecoveryZkTest, TestPseudoReturnFields, ScriptEngineTest, TestClassNameShortening, DistributedTermsComponentTest, TestLFUCache, ZkControllerTest, TestLRUCache, TestFieldCollectionResource, DistributedDebugComponentTest, UnloadDistributedZkTest, IndexSchemaTest, TestCollapseQParserPlugin, TestSolrIndexConfig, BinaryUpdateRequestHandlerTest, DOMUtilTest, SystemInfoHandlerTest, TestWriterPerf, ResponseLogComponentTest, UUIDFieldTest, SuggestComponentTest, TestSort, TestTrie, HdfsBasicDistributedZkTest, AnalysisAfterCoreReloadTest, AnalysisErrorHandlingTest, LukeRequestHandlerTest, TestMergePolicyConfig, QueryElevationComponentTest, LeaderElectionTest, QueryEqualityTest, MBeansHandlerTest, TestLuceneMatchVersion, TermsComponentTest, PeerSyncTest, SampleTest, MoreLikeThisHandlerTest, CircularListTest, TestFaceting, TestElisionMultitermQuery, TestDynamicFieldResource, TestStressVersions, ResourceLoaderTest, ClusterStateUpdateTest, ExternalFileFieldSortTest, DistributedQueryElevationComponentTest, WordBreakSolrSpellCheckerTest, TestSolrXmlPersistor, SimpleFacetsTest, JsonLoaderTest]
   [junit4] Completed on J0 in 2.30s, 14 tests, 1 error <<< FAILURES!

[...truncated 388 lines...]
   [junit4] Suite: org.apache.solr.search.TestSolr4Spatial
   [junit4]   2> 2917644 T8703 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/so

[...truncated too long message...]

lipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229)
   [junit4]    >         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
   [junit4]    >         at org.eclipse.jetty.server.handler.GzipHandler.handle(GzipHandler.java:301)
   [junit4]    >         at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1077)
   [junit4]    >         at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
   [junit4]    >         at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
   [junit4]    >         at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
   [junit4]    >         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
   [junit4]    >         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
   [junit4]    >         at org.eclipse.jetty.server.Server.handle(Server.java:368)
   [junit4]    >         at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
   [junit4]    >         at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
   [junit4]    >         at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)
   [junit4]    >         at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014)
   [junit4]    >         at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:953)
   [junit4]    >         at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
   [junit4]    >         at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
   [junit4]    >         at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
   [junit4]    >         at java.lang.Thread.run(Thread.java:679)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([18448792A411A24B]:0)Throwable #3: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=5542, name=searcherExecutor-3054-thread-1, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:386)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1131)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:679)
   [junit4]    >    2) Thread[id=5536, name=qtp996870534-5536, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:838)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:871)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1201)
   [junit4]    >         at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:214)
   [junit4]    >         at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
   [junit4]    >         at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer.blockUntilFinished(ConcurrentUpdateSolrServer.java:386)
   [junit4]    >         at org.apache.solr.update.StreamingSolrServers.blockUntilFinished(StreamingSolrServers.java:97)
   [junit4]    >         at org.apache.solr.update.SolrCmdDistributor.finish(SolrCmdDistributor.java:69)
   [junit4]    >         at org.apache.solr.update.processor.DistributedUpdateProcessor.doFinish(DistributedUpdateProcessor.java:608)
   [junit4]    >         at org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1451)
   [junit4]    >         at org.apache.solr.update.processor.LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:179)
   [junit4]    >         at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:83)
   [junit4]    >         at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
   [junit4]    >         at org.apache.solr.core.SolrCore.execute(SolrCore.java:1913)
   [junit4]    >         at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:764)
   [junit4]    >         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:418)
   [junit4]    >         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:203)
   [junit4]    >         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
   [junit4]    >         at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:136)
   [junit4]    >         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
   [junit4]    >         at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
   [junit4]    >         at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229)
   [junit4]    >         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
   [junit4]    >         at org.eclipse.jetty.server.handler.GzipHandler.handle(GzipHandler.java:301)
   [junit4]    >         at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1077)
   [junit4]    >         at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
   [junit4]    >         at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
   [junit4]    >         at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
   [junit4]    >         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
   [junit4]    >         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
   [junit4]    >         at org.eclipse.jetty.server.Server.handle(Server.java:368)
   [junit4]    >         at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
   [junit4]    >         at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
   [junit4]    >         at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)
   [junit4]    >         at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014)
   [junit4]    >         at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:953)
   [junit4]    >         at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
   [junit4]    >         at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
   [junit4]    >         at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
   [junit4]    >         at java.lang.Thread.run(Thread.java:679)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([18448792A411A24B]:0)
   [junit4] Completed on J0 in 540.09s, 1 test, 1 failure, 3 errors <<< FAILURES!

[...truncated 159 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/build.xml:466: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/build.xml:439: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/extra-targets.xml:37: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build.xml:189: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/common-build.xml:491: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/common-build.xml:1308: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/common-build.xml:941: There were test failures: 358 suites, 1552 tests, 3 suite-level errors, 2 errors, 5 failures, 188 ignored (5 assumptions)

Total time: 147 minutes 10 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-NightlyTests-4.x - Build # 504 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-4.x/504/

1 tests failed.
FAILED:  org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testDistribSearch

Error Message:
Error executing query

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Error executing query
	at __randomizedtesting.SeedInfo.seed([8643614BBAE8E243:7A5EF53CDB7827F]:0)
	at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:98)
	at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.compareResults(AbstractFullDistribZkTestBase.java:1254)
	at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testIndexingBatchPerRequestWithHttpSolrServer(FullSolrCloudDistribCmdsTest.java:413)
	at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.doTest(FullSolrCloudDistribCmdsTest.java:143)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:867)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:679)
Caused by: org.apache.http.ParseException: Invalid content type: 
	at org.apache.http.entity.ContentType.parse(ContentType.java:233)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:443)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:199)
	at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:90)
	... 45 more




Build Log:
[...truncated 9715 lines...]
   [junit4] Suite: org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
   [junit4]   2> 1673915 T4552 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 1673920 T4552 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./solrtest-FullSolrCloudDistribCmdsTest-1391449179975
   [junit4]   2> 1673922 T4552 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 1673922 T4553 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1674023 T4552 oasc.ZkTestServer.run start zk server on port:63305
   [junit4]   2> 1674025 T4552 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1674028 T4559 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@607a074f name:ZooKeeperConnection Watcher:127.0.0.1:63305 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1674029 T4552 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1674029 T4552 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1674043 T4552 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1674045 T4561 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1677ac88 name:ZooKeeperConnection Watcher:127.0.0.1:63305/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1674046 T4552 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1674046 T4552 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 1674050 T4552 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 1674059 T4552 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 1674061 T4552 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 1674064 T4552 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1674065 T4552 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1674071 T4552 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 1674072 T4552 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 1674209 T4552 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1674210 T4552 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1674241 T4552 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1674242 T4552 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1674248 T4552 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1674249 T4552 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1674307 T4552 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1674308 T4552 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 1674311 T4552 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1674312 T4552 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1674315 T4552 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1674316 T4552 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1674348 T4552 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1674349 T4552 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1674352 T4552 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1674353 T4552 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1674364 T4552 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1674365 T4552 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1674694 T4552 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1674699 T4552 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:63308
   [junit4]   2> 1674700 T4552 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1674700 T4552 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1674701 T4552 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1391449180463
   [junit4]   2> 1674701 T4552 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1391449180463/'
   [junit4]   2> 1674740 T4552 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1391449180463/solr.xml
   [junit4]   2> 1674820 T4552 oasc.CoreContainer.<init> New CoreContainer 1223797503
   [junit4]   2> 1674821 T4552 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1391449180463/]
   [junit4]   2> 1674823 T4552 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1674823 T4552 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1674824 T4552 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1674824 T4552 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1674824 T4552 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1674825 T4552 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1674825 T4552 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1674826 T4552 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1674826 T4552 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1674838 T4552 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1674839 T4552 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1674839 T4552 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1674840 T4552 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:63305/solr
   [junit4]   2> 1674840 T4552 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1674841 T4552 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1674846 T4572 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@721170c2 name:ZooKeeperConnection Watcher:127.0.0.1:63305 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1674847 T4552 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1674854 T4552 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1674859 T4574 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3ca50f74 name:ZooKeeperConnection Watcher:127.0.0.1:63305/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1674859 T4552 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1674867 T4552 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1674871 T4552 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1674881 T4552 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1674883 T4552 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:63308_
   [junit4]   2> 1674885 T4552 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:63308_
   [junit4]   2> 1674895 T4552 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 1674898 T4552 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1674911 T4552 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:63308_
   [junit4]   2> 1674911 T4552 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1674914 T4552 oasc.Overseer.start Overseer (id=91190013459365891-127.0.0.1:63308_-n_0000000000) starting
   [junit4]   2> 1674925 T4552 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1674937 T4576 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1674938 T4552 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1674938 T4576 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 1674941 T4552 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1674943 T4552 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1674945 T4575 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1674952 T4577 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 1674952 T4577 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1674953 T4574 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1674953 T4577 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1674955 T4575 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1674955 T4575 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:63308",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:63308_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1674956 T4575 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 1674956 T4575 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1674965 T4574 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1674966 T4574 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1675955 T4577 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1675955 T4577 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1391449180463/collection1
   [junit4]   2> 1675955 T4577 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 1675956 T4577 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1675956 T4577 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 1675957 T4577 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1391449180463/collection1/'
   [junit4]   2> 1675959 T4577 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1391449180463/collection1/lib/README' to classloader
   [junit4]   2> 1675960 T4577 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1391449180463/collection1/lib/classes/' to classloader
   [junit4]   2> 1676023 T4577 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 1676092 T4577 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1676194 T4577 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1676204 T4577 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1676767 T4577 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1676771 T4577 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1676773 T4577 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1676778 T4577 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1676801 T4577 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1676802 T4577 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1391449180463/collection1/, dataDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1391449179975/control/data/
   [junit4]   2> 1676802 T4577 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@59026e12
   [junit4]   2> 1676804 T4577 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1391449179975/control/data
   [junit4]   2> 1676805 T4577 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1391449179975/control/data/index/
   [junit4]   2> 1676805 T4577 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1391449179975/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1676805 T4577 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1391449179975/control/data/index
   [junit4]   2> 1676806 T4577 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=40, maxMergeAtOnceExplicit=26, maxMergedSegmentMB=37.375, floorSegmentMB=0.396484375, forceMergeDeletesPctAllowed=19.52428388438096, segmentsPerTier=31.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1676807 T4577 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@5c474728 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@3f8c10ca),segFN=segments_1,generation=1}
   [junit4]   2> 1676807 T4577 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1676811 T4577 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1676811 T4577 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1676811 T4577 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1676812 T4577 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1676812 T4577 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1676812 T4577 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1676813 T4577 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1676813 T4577 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1676814 T4577 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1676814 T4577 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1676815 T4577 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1676815 T4577 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1676815 T4577 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1676816 T4577 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1676816 T4577 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1676817 T4577 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1676827 T4577 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1676833 T4577 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1676833 T4577 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1676834 T4577 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=6, maxMergeAtOnceExplicit=5, maxMergedSegmentMB=40.3603515625, floorSegmentMB=1.796875, forceMergeDeletesPctAllowed=12.477895938204101, segmentsPerTier=36.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1676834 T4577 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@5c474728 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@3f8c10ca),segFN=segments_1,generation=1}
   [junit4]   2> 1676835 T4577 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1676835 T4577 oass.SolrIndexSearcher.<init> Opening Searcher@5908c982 main
   [junit4]   2> 1676838 T4578 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5908c982 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1676840 T4577 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1676841 T4552 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1
   [junit4]   2> 1676842 T4552 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1676841 T4581 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:63308 collection:control_collection shard:shard1
   [junit4]   2> 1676843 T4552 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1676844 T4581 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 1676845 T4583 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6a2de9fa name:ZooKeeperConnection Watcher:127.0.0.1:63305/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1676845 T4552 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1676848 T4552 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1676862 T4552 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1676863 T4581 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 1676872 T4574 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1676872 T4581 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1676872 T4581 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C8144 name=collection1 org.apache.solr.core.SolrCore@1d1bd7a2 url=http://127.0.0.1:63308/collection1 node=127.0.0.1:63308_ C8144_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:63308, core=collection1, node_name=127.0.0.1:63308_}
   [junit4]   2> 1676873 T4581 C8144 P63308 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:63308/collection1/
   [junit4]   2> 1676873 T4581 C8144 P63308 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1676873 T4575 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1676873 T4581 C8144 P63308 oasc.SyncStrategy.syncToMe http://127.0.0.1:63308/collection1/ has no replicas
   [junit4]   2> 1676874 T4581 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:63308/collection1/ shard1
   [junit4]   2> 1676874 T4581 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 1676877 T4574 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1676878 T4574 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1676878 T4583 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1676883 T4575 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1676886 T4574 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1676989 T4583 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1676989 T4574 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1677035 T4581 oasc.ZkController.register We are http://127.0.0.1:63308/collection1/ and leader is http://127.0.0.1:63308/collection1/
   [junit4]   2> 1677035 T4581 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:63308
   [junit4]   2> 1677035 T4581 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1677035 T4581 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 1677035 T4581 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1677037 T4574 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1677037 T4574 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1677037 T4574 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1677038 T4581 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1677039 T4575 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1677040 T4575 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:63308",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:63308_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 1677048 T4574 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1677148 T4552 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 1677149 T4552 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1677150 T4583 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1677150 T4574 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1677151 T4552 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:63312
   [junit4]   2> 1677152 T4552 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1677153 T4552 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1677153 T4552 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1391449182917
   [junit4]   2> 1677154 T4552 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1391449182917/'
   [junit4]   2> 1677191 T4552 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1391449182917/solr.xml
   [junit4]   2> 1677271 T4552 oasc.CoreContainer.<init> New CoreContainer 982084149
   [junit4]   2> 1677272 T4552 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1391449182917/]
   [junit4]   2> 1677274 T4552 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1677274 T4552 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1677275 T4552 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1677275 T4552 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1677275 T4552 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1677276 T4552 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1677276 T4552 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1677277 T4552 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1677277 T4552 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1677289 T4552 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1677289 T4552 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1677290 T4552 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1677291 T4552 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:63305/solr
   [junit4]   2> 1677291 T4552 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1677292 T4552 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1677294 T4594 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@139bdb99 name:ZooKeeperConnection Watcher:127.0.0.1:63305 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1677295 T4552 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1677298 T4552 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1677306 T4596 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2dad4c1f name:ZooKeeperConnection Watcher:127.0.0.1:63305/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1677306 T4552 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1677315 T4552 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1678319 T4552 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:63312_
   [junit4]   2> 1678321 T4552 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:63312_
   [junit4]   2> 1678325 T4583 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1678325 T4596 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1678325 T4574 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1678340 T4597 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 1678340 T4597 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1678341 T4574 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1678342 T4597 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1678342 T4574 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1678342 T4574 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1678344 T4575 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1678344 T4575 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:63312",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:63312_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1678345 T4575 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
   [junit4]   2> 1678345 T4575 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1678348 T4574 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1678451 T4596 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1678451 T4574 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1678451 T4583 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1679343 T4597 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1679343 T4597 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1391449182917/collection1
   [junit4]   2> 1679343 T4597 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 1679344 T4597 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1679345 T4597 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1679346 T4597 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1391449182917/collection1/'
   [junit4]   2> 1679347 T4597 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1391449182917/collection1/lib/README' to classloader
   [junit4]   2> 1679348 T4597 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1391449182917/collection1/lib/classes/' to classloader
   [junit4]   2> 1679410 T4597 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 1679480 T4597 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1679583 T4597 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1679593 T4597 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1680158 T4597 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1680163 T4597 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1680165 T4597 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1680170 T4597 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1680194 T4597 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1680194 T4597 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1391449182917/collection1/, dataDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1391449179975/jetty1/
   [junit4]   2> 1680195 T4597 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@59026e12
   [junit4]   2> 1680197 T4597 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1391449179975/jetty1
   [junit4]   2> 1680197 T4597 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1391449179975/jetty1/index/
   [junit4]   2> 1680198 T4597 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1391449179975/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 1680199 T4597 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1391449179975/jetty1/index
   [junit4]   2> 1680199 T4597 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=40, maxMergeAtOnceExplicit=26, maxMergedSegmentMB=37.375, floorSegmentMB=0.396484375, forceMergeDeletesPctAllowed=19.52428388438096, segmentsPerTier=31.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1680200 T4597 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@23198bbe lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@29e2a2c8),segFN=segments_1,generation=1}
   [junit4]   2> 1680200 T4597 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1680205 T4597 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1680205 T4597 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1680206 T4597 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1680206 T4597 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1680206 T4597 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1680207 T4597 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1680207 T4597 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1680208 T4597 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1680208 T4597 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1680209 T4597 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1680209 T4597 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1680210 T4597 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1680210 T4597 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1680211 T4597 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1680211 T4597 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1680212 T4597 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1680222 T4597 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1680227 T4597 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1680227 T4597 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1680228 T4597 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=6, maxMergeAtOnceExplicit=5, maxMergedSegmentMB=40.3603515625, floorSegmentMB=1.796875, forceMergeDeletesPctAllowed=12.477895938204101, segmentsPerTier=36.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1680229 T4597 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@23198bbe lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@29e2a2c8),segFN=segments_1,generation=1}
   [junit4]   2> 1680229 T4597 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1680229 T4597 oass.SolrIndexSearcher.<init> Opening Searcher@109422c3 main
   [junit4]   2> 1680234 T4598 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@109422c3 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1680237 T4597 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1680238 T4552 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1
   [junit4]   2> 1680238 T4552 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1680238 T4601 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:63312 collection:collection1 shard:shard1
   [junit4]   2> 1680240 T4601 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 1680278 T4601 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 1680288 T4574 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1680288 T4574 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1680288 T4601 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1680288 T4601 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 1680288 T4574 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> ASYNC  NEW_CORE C8145 name=collection1 org.apache.solr.core.SolrCore@282596ea url=http://127.0.0.1:63312/collection1 node=127.0.0.1:63312_ C8145_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:63312, core=collection1, node_name=127.0.0.1:63312_}
   [junit4]   2> 1680289 T4601 C8145 P63312 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:63312/collection1/
   [junit4]   2> 1680289 T4601 C8145 P63312 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1680289 T4601 C8145 P63312 oasc.SyncStrategy.syncToMe http://127.0.0.1:63312/collection1/ has no replicas
   [junit4]   2> 1680290 T4601 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:63312/collection1/ shard1
   [junit4]   2> 1680290 T4601 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 1680290 T4575 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1680293 T4574 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1680294 T4596 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1680294 T4583 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1680294 T4574 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1680298 T4575 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1680301 T4574 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1680405 T4583 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1680405 T4574 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1680405 T4596 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1680453 T4601 oasc.ZkController.register We are http://127.0.0.1:63312/collection1/ and leader is http://127.0.0.1:63312/collection1/
   [junit4]   2> 1680453 T4601 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:63312
   [junit4]   2> 1680453 T4601 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1680453 T4601 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 1680454 T4601 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1680455 T4574 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1680455 T4574 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1680455 T4574 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1680456 T4601 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1680457 T4575 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1680458 T4575 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:63312",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:63312_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 1680466 T4574 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1680519 T4552 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 1680520 T4552 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1680523 T4552 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:63315
   [junit4]   2> 1680524 T4552 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1680524 T4552 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1680525 T4552 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1391449186293
   [junit4]   2> 1680525 T4552 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1391449186293/'
   [junit4]   2> 1680563 T4552 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1391449186293/solr.xml
   [junit4]   2> 1680569 T4596 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1680569 T4574 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1680569 T4583 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1680642 T4552 oasc.CoreContainer.<init> New CoreContainer 897540722
   [junit4]   2> 1680643 T4552 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1391449186293/]
   [junit4]   2> 1680645 T4552 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1680645 T4552 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1680646 T4552 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1680646 T4552 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1680647 T4552 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1680647 T4552 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1680648 T4552 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1680648 T4552 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1680648 T4552 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1680658 T4552 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1680659 T4552 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1680659 T4552 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1680660 T4552 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:63305/solr
   [junit4]   2> 1680660 T4552 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1680661 T4552 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1680664 T4612 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3e1845ce name:ZooKeeperConnection Watcher:127.0.0.1:63305 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1680664 T4552 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1680667 T4552 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1680675 T4614 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@42326b30 name:ZooKeeperConnection Watcher:127.0.0.1:63305/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1680675 T4552 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1680680 T4552 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1681683 T4552 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:63315_
   [junit4]   2> 1681695 T4552 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:63315_
   [junit4]   2> 1681699 T4583 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1681699 T4596 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1681699 T4614 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1681699 T4574 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1681708 T4615 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 1681708 T4615 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1681709 T4574 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1681709 T4615 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1681709 T4574 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1681710 T4574 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1681711 T4575 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1681712 T4575 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:63315",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:63315_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1681712 T4575 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 1681713 T4575 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 1681715 T4574 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1681819 T4596 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1681819 T4574 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1681819 T4583 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1681819 T4614 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1682712 T4615 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1682712 T4615 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1391449186293/collection1
   [junit4]   2> 1682712 T4615 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 1682713 T4615 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1682713 T4615 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1682715 T4615 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1391449186293/collection1/'
   [junit4]   2> 1682716 T4615 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1391449186293/collection1/lib/README' to classloader
   [junit4]   2> 1682717 T4615 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1391449186293/collection1/lib/classes/' to classloader
   [junit4]   2> 1682785 T4615 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 1682854 T4615 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1682956 T4615 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1682966 T4615 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1683529 T4615 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1683533 T4615 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1683535 T4615 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1683540 T4615 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1683564 T4615 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1683564 T4615 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1391449186293/collection1/, dataDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1391449179975/jetty2/
   [junit4]   2> 1683564 T4615 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@59026e12
   [junit4]   2> 1683567 T4615 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1391449179975/jetty2
   [junit4]   2> 1683567 T4615 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1391449179975/jetty2/index/
   [junit4]   2> 1683567 T4615 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1391449179975/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 1683568 T4615 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1391449179975/jetty2/index
   [junit4]   2> 1683569 T4615 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=40, maxMergeAtOnceExplicit=26, maxMergedSegmentMB=37.375, floorSegmentMB=0.396484375, forceMergeDeletesPctAllowed=19.52428388438096, segmentsPerTier=31.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1683570 T4615 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@757a9fe lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@69a4090b),segFN=segments_1,generation=1}
   [junit4]   2> 1683570 T4615 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1683574 T4615 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1683575 T4615 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1683575 T4615 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1683575 T4615 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1683576 T4615 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1683576 T4615 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1683577 T4615 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1683577 T4615 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1683577 T4615 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1683578 T4615 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1683579 T4615 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1683579 T4615 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1683579 T4615 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1683580 T4615 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1683581 T4615 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1683581 T4615 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1683591 T4615 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1683596 T4615 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1683596 T4615 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1683597 T4615 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=6, maxMergeAtOnceExplicit=5, maxMergedSegmentMB=40.3603515625, floorSegmentMB=1.796875, forceMergeDeletesPctAllowed=12.477895938204101, segmentsPerTier=36.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1683598 T4615 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@757a9fe lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@69a4090b),segFN=segments_1,generation=1}
   [junit4]   2> 1683598 T4615 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1683599 T4615 oass.SolrIndexSearcher.<init> Opening Searcher@239ed96d main
   [junit4]   2> 1683603 T4616 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@239ed96d main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1683606 T4615 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1683607 T4552 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1
   [junit4]   2> 1683607 T4552 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1683607 T4619 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:63315 collection:collection1 shard:shard2
   [junit4]   2> 1683609 T4619 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 1683627 T4619 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 1683630 T4574 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1683630 T4574 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1683630 T4619 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1683630 T4574 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1683631 T4619 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C8146 name=collection1 org.apache.solr.core.SolrCore@737e0eae url=http://127.0.0.1:63315/collection1 node=127.0.0.1:63315_ C8146_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:63315, core=collection1, node_name=127.0.0.1:63315_}
   [junit4]   2> 1683631 T4619 C8146 P63315 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:63315/collection1/
   [junit4]   2> 1683631 T4575 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1683631 T4619 C8146 P63315 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1683632 T4619 C8146 P63315 oasc.SyncStrategy.syncToMe http://127.0.0.1:63315/collection1/ has no replicas
   [junit4]   2> 1683632 T4619 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:63315/collection1/ shard2
   [junit4]   2> 1683633 T4619 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 1683636 T4574 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1683645 T4596 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1683645 T4583 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1683645 T4614 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1683645 T4574 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1683660 T4575 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1683664 T4574 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1683766 T4596 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1683766 T4583 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1683766 T4614 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1683766 T4574 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1683812 T4619 oasc.ZkController.register We are http://127.0.0.1:63315/collection1/ and leader is http://127.0.0.1:63315/collection1/
   [junit4]   2> 1683812 T4619 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:63315
   [junit4]   2> 1683812 T4619 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1683812 T4619 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 1683813 T4619 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1683814 T4574 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1683814 T4574 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1683815 T4574 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1683815 T4619 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1683816 T4575 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1683817 T4575 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:63315",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:63315_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 1683826 T4574 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1683895 T4552 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 1683896 T4552 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1683898 T4552 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:63318
   [junit4]   2> 1683899 T4552 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1683900 T4552 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1683900 T4552 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1391449189662
   [junit4]   2> 1683901 T4552 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1391449189662/'
   [junit4]   2> 1683930 T4596 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1683930 T4574 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1683930 T4614 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1683930 T4583 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1683939 T4552 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1391449189662/solr.xml
   [junit4]   2> 1684018 T4552 oasc.CoreContainer.<init> New CoreContainer 1526748927
   [junit4]   2> 1684019 T4552 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1391449189662/]
   [junit4]   2> 1684021 T4552 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1684021 T4552 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1684022 T4552 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1684022 T4552 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1684023 T4552 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1684023 T4552 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1684024 T4552 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1684024 T4552 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1684025 T4552 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1684036 T4552 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1684037 T4552 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1684037 T4552 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1684038 T4552 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:63305/solr
   [junit4]   2> 1684038 T4552 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1684039 T4552 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1684043 T4630 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@734fae05 name:ZooKeeperConnection Watcher:127.0.0.1:63305 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1684044 T4552 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1684047 T4552 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1684054 T4632 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@53725972 name:ZooKeeperConnection Watcher:127.0.0.1:63305/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1684054 T4552 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1684059 T4552 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1685063 T4552 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:63318_
   [junit4]   2> 1685066 T4552 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:63318_
   [junit4]   2> 1685069 T4596 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1685069 T4583 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1685069 T4574 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1685070 T4632 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1685070 T4614 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1685085 T4633 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 1685085 T4633 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1685087 T4574 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1685087 T4633 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1685087 T4574 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1685087 T4574 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1685089 T4575 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1685089 T4575 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:63318",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:63318_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1685090 T4575 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 1685090 T4575 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
   [junit4]   2> 1685094 T4574 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1685198 T4614 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 1685198 T4596 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 1685198 T4574 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 1685198 T4632 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 1685198 T4583 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 1686088 T4633 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1686088 T4633 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1391449189662/collection1
   [junit4]   2> 1686088 T4633 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 1686089 T4633 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1686089 T4633 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1686090 T4633 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1391449189662/collection1/'
   [junit4]   2> 1686092 T4633 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1391449189662/collection1/lib/README' to classloader
   [junit4]   2> 1686093 T4633 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1391449189662/collection1/lib/classes/' to classloader
   [junit4]   2> 1686157 T4633 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 1686227 T4633 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1686329 T4633 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1686339 T4633 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1686906 T4633 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1686910 T4633 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1686913 T4633 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1686918 T4633 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1686941 T4633 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1686941 T4633 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspac

[...truncated too long message...]

apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 1788521 T4689 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection2/leader_elect/shard7/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> ASYNC  NEW_CORE C9506 name=collection2_shard3_replica1 org.apache.solr.core.SolrCore@47a4aa49 url=http://127.0.0.1:63318/collection2_shard3_replica1 node=127.0.0.1:63318_ C9506_STATE=coll:collection2 core:collection2_shard3_replica1 props:{state=active, base_url=http://127.0.0.1:63318, core=collection2_shard3_replica1, node_name=127.0.0.1:63318_}
   [junit4]   2> 1788636 T4632 C9506 P63318 oasc.SyncStrategy.sync WARN Closed, skipping sync up.
   [junit4]   2> 1788637 T4632 oasc.ShardLeaderElectionContext.rejoinLeaderElection Not rejoining election because CoreContainer is shutdown
   [junit4]   2> 1788637 T4632 oasc.SolrCore.close [collection2_shard3_replica1]  CLOSING SolrCore org.apache.solr.core.SolrCore@47a4aa49
   [junit4]   2> 1788649 T4632 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=18638,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0,transaction_logs_total_size=2850716,transaction_logs_total_number=1}
   [junit4]   2> 1788649 T4632 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 1788649 T4632 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 1788649 T4632 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 1788650 T4632 oasc.SolrCore.closeSearcher [collection2_shard3_replica1] Closing main searcher on request.
   [junit4]   2> 1788650 T4632 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 1788651 T4632 oasc.CachingDirectoryFactory.closeCacheValue looking to close ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1391449189662/collection2_shard3_replica1/data/index [CachedDir<<refCount=0;path=../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1391449189662/collection2_shard3_replica1/data/index;done=false>>]
   [junit4]   2> 1788651 T4632 oasc.CachingDirectoryFactory.close Closing directory: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1391449189662/collection2_shard3_replica1/data/index
   [junit4]   2> 1788651 T4632 oasc.CachingDirectoryFactory.closeCacheValue looking to close ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1391449189662/collection2_shard3_replica1/data [CachedDir<<refCount=0;path=../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1391449189662/collection2_shard3_replica1/data;done=false>>]
   [junit4]   2> 1788652 T4632 oasc.CachingDirectoryFactory.close Closing directory: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1391449189662/collection2_shard3_replica1/data
   [junit4]   2> 1788652 T4632 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 1788652 T4632 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> NOTE: test params are: codec=Lucene3x, sim=DefaultSimilarity, locale=ar_SY, timezone=Indian/Cocos
   [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Sun Microsystems Inc. 1.6.0_32 (64-bit)/cpus=16,threads=1,free=190860664,total=475791360
   [junit4]   2> NOTE: All tests run in this JVM: [TestCollationField, TestSweetSpotSimilarityFactory, TestReload, SpellCheckCollatorTest, TestDefaultSimilarityFactory, TestDistribDocBasedVersion, TestFastWriter, SolrCmdDistributorTest, TestCoreDiscovery, MigrateRouteKeyTest, BasicZkTest, TestPartialUpdateDeduplication, TestCodecSupport, SignatureUpdateProcessorFactoryTest, TestFastOutputStream, ChaosMonkeySafeLeaderTest, TestUtils, PreAnalyzedFieldTest, TestCollationKeyRangeQueries, ShardSplitTest, HighlighterConfigTest, SuggestComponentTest, JSONWriterTest, TermVectorComponentDistributedTest, LeaderElectionIntegrationTest, TestComponentsName, PathHierarchyTokenizerFactoryTest, IndexBasedSpellCheckerTest, PrimUtilsTest, RAMDirectoryFactoryTest, DateFieldTest, TestIBSimilarityFactory, TestSimpleQParserPlugin, TestFieldResource, TestRangeQuery, TestFileDictionaryLookup, XsltUpdateRequestHandlerTest, OpenCloseCoreStressTest, TestOmitPositions, TestCustomSort, DisMaxRequestHandlerTest, TestSolr4Spatial, TestBinaryResponseWriter, TestQuerySenderNoQuery, TestQueryTypes, OpenExchangeRatesOrgProviderTest, TestNumberUtils, TestCloudManagedSchemaAddField, StressHdfsTest, SimplePostToolTest, FullSolrCloudDistribCmdsTest]
   [junit4] Completed on J1 in 114.84s, 1 test, 1 error <<< FAILURES!

[...truncated 754 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/build.xml:466: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/build.xml:439: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/extra-targets.xml:37: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build.xml:189: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/common-build.xml:491: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/common-build.xml:1308: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/common-build.xml:941: There were test failures: 355 suites, 1549 tests, 1 error, 33 ignored (5 assumptions)

Total time: 164 minutes 56 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-NightlyTests-4.x - Build # 503 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-4.x/503/

1 tests failed.
FAILED:  org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testDistribSearch

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded
	at __randomizedtesting.SeedInfo.seed([5CF454FF344898E8]:0)




Build Log:
[...truncated 9832 lines...]
   [junit4] Suite: org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
   [junit4]   2> 2050131 T3316 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 2050136 T3316 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./solrtest-FullSolrCloudDistribCmdsTest-1391338244846
   [junit4]   2> 2050138 T3316 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 2050138 T3317 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 2050239 T3316 oasc.ZkTestServer.run start zk server on port:63050
   [junit4]   2> 2050241 T3316 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2050272 T3323 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7a842468 name:ZooKeeperConnection Watcher:127.0.0.1:63050 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2050272 T3316 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2050273 T3316 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 2050290 T3316 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2050292 T3325 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@22705bf8 name:ZooKeeperConnection Watcher:127.0.0.1:63050/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2050292 T3316 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2050292 T3316 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 2050301 T3316 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 2050304 T3316 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 2050311 T3316 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 2050314 T3316 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 2050315 T3316 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 2050326 T3316 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 2050327 T3316 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 2050435 T3316 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2050436 T3316 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2050439 T3316 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 2050440 T3316 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 2050443 T3316 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 2050444 T3316 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 2050447 T3316 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 2050448 T3316 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 2050451 T3316 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 2050452 T3316 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 2050455 T3316 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2050456 T3316 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2050463 T3316 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2050464 T3316 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2050467 T3316 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 2050468 T3316 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 2050471 T3316 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 2050472 T3316 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 2050801 T3316 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2050805 T3316 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:48635
   [junit4]   2> 2050806 T3316 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2050806 T3316 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2050807 T3316 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1391338245185
   [junit4]   2> 2050807 T3316 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1391338245185/'
   [junit4]   2> 2050847 T3316 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1391338245185/solr.xml
   [junit4]   2> 2050929 T3316 oasc.CoreContainer.<init> New CoreContainer 377922366
   [junit4]   2> 2050930 T3316 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1391338245185/]
   [junit4]   2> 2050932 T3316 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2050932 T3316 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 2050932 T3316 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2050933 T3316 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2050933 T3316 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2050934 T3316 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2050934 T3316 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2050935 T3316 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2050935 T3316 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2050948 T3316 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2050949 T3316 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2050949 T3316 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2050950 T3316 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:63050/solr
   [junit4]   2> 2050950 T3316 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2050951 T3316 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2050953 T3336 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@614746ff name:ZooKeeperConnection Watcher:127.0.0.1:63050 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2050954 T3316 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2050957 T3316 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2050967 T3338 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@22b204ce name:ZooKeeperConnection Watcher:127.0.0.1:63050/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2050967 T3316 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2050969 T3316 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 2050977 T3316 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 2050980 T3316 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 2050988 T3316 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:48635_
   [junit4]   2> 2050989 T3316 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:48635_
   [junit4]   2> 2050997 T3316 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 2050999 T3316 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 2051012 T3316 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:48635_
   [junit4]   2> 2051012 T3316 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 2051015 T3316 oasc.Overseer.start Overseer (id=91182743214751747-127.0.0.1:48635_-n_0000000000) starting
   [junit4]   2> 2051025 T3316 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 2051036 T3340 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 2051036 T3316 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 2051037 T3340 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 2051039 T3316 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 2051041 T3316 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2051043 T3339 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 2051049 T3341 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 2051049 T3341 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2051050 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2051050 T3341 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2051052 T3339 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2051052 T3339 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:48635",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:48635_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 2051052 T3339 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 2051053 T3339 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 2051061 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2051062 T3338 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2052052 T3341 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 2052052 T3341 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1391338245185/collection1
   [junit4]   2> 2052052 T3341 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 2052053 T3341 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2052053 T3341 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 2052054 T3341 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1391338245185/collection1/'
   [junit4]   2> 2052055 T3341 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1391338245185/collection1/lib/README' to classloader
   [junit4]   2> 2052056 T3341 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1391338245185/collection1/lib/classes/' to classloader
   [junit4]   2> 2052103 T3341 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 2052156 T3341 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2052258 T3341 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2052266 T3341 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 2052833 T3341 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2052837 T3341 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2052840 T3341 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2052845 T3341 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2052870 T3341 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 2052870 T3341 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1391338245185/collection1/, dataDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1391338244846/control/data/
   [junit4]   2> 2052871 T3341 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7c2fbdd2
   [junit4]   2> 2052872 T3341 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1391338244846/control/data
   [junit4]   2> 2052872 T3341 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1391338244846/control/data/index/
   [junit4]   2> 2052872 T3341 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1391338244846/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 2052873 T3341 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1391338244846/control/data/index
   [junit4]   2> 2052873 T3341 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@452c571b
   [junit4]   2> 2052874 T3341 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@274fafe6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@77d8cee5),segFN=segments_1,generation=1}
   [junit4]   2> 2052874 T3341 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2052878 T3341 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 2052878 T3341 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 2052879 T3341 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 2052879 T3341 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2052879 T3341 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2052880 T3341 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 2052880 T3341 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2052881 T3341 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2052881 T3341 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2052882 T3341 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 2052882 T3341 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 2052882 T3341 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 2052883 T3341 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 2052883 T3341 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 2052884 T3341 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 2052884 T3341 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 2053106 T3341 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2053111 T3341 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 2053111 T3341 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 2053112 T3341 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=15, maxMergeAtOnceExplicit=22, maxMergedSegmentMB=87.7158203125, floorSegmentMB=1.3486328125, forceMergeDeletesPctAllowed=21.917569415090774, segmentsPerTier=27.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 2053112 T3341 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@274fafe6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@77d8cee5),segFN=segments_1,generation=1}
   [junit4]   2> 2053112 T3341 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2053113 T3341 oass.SolrIndexSearcher.<init> Opening Searcher@2c4611f8 main
   [junit4]   2> 2053115 T3342 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2c4611f8 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 2053116 T3341 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 2053116 T3316 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1
   [junit4]   2> 2053117 T3316 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2053116 T3345 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:48635 collection:control_collection shard:shard1
   [junit4]   2> 2053118 T3316 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2053119 T3345 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 2053121 T3347 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@78908ef3 name:ZooKeeperConnection Watcher:127.0.0.1:63050/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2053121 T3316 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2053140 T3316 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2053189 T3316 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 2053199 T3345 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 2053207 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2053207 T3345 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 2053207 T3345 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C7244 name=collection1 org.apache.solr.core.SolrCore@a9acd82 url=http://127.0.0.1:48635/collection1 node=127.0.0.1:48635_ C7244_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:48635, core=collection1, node_name=127.0.0.1:48635_}
   [junit4]   2> 2053208 T3345 C7244 P48635 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:48635/collection1/
   [junit4]   2> 2053208 T3339 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2053208 T3345 C7244 P48635 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 2053209 T3345 C7244 P48635 oasc.SyncStrategy.syncToMe http://127.0.0.1:48635/collection1/ has no replicas
   [junit4]   2> 2053209 T3345 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:48635/collection1/ shard1
   [junit4]   2> 2053209 T3345 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 2053212 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2053221 T3338 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2053221 T3347 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2053305 T3339 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2053350 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2053383 T3316 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 2053384 T3316 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2053386 T3316 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:36263
   [junit4]   2> 2053386 T3316 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2053386 T3316 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2053387 T3316 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1391338247898
   [junit4]   2> 2053387 T3316 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1391338247898/'
   [junit4]   2> 2053413 T3316 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1391338247898/solr.xml
   [junit4]   2> 2053453 T3338 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2053453 T3347 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2053467 T3316 oasc.CoreContainer.<init> New CoreContainer 798378623
   [junit4]   2> 2053467 T3316 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1391338247898/]
   [junit4]   2> 2053468 T3316 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2053469 T3316 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 2053469 T3316 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2053469 T3316 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2053470 T3316 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2053470 T3316 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2053470 T3316 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2053470 T3316 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2053471 T3316 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2053478 T3316 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2053479 T3316 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2053479 T3316 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2053479 T3316 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:63050/solr
   [junit4]   2> 2053480 T3316 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2053480 T3316 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2053487 T3358 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@33f4f041 name:ZooKeeperConnection Watcher:127.0.0.1:63050 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2053488 T3316 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2053498 T3316 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2053508 T3345 oasc.ZkController.register We are http://127.0.0.1:48635/collection1/ and leader is http://127.0.0.1:48635/collection1/
   [junit4]   2> 2053508 T3345 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:48635
   [junit4]   2> 2053508 T3345 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 2053508 T3360 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@651bbb3c name:ZooKeeperConnection Watcher:127.0.0.1:63050/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2053508 T3345 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 2053509 T3345 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2053508 T3316 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2053510 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2053511 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2053511 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2053516 T3345 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2053518 T3339 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2053518 T3316 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2053519 T3339 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:48635",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:48635_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 2053527 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2053631 T3338 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2053631 T3360 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2053631 T3347 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2054529 T3316 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36263_
   [junit4]   2> 2054533 T3316 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:36263_
   [junit4]   2> 2054536 T3338 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 2054536 T3347 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 2054536 T3360 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 2054551 T3361 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 2054551 T3361 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2054553 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2054553 T3361 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2054553 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2054553 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2054554 T3339 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2054555 T3339 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:36263",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:36263_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 2054555 T3339 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
   [junit4]   2> 2054556 T3339 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 2054560 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2054663 T3338 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2054663 T3347 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2054663 T3360 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2055554 T3361 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 2055554 T3361 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1391338247898/collection1
   [junit4]   2> 2055554 T3361 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 2055555 T3361 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2055555 T3361 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 2055556 T3361 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1391338247898/collection1/'
   [junit4]   2> 2055557 T3361 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1391338247898/collection1/lib/README' to classloader
   [junit4]   2> 2055558 T3361 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1391338247898/collection1/lib/classes/' to classloader
   [junit4]   2> 2055605 T3361 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 2055651 T3361 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2055752 T3361 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2055762 T3361 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 2056286 T3361 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2056288 T3361 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2056290 T3361 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2056294 T3361 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2056316 T3361 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 2056316 T3361 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1391338247898/collection1/, dataDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1391338244846/jetty1/
   [junit4]   2> 2056316 T3361 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7c2fbdd2
   [junit4]   2> 2056317 T3361 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1391338244846/jetty1
   [junit4]   2> 2056317 T3361 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1391338244846/jetty1/index/
   [junit4]   2> 2056317 T3361 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1391338244846/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 2056318 T3361 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1391338244846/jetty1/index
   [junit4]   2> 2056318 T3361 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@ac072f
   [junit4]   2> 2056319 T3361 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@5a7acf96 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1e2aaf9a),segFN=segments_1,generation=1}
   [junit4]   2> 2056319 T3361 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2056322 T3361 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 2056322 T3361 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 2056322 T3361 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 2056323 T3361 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2056323 T3361 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2056323 T3361 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 2056324 T3361 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2056324 T3361 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2056324 T3361 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2056325 T3361 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 2056325 T3361 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 2056325 T3361 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 2056326 T3361 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 2056326 T3361 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 2056326 T3361 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 2056327 T3361 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 2056457 T3361 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2056461 T3361 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 2056461 T3361 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 2056461 T3361 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=15, maxMergeAtOnceExplicit=22, maxMergedSegmentMB=87.7158203125, floorSegmentMB=1.3486328125, forceMergeDeletesPctAllowed=21.917569415090774, segmentsPerTier=27.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 2056462 T3361 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@5a7acf96 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1e2aaf9a),segFN=segments_1,generation=1}
   [junit4]   2> 2056462 T3361 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2056462 T3361 oass.SolrIndexSearcher.<init> Opening Searcher@333f3699 main
   [junit4]   2> 2056466 T3362 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@333f3699 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 2056468 T3361 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 2056469 T3316 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1
   [junit4]   2> 2056469 T3316 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2056469 T3365 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:36263 collection:collection1 shard:shard1
   [junit4]   2> 2056471 T3365 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 2056538 T3365 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 2056541 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2056541 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2056541 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2056541 T3365 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 2056541 T3365 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C7245 name=collection1 org.apache.solr.core.SolrCore@7b7e28ba url=http://127.0.0.1:36263/collection1 node=127.0.0.1:36263_ C7245_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:36263, core=collection1, node_name=127.0.0.1:36263_}
   [junit4]   2> 2056542 T3365 C7245 P36263 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:36263/collection1/
   [junit4]   2> 2056542 T3365 C7245 P36263 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 2056542 T3365 C7245 P36263 oasc.SyncStrategy.syncToMe http://127.0.0.1:36263/collection1/ has no replicas
   [junit4]   2> 2056542 T3365 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:36263/collection1/ shard1
   [junit4]   2> 2056542 T3365 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 2056542 T3339 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2056621 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2056629 T3347 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2056629 T3360 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2056629 T3338 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2056675 T3339 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2056727 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2056778 T3316 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 2056779 T3316 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2056782 T3316 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:15397
   [junit4]   2> 2056783 T3316 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2056783 T3316 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2056784 T3316 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1391338251179
   [junit4]   2> 2056784 T3316 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1391338251179/'
   [junit4]   2> 2056824 T3316 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1391338251179/solr.xml
   [junit4]   2> 2056831 T3338 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2056831 T3360 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2056831 T3347 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2056877 T3365 oasc.ZkController.register We are http://127.0.0.1:36263/collection1/ and leader is http://127.0.0.1:36263/collection1/
   [junit4]   2> 2056877 T3365 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:36263
   [junit4]   2> 2056877 T3365 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 2056877 T3365 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 2056878 T3365 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2056879 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2056879 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2056879 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2056879 T3365 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2056881 T3339 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2056881 T3339 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:36263",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:36263_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 2056905 T3316 oasc.CoreContainer.<init> New CoreContainer 1329954634
   [junit4]   2> 2056906 T3316 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1391338251179/]
   [junit4]   2> 2056908 T3316 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2056908 T3316 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 2056909 T3316 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2056909 T3316 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2056909 T3316 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2056910 T3316 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2056910 T3316 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2056911 T3316 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2056911 T3316 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2056911 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2056924 T3316 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2056925 T3316 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2056925 T3316 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2056926 T3316 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:63050/solr
   [junit4]   2> 2056926 T3316 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2056927 T3316 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2056929 T3376 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@718cf544 name:ZooKeeperConnection Watcher:127.0.0.1:63050 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2056930 T3316 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2056932 T3316 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2056939 T3378 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@60b601dc name:ZooKeeperConnection Watcher:127.0.0.1:63050/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2056939 T3316 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2056948 T3316 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2057014 T3338 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2057014 T3347 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2057014 T3360 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2057014 T3378 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2057952 T3316 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:15397_
   [junit4]   2> 2057958 T3316 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:15397_
   [junit4]   2> 2057961 T3338 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 2057961 T3360 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 2057961 T3347 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 2057961 T3378 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 2057969 T3379 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 2057970 T3379 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2057972 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2057972 T3379 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2057972 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2057972 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2057973 T3339 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2057974 T3339 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:15397",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:15397_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 2057974 T3339 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 2057975 T3339 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 2057978 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2058080 T3378 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2058081 T3360 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2058081 T3347 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2058080 T3338 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2058973 T3379 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 2058973 T3379 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1391338251179/collection1
   [junit4]   2> 2058973 T3379 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 2058974 T3379 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2058974 T3379 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 2058975 T3379 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1391338251179/collection1/'
   [junit4]   2> 2058976 T3379 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1391338251179/collection1/lib/README' to classloader
   [junit4]   2> 2058977 T3379 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1391338251179/collection1/lib/classes/' to classloader
   [junit4]   2> 2059024 T3379 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 2059070 T3379 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2059171 T3379 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2059181 T3379 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 2059580 T3379 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2059584 T3379 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2059586 T3379 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2059591 T3379 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2059611 T3379 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 2059612 T3379 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1391338251179/collection1/, dataDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1391338244846/jetty2/
   [junit4]   2> 2059612 T3379 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7c2fbdd2
   [junit4]   2> 2059613 T3379 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1391338244846/jetty2
   [junit4]   2> 2059613 T3379 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1391338244846/jetty2/index/
   [junit4]   2> 2059614 T3379 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1391338244846/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 2059615 T3379 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1391338244846/jetty2/index
   [junit4]   2> 2059615 T3379 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@72d983d
   [junit4]   2> 2059616 T3379 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@75661a68 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@56574c78),segFN=segments_1,generation=1}
   [junit4]   2> 2059616 T3379 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2059620 T3379 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 2059621 T3379 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 2059621 T3379 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 2059621 T3379 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2059622 T3379 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2059622 T3379 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 2059622 T3379 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2059623 T3379 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2059623 T3379 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2059624 T3379 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 2059624 T3379 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 2059625 T3379 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 2059625 T3379 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 2059626 T3379 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 2059626 T3379 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 2059627 T3379 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 2059637 T3379 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2059642 T3379 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 2059643 T3379 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 2059643 T3379 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=15, maxMergeAtOnceExplicit=22, maxMergedSegmentMB=87.7158203125, floorSegmentMB=1.3486328125, forceMergeDeletesPctAllowed=21.917569415090774, segmentsPerTier=27.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 2059644 T3379 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@75661a68 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@56574c78),segFN=segments_1,generation=1}
   [junit4]   2> 2059644 T3379 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2059645 T3379 oass.SolrIndexSearcher.<init> Opening Searcher@2d00b3ba main
   [junit4]   2> 2059649 T3380 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2d00b3ba main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 2059651 T3379 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 2059652 T3316 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1
   [junit4]   2> 2059653 T3316 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2059652 T3383 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:15397 collection:collection1 shard:shard2
   [junit4]   2> 2059655 T3383 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 2059697 T3383 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 2059708 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2059708 T3383 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 2059709 T3383 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 2059708 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> ASYNC  NEW_CORE C7246 name=collection1 org.apache.solr.core.SolrCore@6dd5819d url=http://127.0.0.1:15397/collection1 node=127.0.0.1:15397_ C7246_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:15397, core=collection1, node_name=127.0.0.1:15397_}
   [junit4]   2> 2059709 T3383 C7246 P15397 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:15397/collection1/
   [junit4]   2> 2059709 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2059710 T3339 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2059709 T3383 C7246 P15397 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 2059710 T3383 C7246 P15397 oasc.SyncStrategy.syncToMe http://127.0.0.1:15397/collection1/ has no replicas
   [junit4]   2> 2059710 T3383 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:15397/collection1/ shard2
   [junit4]   2> 2059710 T3383 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 2059713 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2059715 T3338 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2059715 T3360 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2059715 T3378 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2059715 T3347 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2059731 T3339 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2059734 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2059838 T3338 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2059838 T3360 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2059838 T3347 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2059838 T3378 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2059882 T3383 oasc.ZkController.register We are http://127.0.0.1:15397/collection1/ and leader is http://127.0.0.1:15397/collection1/
   [junit4]   2> 2059882 T3383 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:15397
   [junit4]   2> 2059882 T3383 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 2059882 T3383 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 2059882 T3383 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2059884 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2059884 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2059884 T3383 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2059884 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2059886 T3339 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2059886 T3339 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:15397",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:15397_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 2059895 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2059975 T3316 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 2059976 T3316 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2059979 T3316 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:39451
   [junit4]   2> 2059979 T3316 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2059980 T3316 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2059980 T3316 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1391338254363
   [junit4]   2> 2059981 T3316 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1391338254363/'
   [junit4]   2> 2059997 T3378 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2059998 T3347 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2059997 T3360 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2059997 T3338 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2060023 T3316 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1391338254363/solr.xml
   [junit4]   2> 2060107 T3316 oasc.CoreContainer.<init> New CoreContainer 607086511
   [junit4]   2> 2060108 T3316 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1391338254363/]
   [junit4]   2> 2060110 T3316 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2060110 T3316 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 2060111 T3316 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2060111 T3316 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2060112 T3316 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2060112 T3316 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2060113 T3316 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2060113 T3316 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2060114 T3316 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2060126 T3316 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2060127 T3316 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2060127 T3316 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2060128 T3316 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:63050/solr
   [junit4]   2> 2060128 T3316 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2060129 T3316 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2060132 T3394 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@244024b8 name:ZooKeeperConnection Watcher:127.0.0.1:63050 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2060133 T3316 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2060136 T3316 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2060142 T3396 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5ddb9340 name:ZooKeeperConnection Watcher:127.0.0.1:63050/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2060143 T3316 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2060147 T3316 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2061149 T3316 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39451_
   [junit4]   2> 2061204 T3316 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:39451_
   [junit4]   2> 2061236 T3347 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 2061236 T3396 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 2061236 T3378 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 2061236 T3338 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 2061236 T3360 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 2061258 T3397 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 2061258 T3397 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2061260 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2061260 T3397 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2061260 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2061260 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2061262 T3339 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2061262 T3339 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:39451",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:39451_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 2061263 T3339 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 2061263 T3339 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
   [junit4]   2> 2061280 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2061384 T3396 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 2061384 T3338 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 2061384 T3347 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 2061384 T3360 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 2061384 T3378 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 2062261 T3397 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 2062261 T3397 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1391338254363/collection1
   [junit4]   2> 2062261 T3397 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 2062262 T3397 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2062262 T3397 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 2062263 T3397 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1391338254363/collection1/'
   [junit4]   2> 2062265 T3397 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1391338254363/collection1/lib/README' to classloader
   [junit4]   2> 2062265 T3397 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1391338254363/collection1/lib/classes/' to classloader
   [junit4]   2> 2062328 T3397 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 2062397 T3397 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2062498 T3397 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2062508 T3397 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 2063060 T3397 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2063064 T3397 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2063066 T3397 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2063071 T3397 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2063094 T3397 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 2063094 T3397 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1391338254363/collection1/, dataDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1391338244846/jetty3/
   [junit4]   2> 2063094 T3397 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7c2fbdd2
   [junit4]   2> 2063095 T3397 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1391338244846/jetty3
   [junit4]   2> 2063096 T3397 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1391338244846/jetty3/index/
   [junit4]   2> 2063096 T3397 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1391338244846/jetty3/index' doesn't exist. Creating new index...
   [junit4]   2> 2063097 T3397 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1391338244846/jetty3/index
   [junit4]   2> 2063097 T3397 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@594a556b
   [junit4]   2> 2063098 T3397 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@10cc578d lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@a854f03),segFN=segments_1,generation=1}
   [junit4]   2> 2063098 T3397 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2063103 T3397 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 2063103 T3397 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 2063104 T3397 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 2063104 T3397 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2063104 T3397 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2063105 T3397 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 2063105 T3397 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2063105 T3397 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2063106 T3397 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2063106 T3397 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 2063107 T3397 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 2063107 T3397 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 2063108 T3397 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 2063108 T3397 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 2063109 T3397 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 2063109 T3397 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 2063118 T3397 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2063123 T3397 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 2063124 T3397 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 2063125 T3397 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=15, maxMergeAtOnceExplicit=22, maxMergedSegmentMB=87.7158203125, floorSegmentMB=1.3486328125, forceMergeDeletesPctAllowed=21.917569415090774, segmentsPerTier=27.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 2063125 T3397 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@10cc578d lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@a854f03),segFN=segments_1,generation=1}
   [junit4]   2> 2063126 T3397 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2063126 T3397 oass.SolrIndexSearcher.<init> Opening Searcher@63a3ced6 main
   [junit4]   2> 2063130 T3398 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@63a3ced6 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 2063133 T3397 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 2063133 T3316 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1
   [junit4]   2> 2063134 T3316 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2063133 T3401 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:39451 collection:collection1 shard:shard3
   [junit4]   2> 2063135 T3401 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
   [junit4]   2> 2063151 T3401 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard3
   [junit4]   2> 2063153 T3338 oasc.DistributedQueue$LatchCh

[...truncated too long message...]

stingsFormat(name=FSTOrdPulsing41), 10parent_f2_s=PostingsFormat(name=FSTOrdPulsing41), 15parent_f1_s=PostingsFormat(name=FSTOrdPulsing41), 28parent_f2_s=PostingsFormat(name=Asserting), 33parent_f1_s=PostingsFormat(name=FSTOrdPulsing41), 44parent_f1_s=PostingsFormat(name=FSTOrdPulsing41), 2grand1_s=PostingsFormat(name=Asserting), 7grand4_s=PostingsFormat(name=Asserting), 7parent_f2_s=PostingsFormat(name=FSTOrdPulsing41), 13parent_f2_s=PostingsFormat(name=Asserting), text_t=PostingsFormat(name=SimpleText), 4parent_f1_s=PostingsFormat(name=FSTOrdPulsing41), 12parent_f2_s=PostingsFormat(name=FSTOrdPulsing41), 11parent_f2_s=Pulsing41(freqCutoff=15 minBlockSize=18 maxBlockSize=64), 44parent_f2_s=Pulsing41(freqCutoff=15 minBlockSize=18 maxBlockSize=64), type_s=PostingsFormat(name=SimpleText), 7grand5_s=PostingsFormat(name=FSTOrdPulsing41), 15parent_f2_s=Pulsing41(freqCutoff=15 minBlockSize=18 maxBlockSize=64), 4grand3_s=PostingsFormat(name=Asserting), 46parent_f1_s=PostingsFormat(name=FSTOrdPulsing41), 46parent_f2_s=PostingsFormat(name=Asserting), 2child0_s=PostingsFormat(name=FSTOrdPulsing41), 1grand0_s=Pulsing41(freqCutoff=15 minBlockSize=18 maxBlockSize=64), 5grand0_s=Pulsing41(freqCutoff=15 minBlockSize=18 maxBlockSize=64), 36parent_f1_s=Pulsing41(freqCutoff=15 minBlockSize=18 maxBlockSize=64), 7grand2_s=Pulsing41(freqCutoff=15 minBlockSize=18 maxBlockSize=64), 5grand4_s=Pulsing41(freqCutoff=15 minBlockSize=18 maxBlockSize=64), 6grand1_s=PostingsFormat(name=Asserting), 45parent_f2_s=PostingsFormat(name=FSTOrdPulsing41), 36parent_f2_s=PostingsFormat(name=FSTOrdPulsing41), 6grand3_s=Pulsing41(freqCutoff=15 minBlockSize=18 maxBlockSize=64), 16parent_f2_s=PostingsFormat(name=FSTOrdPulsing41), 4grand1_s=Pulsing41(freqCutoff=15 minBlockSize=18 maxBlockSize=64), 9parent_f1_s=PostingsFormat(name=Asserting), 30parent_f1_s=PostingsFormat(name=Asserting), 5grand2_s=PostingsFormat(name=Asserting), 14parent_f2_s=PostingsFormat(name=FSTOrdPulsing41), 28parent_f1_s=PostingsFormat(name=FSTOrdPulsing41), a_t=PostingsFormat(name=FSTOrdPulsing41), 7grand0_s=PostingsFormat(name=Asserting), 37parent_f1_s=PostingsFormat(name=FSTOrdPulsing41), 6grand5_s=PostingsFormat(name=Asserting), 4child1_s=Pulsing41(freqCutoff=15 minBlockSize=18 maxBlockSize=64), _root_=PostingsFormat(name=SimpleText), 39parent_f2_s=PostingsFormat(name=Asserting), 6parent_f1_s=PostingsFormat(name=FSTOrdPulsing41), 4child3_s=PostingsFormat(name=Asserting), 6parent_f2_s=PostingsFormat(name=Asserting), 26parent_f1_s=PostingsFormat(name=FSTOrdPulsing41), 24parent_f1_s=PostingsFormat(name=FSTOrdPulsing41), 11parent_f1_s=PostingsFormat(name=FSTOrdPulsing41), 38parent_f1_s=PostingsFormat(name=Asserting), 27parent_f2_s=PostingsFormat(name=FSTOrdPulsing41), 34parent_f1_s=PostingsFormat(name=Asserting), 26parent_f2_s=Pulsing41(freqCutoff=15 minBlockSize=18 maxBlockSize=64), 20parent_f1_s=PostingsFormat(name=FSTOrdPulsing41), rnd_b=PostingsFormat(name=Asserting), 38parent_f2_s=PostingsFormat(name=FSTOrdPulsing41), 3child1_s=PostingsFormat(name=FSTOrdPulsing41), 1child0_s=Pulsing41(freqCutoff=15 minBlockSize=18 maxBlockSize=64), 22parent_f2_s=Pulsing41(freqCutoff=15 minBlockSize=18 maxBlockSize=64), 2parent_f1_s=PostingsFormat(name=FSTOrdPulsing41), 3parent_f2_s=PostingsFormat(name=FSTOrdPulsing41), 42parent_f2_s=PostingsFormat(name=Asserting), 21parent_f1_s=Pulsing41(freqCutoff=15 minBlockSize=18 maxBlockSize=64), 29parent_f1_s=Pulsing41(freqCutoff=15 minBlockSize=18 maxBlockSize=64), 27parent_f1_s=PostingsFormat(name=Asserting), 42parent_f1_s=PostingsFormat(name=FSTOrdPulsing41), 2parent_f2_s=PostingsFormat(name=Asserting), 32parent_f2_s=PostingsFormat(name=FSTOrdPulsing41), 4grand0_s=PostingsFormat(name=SimpleText), 43parent_f2_s=PostingsFormat(name=FSTOrdPulsing41), 18parent_f2_s=PostingsFormat(name=FSTOrdPulsing41), 34parent_f2_s=PostingsFormat(name=FSTOrdPulsing41), 41parent_f1_s=PostingsFormat(name=Asserting), 43parent_f1_s=Pulsing41(freqCutoff=15 minBlockSize=18 maxBlockSize=64), 12parent_f1_s=PostingsFormat(name=Asserting), 4parent_f2_s=Pulsing41(freqCutoff=15 minBlockSize=18 maxBlockSize=64), 31parent_f1_s=PostingsFormat(name=FSTOrdPulsing41), 41parent_f2_s=PostingsFormat(name=FSTOrdPulsing41), 7parent_f1_s=Pulsing41(freqCutoff=15 minBlockSize=18 maxBlockSize=64), 18parent_f1_s=Pulsing41(freqCutoff=15 minBlockSize=18 maxBlockSize=64), 6grand2_s=PostingsFormat(name=SimpleText), 8parent_f1_s=PostingsFormat(name=FSTOrdPulsing41), 7grand6_s=Pulsing41(freqCutoff=15 minBlockSize=18 maxBlockSize=64), 6grand4_s=PostingsFormat(name=FSTOrdPulsing41), 7grand3_s=PostingsFormat(name=FSTOrdPulsing41), 39parent_f1_s=PostingsFormat(name=FSTOrdPulsing41), 2child1_s=PostingsFormat(name=Asserting), 45parent_f1_s=PostingsFormat(name=Asserting), 8parent_f2_s=Pulsing41(freqCutoff=15 minBlockSize=18 maxBlockSize=64), 13parent_f1_s=PostingsFormat(name=FSTOrdPulsing41), id=PostingsFormat(name=Asserting), 7grand1_s=PostingsFormat(name=FSTOrdPulsing41), 3parent_f1_s=Pulsing41(freqCutoff=15 minBlockSize=18 maxBlockSize=64), 31parent_f2_s=PostingsFormat(name=Asserting), 4grand2_s=PostingsFormat(name=FSTOrdPulsing41), 14parent_f1_s=Pulsing41(freqCutoff=15 minBlockSize=18 maxBlockSize=64), 5parent_f2_s=PostingsFormat(name=FSTOrdPulsing41), 35parent_f1_s=PostingsFormat(name=FSTOrdPulsing41), _version_=PostingsFormat(name=SimpleText), 5grand3_s=PostingsFormat(name=FSTOrdPulsing41), 4child2_s=PostingsFormat(name=SimpleText), 22parent_f1_s=PostingsFormat(name=FSTOrdPulsing41), 19parent_f2_s=Pulsing41(freqCutoff=15 minBlockSize=18 maxBlockSize=64), 30parent_f2_s=PostingsFormat(name=FSTOrdPulsing41), 5grand1_s=PostingsFormat(name=SimpleText), 5parent_f1_s=PostingsFormat(name=Asserting), 4child0_s=PostingsFormat(name=FSTOrdPulsing41), 17parent_f1_s=PostingsFormat(name=FSTOrdPulsing41), 35parent_f2_s=PostingsFormat(name=Asserting), 17parent_f2_s=PostingsFormat(name=Asserting), 40parent_f1_s=PostingsFormat(name=FSTOrdPulsing41), 16parent_f1_s=PostingsFormat(name=Asserting)}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=true,coord=yes): {}, locale=uk, timezone=Asia/Samarkand
   [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Sun Microsystems Inc. 1.6.0_32 (64-bit)/cpus=16,threads=2,free=323043584,total=476643328
   [junit4]   2> NOTE: All tests run in this JVM: [TestQuerySenderListener, OutputWriterTest, StressHdfsTest, HdfsDirectoryTest, AddBlockUpdateTest, AlternateDirectoryTest, TestSolr4Spatial, TestCustomSort, SolrCoreCheckLockOnStartupTest, TestZkChroot, RAMDirectoryFactoryTest, PingRequestHandlerTest, NumericFieldsTest, ShardRoutingCustomTest, ShardSplitTest, BlockDirectoryTest, DocValuesMissingTest, CursorPagingTest, TestReload, BasicDistributedZk2Test, TestUpdate, DocumentAnalysisRequestHandlerTest, SolrTestCaseJ4Test, TestDefaultSimilarityFactory, HighlighterTest, TestNoOpRegenerator, StandardRequestHandlerTest, SortByFunctionTest, SolrIndexConfigTest, TestSolrDeletionPolicy2, TestRecovery, TestStressRecovery, TestCSVLoader, TestSchemaResource, DirectSolrConnectionTest, FieldAnalysisRequestHandlerTest, ChaosMonkeySafeLeaderTest, TestDistribDocBasedVersion, TestFieldTypeResource, TestComponentsName, SuggestComponentTest, CursorMarkTest, TestDistributedSearch, HdfsRecoveryZkTest, TestNRTOpen, XmlUpdateRequestHandlerTest, TestDFRSimilarityFactory, CurrencyFieldOpenExchangeTest, HdfsUnloadDistributedZkTest, LeaderElectionIntegrationTest, TestSimpleQParserPlugin, TestAtomicUpdateErrorCases, StatsComponentTest, TestIndexingPerformance, DirectSolrSpellCheckerTest, TestCollationKeyRangeQueries, TestSolrJ, TestCursorMarkWithoutUniqueKey, HighlighterConfigTest, HardAutoCommitTest, TestCloudManagedSchemaAddField, TestFuzzyAnalyzedSuggestions, UniqFieldsUpdateProcessorFactoryTest, TestUtils, TestFastWriter, TestJmxIntegration, CachingDirectoryFactoryTest, TestFileDictionaryLookup, PathHierarchyTokenizerFactoryTest, OpenCloseCoreStressTest, FullSolrCloudDistribCmdsTest]
   [junit4] Completed on J1 in 368.72s, 1 test, 1 error <<< FAILURES!

[...truncated 583 lines...]
   [junit4] JVM J1: stdout was not empty, see: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/temp/junit4-J1-20140202_101630_865.sysout
   [junit4] >>> JVM J1: stdout (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4] Dumping heap to /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/heapdumps/java_pid78088.hprof ...
   [junit4] Heap dump file created [526415055 bytes in 17.294 secs]
   [junit4] <<< JVM J1: EOF ----

[...truncated 24 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/build.xml:466: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/build.xml:439: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/extra-targets.xml:37: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build.xml:189: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/common-build.xml:491: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/common-build.xml:1308: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/common-build.xml:941: There were test failures: 355 suites, 1549 tests, 1 error, 33 ignored (5 assumptions)

Total time: 179 minutes 13 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure