You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/12/20 01:32:12 UTC

[JENKINS] Lucene-Solr-trunk-Windows (32bit/jdk1.8.0_20) - Build # 4501 - Still Failing!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Windows/4501/
Java: 32bit/jdk1.8.0_20 -server -XX:+UseSerialGC (asserts: true)

1 tests failed.
FAILED:  org.apache.lucene.facet.TestRandomSamplingFacetsCollector.testRandomSampling

Error Message:
54

Stack Trace:
java.lang.ArrayIndexOutOfBoundsException: 54
	at __randomizedtesting.SeedInfo.seed([36B2EF8F9284ED2B:DCAA869223D31A9C]:0)
	at org.apache.lucene.facet.TestRandomSamplingFacetsCollector.testRandomSampling(TestRandomSamplingFacetsCollector.java:133)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 5831 lines...]
   [junit4] Suite: org.apache.lucene.facet.TestRandomSamplingFacetsCollector
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestRandomSamplingFacetsCollector -Dtests.method=testRandomSampling -Dtests.seed=36B2EF8F9284ED2B -Dtests.slow=true -Dtests.locale=es_GT -Dtests.timezone=America/Araguaina -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   1.08s | TestRandomSamplingFacetsCollector.testRandomSampling <<<
   [junit4]    > Throwable #1: java.lang.ArrayIndexOutOfBoundsException: 54
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([36B2EF8F9284ED2B:DCAA869223D31A9C]:0)
   [junit4]    > 	at org.apache.lucene.facet.TestRandomSamplingFacetsCollector.testRandomSampling(TestRandomSamplingFacetsCollector.java:133)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50): {$full_path$=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), $facets=PostingsFormat(name=LuceneFixedGap), $payloads$=Lucene50(blocksize=128), EvenOdd=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128)))}, docValues:{$facets=DocValuesFormat(name=Asserting)}, sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {EvenOdd=IB SPL-L3(800.0)}, locale=es_GT, timezone=America/Araguaina
   [junit4]   2> NOTE: Windows 7 6.1 x86/Oracle Corporation 1.8.0_20 (32-bit)/cpus=2,threads=1,free=16838256,total=58535936
   [junit4]   2> NOTE: All tests run in this JVM: [TestDrillDownQuery, TestTaxonomyFacetCounts, TestTaxonomyFacetSumValueSource, TestSearcherTaxonomyManager, TestRandomSamplingFacetsCollector]
   [junit4] Completed in 1.14s, 1 test, 1 error <<< FAILURES!

[...truncated 63 lines...]
BUILD FAILED
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:519: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:467: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:61: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\extra-targets.xml:39: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build.xml:456: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:2070: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\module-build.xml:58: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:1349: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:956: There were test failures: 23 suites, 154 tests, 1 error

Total time: 23 minutes 43 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 32bit/jdk1.8.0_20 -server -XX:+UseSerialGC (asserts: true)
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Windows (64bit/jdk1.8.0_40-ea-b09) - Build # 4503 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Windows/4503/
Java: 64bit/jdk1.8.0_40-ea-b09 -XX:+UseCompressedOops -XX:+UseSerialGC (asserts: false)

1 tests failed.
FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testDistribSearch

Error Message:
expected:<...F3081D5F614AC-001\te[mpDir-003\solrj_test_core_props_shard1_replica1]> but was:<...F3081D5F614AC-001\te[stPropertyParamsForCreate-001\instanceDir-bvc]>

Stack Trace:
org.junit.ComparisonFailure: expected:<...F3081D5F614AC-001\te[mpDir-003\solrj_test_core_props_shard1_replica1]> but was:<...F3081D5F614AC-001\te[stPropertyParamsForCreate-001\instanceDir-bvc]>
	at __randomizedtesting.SeedInfo.seed([408F3081D5F614AC:C169BE99A2A97490]:0)
	at org.junit.Assert.assertEquals(Assert.java:125)
	at org.junit.Assert.assertEquals(Assert.java:147)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.checkInstanceDirs(CollectionsAPIDistributedZkTest.java:1154)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testCollectionsAPI(CollectionsAPIDistributedZkTest.java:901)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:205)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
	at sun.reflect.GeneratedMethodAccessor62.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 9413 lines...]
   [junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
   [junit4]   2> Creating dataDir: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-408F3081D5F614AC-001\init-core-data-001
   [junit4]   2> 3861699 T11177 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (true)
   [junit4]   2> 3861699 T11177 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /wxx/
   [junit4]   2> 3861702 T11177 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 3861704 T11177 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 3861705 T11178 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 3861803 T11177 oasc.ZkTestServer.run start zk server on port:58775
   [junit4]   2> 3861803 T11177 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 3861806 T11177 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3861812 T11185 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@785ecc8c name:ZooKeeperConnection Watcher:127.0.0.1:58775 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3861812 T11177 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3861812 T11177 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 3861812 T11177 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 3861820 T11177 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 3861822 T11177 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3861825 T11188 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7e276940 name:ZooKeeperConnection Watcher:127.0.0.1:58775/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3861825 T11177 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3861825 T11177 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 3861826 T11177 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 3861831 T11177 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 3861835 T11177 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 3861838 T11177 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 3861843 T11177 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 3861844 T11177 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 3861851 T11177 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 3861851 T11177 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 3861856 T11177 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 3861856 T11177 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 3861863 T11177 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 3861863 T11177 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 3861869 T11177 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 3861869 T11177 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 3861873 T11177 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 3861873 T11177 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 3861879 T11177 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 3861879 T11177 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 3861885 T11177 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 3861887 T11177 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 3861894 T11177 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 3861894 T11177 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 3861899 T11177 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 3861899 T11177 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 3861908 T11177 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 3861909 T11177 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 3861916 T11177 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 3861919 T11177 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3861921 T11191 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6e8d0b3b name:ZooKeeperConnection Watcher:127.0.0.1:58775/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3861922 T11177 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3861922 T11177 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> Using old style solr.xml
   [junit4]   2> 3861925 T11177 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 3861928 T11177 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3861930 T11194 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@158faafd name:ZooKeeperConnection Watcher:127.0.0.1:58775 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3861931 T11177 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3861931 T11177 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 3861931 T11177 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 3861936 T11177 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 3861938 T11177 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3861941 T11197 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@98f9512 name:ZooKeeperConnection Watcher:127.0.0.1:58775/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3861942 T11177 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3861942 T11177 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 3861944 T11177 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\solrconfig-tlog.xml to /configs/conf2/solrconfig.xml
   [junit4]   2> 3861944 T11177 oascc.SolrZkClient.makePath makePath: /configs/conf2/solrconfig.xml
   [junit4]   2> 3861948 T11177 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\schema.xml to /configs/conf2/schema.xml
   [junit4]   2> 3861949 T11177 oascc.SolrZkClient.makePath makePath: /configs/conf2/schema.xml
   [junit4]   2> 3861953 T11177 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\solrconfig.snippet.randomindexconfig.xml to /configs/conf2/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 3861953 T11177 oascc.SolrZkClient.makePath makePath: /configs/conf2/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 3861958 T11177 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\stopwords.txt to /configs/conf2/stopwords.txt
   [junit4]   2> 3861958 T11177 oascc.SolrZkClient.makePath makePath: /configs/conf2/stopwords.txt
   [junit4]   2> 3861962 T11177 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\protwords.txt to /configs/conf2/protwords.txt
   [junit4]   2> 3861962 T11177 oascc.SolrZkClient.makePath makePath: /configs/conf2/protwords.txt
   [junit4]   2> 3861967 T11177 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\currency.xml to /configs/conf2/currency.xml
   [junit4]   2> 3861967 T11177 oascc.SolrZkClient.makePath makePath: /configs/conf2/currency.xml
   [junit4]   2> 3861972 T11177 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\open-exchange-rates.json to /configs/conf2/open-exchange-rates.json
   [junit4]   2> 3861972 T11177 oascc.SolrZkClient.makePath makePath: /configs/conf2/open-exchange-rates.json
   [junit4]   2> 3861977 T11177 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\mapping-ISOLatin1Accent.txt to /configs/conf2/mapping-ISOLatin1Accent.txt
   [junit4]   2> 3861977 T11177 oascc.SolrZkClient.makePath makePath: /configs/conf2/mapping-ISOLatin1Accent.txt
   [junit4]   2> 3861982 T11177 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\old_synonyms.txt to /configs/conf2/old_synonyms.txt
   [junit4]   2> 3861982 T11177 oascc.SolrZkClient.makePath makePath: /configs/conf2/old_synonyms.txt
   [junit4]   2> 3861987 T11177 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\synonyms.txt to /configs/conf2/synonyms.txt
   [junit4]   2> 3861987 T11177 oascc.SolrZkClient.makePath makePath: /configs/conf2/synonyms.txt
   [junit4]   2> 3861991 T11177 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\elevate.xml to /configs/conf2/elevate.xml
   [junit4]   2> 3861991 T11177 oascc.SolrZkClient.makePath makePath: /configs/conf2/elevate.xml
   [junit4]   2> 3861999 T11179 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x14a696e7a6f0004, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 3863743 T11177 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 3863758 T11177 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 3863761 T11177 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:58791
   [junit4]   2> 3863764 T11177 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@c387f44
   [junit4]   2> 3863765 T11177 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 3863765 T11177 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-408F3081D5F614AC-001\tempDir-002
   [junit4]   2> 3863766 T11177 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-408F3081D5F614AC-001\tempDir-002\'
   [junit4]   2> 3863804 T11177 oasc.ConfigSolr.fromFile Loading container configuration from C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-408F3081D5F614AC-001\tempDir-002\solr.xml
   [junit4]   2> 3863843 T11177 oasc.CoreContainer.<init> New CoreContainer 1861171130
   [junit4]   2> 3863844 T11177 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-408F3081D5F614AC-001\tempDir-002\]
   [junit4]   2> 3863845 T11177 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 3863845 T11177 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 3863845 T11177 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 3863846 T11177 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 3863846 T11177 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 3863846 T11177 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 3863847 T11177 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 3863847 T11177 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 3863847 T11177 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 3863848 T11177 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 3863853 T11177 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=3400000&connTimeout=45000&retry=false
   [junit4]   2> 3863856 T11177 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 3863856 T11177 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 3863857 T11177 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 3863857 T11177 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:58775/solr
   [junit4]   2> 3863857 T11177 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 3863858 T11177 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 3863860 T11177 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3863862 T11209 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@537d7f8e name:ZooKeeperConnection Watcher:127.0.0.1:58775 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3863863 T11177 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3863864 T11177 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 3863868 T11177 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3863870 T11212 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@392e4780 name:ZooKeeperConnection Watcher:127.0.0.1:58775/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3863872 T11177 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3863873 T11177 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 3863878 T11177 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 3863883 T11177 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 3863888 T11177 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 3863893 T11177 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 3863899 T11177 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 3863902 T11177 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:58791_wxx
   [junit4]   2> 3863903 T11177 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:58791_wxx
   [junit4]   2> 3863907 T11177 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 3863910 T11177 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 3863913 T11177 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 3863916 T11177 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:58791_wxx
   [junit4]   2> 3863916 T11177 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 3863920 T11177 oasc.Overseer.start Overseer (id=93002665535930374-127.0.0.1:58791_wxx-n_0000000000) starting
   [junit4]   2> 3863924 T11177 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 3863942 T11177 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 3863943 T11214 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 3863945 T11177 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 3863950 T11177 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 3863954 T11177 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 3863965 T11213 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 3863968 T11177 oass.SolrDispatchFilter.init user.dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0
   [junit4]   2> 3863968 T11177 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 3863978 T11177 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 3863980 T11177 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3863982 T11219 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@55016e42 name:ZooKeeperConnection Watcher:127.0.0.1:58775/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3863983 T11177 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3863983 T11177 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 3863986 T11177 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 3863994 T11177 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 3863994 T11177 oasc.AbstractFullDistribZkTestBase.createJettys Creating collection1 with stateFormat=2
   [junit4]   2> 3863994 T11177 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 3863996 T11177 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3863999 T11222 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d9a83a3 name:ZooKeeperConnection Watcher:127.0.0.1:58775/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3863999 T11177 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3863999 T11177 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 3864003 T11212 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 3864005 T11179 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x14a696e7a6f0008, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 3864012 T11213 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 0, message = {
   [junit4]   2> 	  "operation":"create",
   [junit4]   2> 	  "name":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "stateFormat":2} current state version: 0
   [junit4]   2> 3864012 T11213 oasco.ClusterStateMutator.createCollection building a new cName: collection1
   [junit4]   2> 3864014 T11213 oasco.ZkStateWriter.writePendingUpdates going to create_collection /collections/collection1/state.json
   [junit4]   2> 3864015 T11212 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 3864016 T11219 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 3865967 T11177 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 3865969 T11177 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 3865979 T11177 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 3865980 T11177 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:58806
   [junit4]   2> 3865987 T11177 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@c387f44
   [junit4]   2> 3865987 T11177 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 3865987 T11177 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-408F3081D5F614AC-001\tempDir-003
   [junit4]   2> 3865988 T11177 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-408F3081D5F614AC-001\tempDir-003\'
   [junit4]   2> 3866022 T11177 oasc.ConfigSolr.fromFile Loading container configuration from C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-408F3081D5F614AC-001\tempDir-003\solr.xml
   [junit4]   2> 3866038 T11177 oasc.CoreContainer.<init> New CoreContainer 386133806
   [junit4]   2> 3866039 T11177 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-408F3081D5F614AC-001\tempDir-003\]
   [junit4]   2> 3866039 T11177 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 3866040 T11177 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 3866040 T11177 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 3866040 T11177 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 3866040 T11177 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 3866041 T11177 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 3866041 T11177 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 3866041 T11177 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 3866041 T11177 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 3866042 T11177 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 3866046 T11177 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=3400000&connTimeout=45000&retry=false
   [junit4]   2> 3866049 T11177 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 3866049 T11177 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 3866049 T11177 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 3866050 T11177 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:58775/solr
   [junit4]   2> 3866050 T11177 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 3866051 T11177 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 3866052 T11177 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3866055 T11234 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3577a506 name:ZooKeeperConnection Watcher:127.0.0.1:58775 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3866056 T11177 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3866056 T11177 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 3866059 T11179 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x14a696e7a6f0009, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 3866061 T11177 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3866065 T11237 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4d06db19 name:ZooKeeperConnection Watcher:127.0.0.1:58775/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3866066 T11177 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3866086 T11177 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 3867175 T11177 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:58806_wxx
   [junit4]   2> 3867178 T11177 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:58806_wxx
   [junit4]   2> 3867191 T11177 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 3867198 T11177 oass.SolrDispatchFilter.init user.dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0
   [junit4]   2> 3867199 T11177 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 3869019 T11177 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 3869020 T11177 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 3869031 T11177 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 3869033 T11177 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:58815
   [junit4]   2> 3869038 T11177 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@c387f44
   [junit4]   2> 3869039 T11177 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 3869039 T11177 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-408F3081D5F614AC-001\tempDir-004
   [junit4]   2> 3869040 T11177 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-408F3081D5F614AC-001\tempDir-004\'
   [junit4]   2> 3869082 T11177 oasc.ConfigSolr.fromFile Loading container configuration from C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-408F3081D5F614AC-001\tempDir-004\solr.xml
   [junit4]   2> 3869096 T11177 oasc.CoreContainer.<init> New CoreContainer 1522008760
   [junit4]   2> 3869097 T11177 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-408F3081D5F614AC-001\tempDir-004\]
   [junit4]   2> 3869097 T11177 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 3869098 T11177 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 3869098 T11177 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 3869098 T11177 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 3869099 T11177 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 3869099 T11177 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 3869099 T11177 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 3869100 T11177 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 3869100 T11177 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 3869100 T11177 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 3869104 T11177 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=3400000&connTimeout=45000&retry=false
   [junit4]   2> 3869107 T11177 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 3869108 T11177 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 3869108 T11177 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 3869108 T11177 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:58775/solr
   [junit4]   2> 3869109 T11177 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 3869109 T11177 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 3869111 T11177 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3869114 T11250 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6c53b511 name:ZooKeeperConnection Watcher:127.0.0.1:58775 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3869116 T11177 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3869117 T11177 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 3869119 T11179 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x14a696e7a6f000b, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 3869121 T11177 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3869123 T11253 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@424a9746 name:ZooKeeperConnection Watcher:127.0.0.1:58775/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3869123 T11177 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3869137 T11177 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 3870227 T11177 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:58815_wxx
   [junit4]   2> 3870230 T11177 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:58815_wxx
   [junit4]   2> 3870244 T11177 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 3870253 T11177 oass.SolrDispatchFilter.init user.dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0
   [junit4]   2> 3870253 T11177 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 3872301 T11177 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 3872301 T11177 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 3872321 T11177 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 3872324 T11177 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:58824
   [junit4]   2> 3872331 T11177 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@c387f44
   [junit4]   2> 3872332 T11177 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 3872332 T11177 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-408F3081D5F614AC-001\tempDir-005
   [junit4]   2> 3872333 T11177 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-408F3081D5F614AC-001\tempDir-005\'
   [junit4]   2> 3872389 T11177 oasc.ConfigSolr.fromFile Loading container configuration from C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-408F3081D5F614AC-001\tempDir-005\solr.xml
   [junit4]   2> 3872409 T11177 oasc.CoreContainer.<init> New CoreContainer 1329952549
   [junit4]   2> 3872410 T11177 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-408F3081D5F614AC-001\tempDir-005\]
   [junit4]   2> 3872411 T11177 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 3872411 T11177 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 3872412 T11177 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 3872412 T11177 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 3872413 T11177 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 3872413 T11177 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 3872413 T11177 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 3872414 T11177 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 3872414 T11177 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 3872415 T11177 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 3872421 T11177 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=3400000&connTimeout=45000&retry=false
   [junit4]   2> 3872426 T11177 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 3872426 T11177 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 3872427 T11177 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 3872427 T11177 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:58775/solr
   [junit4]   2> 3872428 T11177 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 3872428 T11177 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 3872431 T11177 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3872433 T11266 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@203db165 name:ZooKeeperConnection Watcher:127.0.0.1:58775 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3872434 T11177 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3872435 T11177 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 3872440 T11177 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3872442 T11269 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3d1040d8 name:ZooKeeperConnection Watcher:127.0.0.1:58775/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3872443 T11177 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3872455 T11177 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 3873545 T11177 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:58824_wxx
   [junit4]   2> 3873549 T11177 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:58824_wxx
   [junit4]   2> 3873560 T11177 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 3873567 T11177 oass.SolrDispatchFilter.init user.dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0
   [junit4]   2> 3873567 T11177 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 3875768 T11177 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 4
   [junit4]   2> 3875769 T11177 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 3875792 T11177 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 3875797 T11177 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:58833
   [junit4]   2> 3875803 T11177 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@c387f44
   [junit4]   2> 3875803 T11177 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 3875803 T11177 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-408F3081D5F614AC-001\tempDir-006
   [junit4]   2> 3875803 T11177 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-408F3081D5F614AC-001\tempDir-006\'
   [junit4]   2> 3875841 T11177 oasc.ConfigSolr.fromFile Loading container configuration from C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-408F3081D5F614AC-001\tempDir-006\solr.xml
   [junit4]   2> 3875899 T11177 oasc.CoreContainer.<init> New CoreContainer 306937458
   [junit4]   2> 3875899 T11177 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-408F3081D5F614AC-001\tempDir-006\]
   [junit4]   2> 3875900 T11177 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 3875900 T11177 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 3875900 T11177 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 3875901 T11177 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 3875901 T11177 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 3875901 T11177 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 3875901 T11177 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 3875902 T11177 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 3875902 T11177 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 3875902 T11177 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 3875907 T11177 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=3400000&connTimeout=45000&retry=false
   [junit4]   2> 3875910 T11177 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 3875911 T11177 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 3875911 T11177 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 3875911 T11177 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:58775/solr
   [junit4]   2> 3875913 T11177 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 3875913 T11177 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 3875915 T11177 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3875922 T11282 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@63b90e9d name:ZooKeeperConnection Watcher:127.0.0.1:58775 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3875923 T11177 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3875923 T11177 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 3875926 T11177 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3875929 T11285 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6e28cd9e name:ZooKeeperConnection Watcher:127.0.0.1:58775/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3875929 T11177 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3875941 T11177 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 3877142 T11177 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:58833_wxx
   [junit4]   2> 3877144 T11177 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:58833_wxx
   [junit4]   2> 3877156 T11177 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 3877164 T11177 oass.SolrDispatchFilter.init user.dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0
   [junit4]   2> 3877166 T11177 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 3877226 T11230 oasha.CollectionsHandler.handleCreateAction Creating Collection : replicationFactor=2&collection.configName=conf1&router.field=myOwnField&name=solrj_collection&action=CREATE&numShards=2&wt=javabin&version=2
   [junit4]   2> 3877228 T11212 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/collection-queue-work state SyncConnected
   [junit4]   2> 3877232 T11214 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"create",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"solrj_collection",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "collection.configName":"conf1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "router.field":"myOwnField"}
   [junit4]   2> 3877234 T11287 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : create , {
   [junit4]   2> 	  "operation":"create",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"solrj_collection",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "collection.configName":"conf1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "router.field":"myOwnField"}
   [junit4]   2> 3877241 T11287 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/solrj_collection 
   [junit4]   2> 3877243 T11287 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection
   [junit4]   2> 3877251 T11212 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 3877254 T11213 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"create",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"solrj_collection",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "collection.configName":"conf1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "router.field":"myOwnField"} current state version: 1
   [junit4]   2> 3877255 T11213 oasco.ClusterStateMutator.createCollection building a new cName: solrj_collection
   [junit4]   2> 3877258 T11219 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 3877259 T11212 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 3877259 T11237 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 3877261 T11285 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 3877261 T11253 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 3877262 T11269 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 3877491 T11287 oasc.OverseerCollectionProcessor.createCollection Creating SolrCores for new collection solrj_collection, shardNames [shard1, shard2] , replicationFactor : 2
   [junit4]   2> 3877491 T11287 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard1_replica1 as part of slice shard1 of collection solrj_collection on 127.0.0.1:58824_wxx
   [junit4]   2> 3877493 T11287 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard1_replica2 as part of slice shard1 of collection solrj_collection on 127.0.0.1:58833_wxx
   [junit4]   2> 3877500 T11287 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard2_replica1 as part of slice shard2 of collection solrj_collection on 127.0.0.1:58791_wxx
   [junit4]   2> 3877559 T11259 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&collection.configName=conf1&name=solrj_collection_shard1_replica1&action=CREATE&numShards=2&collection=solrj_collection&shard=shard1&wt=javabin&version=2
   [junit4]   2> 3877565 T11287 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard2_replica2 as part of slice shard2 of collection solrj_collection on 127.0.0.1:58815_wxx
   [junit4]   2> 3877576 T11279 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&collection.configName=conf1&name=solrj_collection_shard1_replica2&action=CREATE&numShards=2&collection=solrj_collection&shard=shard1&wt=javabin&version=2
   [junit4]   2> 3877583 T11259 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica1 state=down collection=solrj_collection
   [junit4]   2> 3877591 T11212 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 3877605 T11279 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica2 state=down collection=solrj_collection
   [junit4]   2> 3877591 T11259 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 3877623 T11279 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 3877636 T11204 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&collection.configName=conf1&name=solrj_collection_shard2_replica1&action=CREATE&numShards=2&collection=solrj_collection&shard=shard2&wt=javabin&version=2
   [junit4]   2> 3877644 T11213 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 2, message = {
   [junit4]   2> 	  "core":"solrj_collection_shard1_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:58824/wxx",
   [junit4]   2> 	  "node_name":"127.0.0.1:58824_wxx",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "operation":"state"} current state version: 2
   [junit4]   2> 3877649 T11246 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&collection.configName=conf1&name=solrj_collection_shard2_replica2&action=CREATE&numShards=2&collection=solrj_collection&shard=shard2&wt=javabin&version=2
   [junit4]   2> 3877651 T11213 oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"solrj_collection_shard1_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:58824/wxx",
   [junit4]   2> 	  "node_name":"127.0.0.1:58824_wxx",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 3877652 T11204 oasc.ZkController.publish publishing core=solrj_collection_shard2_replica1 state=down collection=solrj_collection
   [junit4]   2> 3877654 T11204 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 3877657 T11246 oasc.ZkController.publish publishing core=solrj_collection_shard2_replica2 state=down collection=solrj_collection
   [junit4]   2> 3877659 T11246 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 3877661 T11213 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 3, message = {
   [junit4]   2> 	  "core":"solrj_collection_shard1_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:58833/wxx",
   [junit4]   2> 	  "node_name":"127.0.0.1:58833_wxx",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "operation":"state"} current state version: 2
   [junit4]   2> 3877665 T11213 oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"solrj_collection_shard1_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:58833/wxx",
   [junit4]   2> 	  "node_name":"127.0.0.1:58833_wxx",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 3877671 T11212 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 3877672 T11213 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 3, message = {
   [junit4]   2> 	  "core":"solrj_collection_shard2_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:58791/wxx",
   [junit4]   2> 	  "node_name":"127.0.0.1:58791_wxx",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "operation":"state"} current state version: 2
   [junit4]   2> 3877674 T11213 oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"solrj_collection_shard2_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:58791/wxx",
   [junit4]   2> 	  "node_name":"127.0.0.1:58791_wxx",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 3877681 T11213 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"solrj_collection_shard2_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:58815/wxx",
   [junit4]   2> 	  "node_name":"127.0.0.1:58815_wxx",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "operation":"state"} current state version: 2
   [junit4]   2> 3877684 T11213 oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"solrj_collection_shard2_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:58815/wxx",
   [junit4]   2> 	  "node_name":"127.0.0.1:58815_wxx",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 3877689 T11212 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 3877691 T11212 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 3877693 T11219 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 3877693 T11237 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 3877693 T11285 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 3877694 T11253 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 3877694 T11269 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 3878792 T11259 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard1_replica1
   [junit4]   2> 3878792 T11259 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 3878793 T11259 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 3878794 T11259 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 3878795 T11259 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 3878796 T11259 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-408F3081D5F614AC-001\tempDir-005\solrj_collection_shard1_replica1\'
   [junit4]   2> 3878818 T11279 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard1_replica2
   [junit4]   2> 3878818 T11279 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 3878821 T11279 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 3878821 T11279 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 3878823 T11279 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 3878823 T11279 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-408F3081D5F614AC-001\tempDir-006\solrj_collection_shard1_replica2\'
   [junit4]   2> 3878825 T11259 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 3878833 T11259 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 3878848 T11279 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 3878855 T11204 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard2_replica1
   [junit4]   2> 3878855 T11204 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 3878856 T11246 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard2_replica2
   [junit4]   2> 3878856 T11246 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 3878860 T11279 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 3878861 T11204 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 3878862 T11204 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 3878862 T11246 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 3878862 T11246 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 3878866 T11259 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 3878878 T11246 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 3878878 T11246 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-408F3081D5F614AC-001\tempDir-004\solrj_collection_shard2_replica2\'
   [junit4]   2> 3878884 T11279 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 3878878 T11259 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 3878889 T11204 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 3878893 T11204 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-408F3081D5F614AC-001\tempDir-002\solrj_collection_shard2_replica1\'
   [junit4]   2> 3878905 T11279 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 3878912 T11259 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 3878919 T11259 oass.IndexSchema.readSchema [solrj_collection_shard1_replica1] Schema name=test
   [junit4]   2> 3878925 T11246 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 3878971 T11279 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 3878977 T11204 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 3878983 T11279 oass.IndexSchema.readSchema [solrj_collection_shard1_replica2] Schema name=test
   [junit4]   2> 3879117 T11279 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 3879124 T11259 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 3879124 T11279 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 3879126 T11279 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 3879130 T11259 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 3879132 T11259 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 3879136 T11246 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 3879138 T11204 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 3879159 T11204 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 3879192 T11246 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 3879192 T11204 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 3879205 T11204 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 3879205 T11246 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 3879215 T11204 oass.IndexSchema.readSchema [solrj_collection_shard2_replica1] Schema name=test
   [junit4]   2> 3879217 T11246 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 3879222 T11246 oass.IndexSchema.readSchema [solrj_collection_shard2_replica2] Schema name=test
   [junit4]   2> 3879393 T11204 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 3879397 T11204 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 3879398 T11204 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 3879412 T11246 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 3879412 T11279 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 3879418 T11246 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 3879419 T11259 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 3879420 T11246 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 3879423 T11279 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 3879429 T11279 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 3879430 T11259 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 3879432 T11279 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 3879437 T11279 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 3879438 T11279 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 3879441 T11259 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 3879445 T11279 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 3879446 T11279 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 3879447 T11279 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard1_replica2' using configuration from collection solrj_collection
   [junit4]   2> 3879447 T11279 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 3879446 T11259 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 3879448 T11259 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 3879448 T11259 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 3879448 T11279 oasc.SolrCore.<init> [solrj_collection_shard1_replica2] Opening new SolrCore at ..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-408F3081D5F614AC-001\tempDir-006\solrj_collection_shard1_replica2\, dataDir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-408F3081D5F614AC-001\tempDir-006\solrj_collection_shard1_replica2\data\
   [junit4]   2> 3879451 T11279 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@50689a6a
   [junit4]   2> 3879452 T11279 oasc.CachingDirectoryFactory.get return new directory for C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-408F3081D5F614AC-001\tempDir-006\solrj_collection_shard1_replica2\data\
   [junit4]   2> 3879452 T11259 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 3879453 T11259 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 3879453 T11279 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-408F3081D5F614AC-001\tempDir-006\solrj_collection_shard1_replica2\data\index/
   [junit4]   2> 3879454 T11279 oasc.SolrCore.initIndex WARN [solrj_collection_shard1_replica2] Solr index directory 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-408F3081D5F614AC-001\tempDir-006\solrj_collection_shard1_replica2\data\index' doesn't exist. Creating new index...
   [junit4]   2> 3879453 T11259 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard1_replica1' using configuration from collection solrj_collection
   [junit4]   2> 3879455 T11259 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 3879456 T11259 oasc.SolrCore.<init> [solrj_collection_shard1_replica1] Opening new SolrCore at C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-408F3081D5F614AC-001\tempDir-005\solrj_collection_shard1_replica1\, dataDir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-408F3081D5F614AC-001\tempDir-005\solrj_collection_shard1_replica1\data\
   [junit4]   2> 3879456 T11259 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@50689a6a
   [junit4]   2> 3879459 T11259 oasc.CachingDirectoryFactory.get return new directory for C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-408F3081D5F614AC-001\tempDir-005\solrj_collection_shard1_replica1\data\
   [junit4]   2> 3879459 T11259 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-408F3081D5F614AC-001\tempDir-005\solrj_collection_shard1_replica1\data\index/
   [junit4]   2> 3879459 T11259 oasc.SolrCore.initIndex WARN [solrj_collection_shard1_replica1] Solr index directory 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-408F3081D5F614AC-001\tempDir-005\solrj_collection_shard1_replica1\data\index' doesn't exist. Creating new index...
   [junit4]   2> 3879460 T11259 oasc.CachingDirectoryFactory.get return new directory for C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-408F3081D5F614AC-001\tempDir-005\solrj_collection_shard1_replica1\data\index
   [junit4]   2> 3879455 T11279 oasc.CachingDirectoryFactory.get return new directory for C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-408F3081D5F614AC-001\tempDir-006\solrj_collection_shard1_replica2\data\index
   [junit4]   2> 3879460 T11259 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=34, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6579631504620821]
   [junit4]   2> 3879462 T11204 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 3879461 T11279 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=34, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6579631504620821]
   [junit4]   2> 3879462 T11259 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@22e8b840 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4bfc4c33),segFN=segments_1,generation=1}
   [junit4]   2> 3879463 T11259 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 3879464 T11279 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@5f670236 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1d0b890b),segFN=segments_1,generation=1}
   [junit4]   2> 3879466 T11279 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 3879470 T11259 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 3879470 T11204 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 3879470 T11259 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 3879471 T11259 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 3879471 T11259 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 3879473 T11259 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 3879473 T11259 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 3879473 T11259 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 3879474 T11259 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 3879474 T11259 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 3879474 T11259 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 3879475 T11259 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 3879475 T11259 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 3879475 T11259 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 3879476 T11259 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 3879476 T11279 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 3879476 T11259 oasc.RequestHandlers.initHandlersFromConfig created /schema: org.apache.solr.handler.SchemaHandler
   [junit4]   2> 3879478 T11259 oasc.RequestHandlers.initHandlersFromConfig created /replication: org.apache.solr.handler.ReplicationHandler
   [junit4]   2> 3879479 T11259 oasc.RequestHandlers.initHandlersFromConfig created /get: org.apache.solr.handler.RealTimeGetHandler
   [junit4]   2> 3879479 T11259 oasc.RequestHandlers.initHandlersFromConfig created /admin/luke: org.apache.solr.handler.admin.LukeRequestHandler
   [junit4]   2> 3879478 T11279 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 3879483 T11279 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 3879483 T11279 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 3879484 T11279 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 3879482 T11259 oasc.RequestHandlers.initHandlersFromConfig created /admin/system: org.apache.solr.handler.admin.SystemInfoHandler
   [junit4]   2> 3879484 T11279 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 3879485 T11259 oasc.RequestHandlers.initHandlersFromConfig created /admin/mbeans: org.apache.solr.handler.admin.SolrInfoMBeanHandler
   [junit4]   2> 3879484 T11204 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 3879485 T11259 oasc.RequestHandlers.initHandlersFromConfig created /admin/plugins: org.apache.solr.handler.admin.PluginInfoHandler
   [junit4]   2> 3879486 T11259 oasc.RequestHandlers.initHandlersFromConfig created /admin/threads: org.apache.solr.handler.admin.ThreadDumpHandler
   [junit4]   2> 3879488 T11246 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 3879485 T11279 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 3879489 T11204 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 3879490 T11204 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 3879490 T11204 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 3879488 T11259 oasc.RequestHandlers.initHandlersFromConfig created /admin/properties: org.apache.solr.handler.admin.PropertiesRequestHandler
   [junit4]   2> 3879491 T11259 oasc.RequestHandlers.initHandlersFromConfig created /admin/logging: org.apache.solr.handler.admin.LoggingHandler
   [junit4]   2> 3879492 T11259 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: org.apache.solr.handler.PingRequestHandler
   [junit4]   2> 3879493 T11259 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 3879493 T11259 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 3879495 T11259 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 3879495 T11246 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 3879489 T11279 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 3879496 T11279 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 3879496 T11279 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 3879497 T11279 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 3879497 T11279 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 3879497 T11279 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 3879498 T11279 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 3879499 T11279 oasc.RequestHandlers.initHandlersFromConfig created /schema: org.apache.solr.handler.SchemaHandler
   [junit4]   2> 3879499 T11279 oasc.RequestHandlers.initHandlersFromConfig created /replication: org.apache.solr.handler.ReplicationHandler
   [junit4]   2> 3879499 T11279 oasc.RequestHandlers.initHandlersFromConfig created /get: org.apache.solr.handler.RealTimeGetHandler
   [junit4]   2> 3879500 T11279 oasc.RequestHandlers.initHandlersFromConfig created /admin/luke: org.apache.solr.handler.admin.LukeRequestHandler
   [junit4]   2> 3879501 T11279 oasc.RequestHandlers.initHandlersFromConfig created /admin/system: org.apache.solr.handler.admin.SystemInfoHandler
   [junit4]   2> 3879503 T11279 oasc.RequestHandlers.initHandlersFromConfig created /admin/mbeans: org.apache.solr.handler.admin.SolrInfoMBeanHandler
   [junit4]   2> 3879504 T11279 oasc.RequestHandlers.initHandlersFromConfig created /admin/plugins: org.apache.solr.handler.admin.PluginInfoHandler
   [junit4]   2> 3879504 T11279 oasc.RequestHandlers.initHandlersFromConfig created /admin/threads: org.apache.solr.handler.admin.ThreadDumpHandler
   [junit4]   2> 3879505 T11279 oasc.RequestHandlers.initHandlersFromConfig created /admin/properties: org.apache.solr.handler.admin.PropertiesRequestHandler
   [junit4]   2> 3879504 T11259 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 3879505 T11279 oasc.RequestHandlers.initHandlersFromConfig created /admin/logging: org.apache.solr.handler.admin.LoggingHandler
   [junit4]   2> 3879507 T11279 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: org.apache.solr.handler.PingRequestHandler
   [junit

[...truncated too long message...]

er watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 3929338 T11177 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/wxx,null}
   [junit4]   2> 3933878 T11177 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
   [junit4]   2> 3933881 T11177 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:58775 58775
   [junit4]   2> 3936005 T11178 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:58775 58775
   [junit4]   2> 3936008 T11178 oasc.ZkTestServer$ZKServerMain.runFromConfig WARN Watch limit violations: 
   [junit4]   2> 	Maximum concurrent create/delete watches above limit:
   [junit4]   2> 	
   [junit4]   2> 		8	/solr/configs/conf1
   [junit4]   2> 		8	/solr/aliases.json
   [junit4]   2> 		8	/solr/clusterstate.json
   [junit4]   2> 		3	/solr/configs/conf2
   [junit4]   2> 		3	/solr/collections/awholynewcollection_0/state.json
   [junit4]   2> 		2	/solr/collections/solrj_newstateformat/state.json
   [junit4]   2> 	
   [junit4]   2> 	Maximum concurrent children watches above limit:
   [junit4]   2> 	
   [junit4]   2> 		8	/solr/live_nodes
   [junit4]   2> 		5	/solr/overseer/queue
   [junit4]   2> 		5	/solr/overseer/collection-queue-work
   [junit4]   2> 	
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=CollectionsAPIDistributedZkTest -Dtests.method=testDistribSearch -Dtests.seed=408F3081D5F614AC -Dtests.slow=true -Dtests.locale=mt_MT -Dtests.timezone=America/Grenada -Dtests.asserts=false -Dtests.file.encoding=US-ASCII
   [junit4] FAILURE 74.3s | CollectionsAPIDistributedZkTest.testDistribSearch <<<
   [junit4]    > Throwable #1: org.junit.ComparisonFailure: expected:<...F3081D5F614AC-001\te[mpDir-003\solrj_test_core_props_shard1_replica1]> but was:<...F3081D5F614AC-001\te[stPropertyParamsForCreate-001\instanceDir-bvc]>
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([408F3081D5F614AC:C169BE99A2A97490]:0)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.checkInstanceDirs(CollectionsAPIDistributedZkTest.java:1154)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testCollectionsAPI(CollectionsAPIDistributedZkTest.java:901)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:205)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 3936032 T11177 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-408F3081D5F614AC-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50): {}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=false,coord=crazy): {}, locale=mt_MT, timezone=America/Grenada
   [junit4]   2> NOTE: Windows 7 6.1 amd64/Oracle Corporation 1.8.0_40-ea (64-bit)/cpus=2,threads=1,free=182305520,total=471019520
   [junit4]   2> NOTE: All tests run in this JVM: [TestSolrDeletionPolicy1, TestSolrDeletionPolicy2, RequiredFieldsTest, TestXIncludeConfig, AnalysisErrorHandlingTest, HdfsRecoveryZkTest, TestSolrXmlPersistor, CoreContainerCoreInitFailuresTest, DistribCursorPagingTest, FileUtilsTest, TestTolerantSearch, FieldMutatingUpdateProcessorTest, TestLuceneMatchVersion, DocumentAnalysisRequestHandlerTest, TestFunctionQuery, TestAnalyzeInfixSuggestions, SolrPluginUtilsTest, PreAnalyzedUpdateProcessorTest, TestPivotHelperCode, TestBlendedInfixSuggestions, DefaultValueUpdateProcessorTest, ZkCLITest, CurrencyFieldXmlFileTest, TestRawResponseWriter, TestBulkSchemaAPI, TestSolr4Spatial, TestArbitraryIndexDir, TestSolrDynamicMBean, TestDFRSimilarityFactory, TestCloudSchemaless, TestCloudManagedSchemaConcurrent, TriLevelCompositeIdRoutingTest, TestDynamicFieldCollectionResource, TestMiniSolrCloudCluster, CoreMergeIndexesAdminHandlerTest, TestMergePolicyConfig, CollectionsAPIAsyncDistributedZkTest, TestSolrQueryParserResource, DebugComponentTest, CopyFieldTest, JsonLoaderTest, ModifyConfFileTest, SpellingQueryConverterTest, TestConfigOverlay, SolrCoreTest, ResponseHeaderTest, TestCollationField, TestComplexPhraseQParserPlugin, DistributedFacetPivotWhiteBoxTest, TestOverriddenPrefixQueryForCustomFieldType, DocExpirationUpdateProcessorFactoryTest, TestAnalyzedSuggestions, SpatialFilterTest, TestRandomFaceting, TestSweetSpotSimilarityFactory, TestInitParams, CursorPagingTest, PathHierarchyTokenizerFactoryTest, TestSchemaSimilarityResource, MergeStrategyTest, TestCloudInspectUtil, TestCopyFieldCollectionResource, JSONWriterTest, TestSerializedLuceneMatchVersion, IndexSchemaRuntimeFieldTest, TestSolrQueryParser, TestNonDefinedSimilarityFactory, DistanceFunctionTest, TestCustomSort, TestStressVersions, TestBlobHandler, TestBadConfig, AliasIntegrationTest, StandardRequestHandlerTest, DocumentBuilderTest, ReturnFieldsTest, LeaderInitiatedRecoveryOnCommitTest, TestCollapseQParserPlugin, TestDistributedSearch, SimpleMLTQParserTest, StressHdfsTest, LeaderFailoverAfterPartitionTest, TestSolrXml, TestLMJelinekMercerSimilarityFactory, CustomCollectionTest, BasicZkTest, TestSolrIndexConfig, TestSystemIdResolver, SharedFSAutoReplicaFailoverTest, DirectSolrConnectionTest, TestClassNameShortening, DistributedSuggestComponentTest, TestQuerySenderListener, DistributedIntervalFacetingTest, TestValueSourceCache, XmlUpdateRequestHandlerTest, EchoParamsTest, TestPerFieldSimilarity, TestAtomicUpdateErrorCases, SimpleCollectionCreateDeleteTest, PrimitiveFieldTypeTest, TestRestManager, HardAutoCommitTest, SuggesterFSTTest, ShardRoutingTest, OpenCloseCoreStressTest, SolrIndexSplitterTest, DistributedTermsComponentTest, EnumFieldTest, PolyFieldTest, QueryResultKeyTest, TestFaceting, UUIDUpdateProcessorFallbackTest, UpdateParamsTest, TestLMDirichletSimilarityFactory, HdfsWriteToMultipleCollectionsTest, OpenExchangeRatesOrgProviderTest, ConvertedLegacyTest, DocValuesTest, TestComponentsName, UniqFieldsUpdateProcessorFactoryTest, BlockCacheTest, TestReplicationHandler, BasicDistributedZk2Test, TestNoOpRegenerator, FullSolrCloudDistribCmdsTest, HdfsUnloadDistributedZkTest, AnalysisAfterCoreReloadTest, UnloadDistributedZkTest, TestOmitPositions, SolrCoreCheckLockOnStartupTest, SimplePostToolTest, CoreAdminCreateDiscoverTest, DeleteReplicaTest, PreAnalyzedFieldTest, TestDocBasedVersionConstraints, BinaryUpdateRequestHandlerTest, TestCloudPivotFacet, ExternalCollectionsTest, TestChildDocTransformer, TestAddFieldRealTimeGet, TestCodecSupport, TestRemoteStreaming, SolrXmlInZkTest, SharedFSAutoReplicaFailoverUtilsTest, TestFieldCollectionResource, PluginInfoTest, ChangedSchemaMergeTest, PeerSyncTest, TestIndexSearcher, RemoteQueryErrorTest, TestFieldSortValues, TestStressUserVersions, TestDocSet, TestStressLucene, HighlighterConfigTest, OutputWriterTest, DOMUtilTest, TestFileDictionaryLookup, DirectUpdateHandlerTest, TestSurroundQueryParser, SolrInfoMBeanTest, TestQuerySenderNoQuery, TestHighlightDedupGrouping, SignatureUpdateProcessorFactoryTest, HdfsLockFactoryTest, HdfsSyncSliceTest, SuggesterWFSTTest, MigrateRouteKeyTest, TestSolrConfigHandler, TestIndexingPerformance, TestIBSimilarityFactory, DateMathParserTest, SoftAutoCommitTest, TestCloudManagedSchema, TestJmxIntegration, TestCSVLoader, TestLeaderElectionZkExpiry, OverseerCollectionProcessorTest, DistributedFacetPivotSmallAdvancedTest, SpellPossibilityIteratorTest, TestManagedResourceStorage, TestFuzzyAnalyzedSuggestions, TestShardHandlerFactory, IndexSchemaTest, SyncSliceTest, TestShortCircuitedRequests, TestBM25SimilarityFactory, HighlighterMaxOffsetTest, MultiTermTest, TestCoreDiscovery, TestSimpleQParserPlugin, SimpleFacetsTest, TestCharFilters, DeleteShardTest, TestPostingsSolrHighlighter, TestSolrCoreProperties, RollingRestartTest, TestPseudoReturnFields, TestGroupingSearch, TestConfigSets, TimeZoneUtilsTest, BJQParserTest, TestSolrJ, QueryParsingTest, TestHighFrequencyDictionaryFactory, TestCSVResponseWriter, MoreLikeThisHandlerTest, QueryElevationComponentTest, CloudMLTQParserTest, ShardRoutingCustomTest, MinimalSchemaTest, CSVRequestHandlerTest, TestReplicationHandlerBackup, TestUtils, TestLFUCache, TestInitQParser, BasicDistributedZkTest, ChaosMonkeySafeLeaderTest, CollectionsAPIDistributedZkTest]
   [junit4] Completed in 74.37s, 1 test, 1 failure <<< FAILURES!

[...truncated 697 lines...]
BUILD FAILED
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:519: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:467: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:61: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\extra-targets.xml:39: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build.xml:187: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\common-build.xml:510: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:1349: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:956: There were test failures: 453 suites, 1850 tests, 1 failure, 56 ignored (31 assumptions)

Total time: 149 minutes 2 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.8.0_40-ea-b09 -XX:+UseCompressedOops -XX:+UseSerialGC (asserts: false)
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Windows (64bit/jdk1.8.0_20) - Build # 4502 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Windows/4502/
Java: 64bit/jdk1.8.0_20 -XX:+UseCompressedOops -XX:+UseParallelGC (asserts: true)

4 tests failed.
FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testDistribSearch

Error Message:
expected:<...7FD839CADB029-001\te[mpDir-003\solrj_test_core_props_shard1_replica1]> but was:<...7FD839CADB029-001\te[stPropertyParamsForCreate-001\instanceDir-a]>

Stack Trace:
org.junit.ComparisonFailure: expected:<...7FD839CADB029-001\te[mpDir-003\solrj_test_core_props_shard1_replica1]> but was:<...7FD839CADB029-001\te[stPropertyParamsForCreate-001\instanceDir-a]>
	at __randomizedtesting.SeedInfo.seed([F397FD839CADB029:7271739BEBF2D015]:0)
	at org.junit.Assert.assertEquals(Assert.java:125)
	at org.junit.Assert.assertEquals(Assert.java:147)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.checkInstanceDirs(CollectionsAPIDistributedZkTest.java:1154)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testCollectionsAPI(CollectionsAPIDistributedZkTest.java:901)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:205)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CollectionsAPIDistributedZkTest

Error Message:
ERROR: SolrZkClient opens=18 closes=17

Stack Trace:
java.lang.AssertionError: ERROR: SolrZkClient opens=18 closes=17
	at __randomizedtesting.SeedInfo.seed([F397FD839CADB029]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.SolrTestCaseJ4.endTrackingZkClients(SolrTestCaseJ4.java:461)
	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:188)
	at sun.reflect.GeneratedMethodAccessor32.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:790)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CollectionsAPIDistributedZkTest

Error Message:
6 threads leaked from SUITE scope at org.apache.solr.cloud.CollectionsAPIDistributedZkTest:     1) Thread[id=3007, name=zkCallback-358-thread-1, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=3006, name=TEST-CollectionsAPIDistributedZkTest.testDistribSearch-seed#[F397FD839CADB029]-EventThread, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)    3) Thread[id=3005, name=TEST-CollectionsAPIDistributedZkTest.testDistribSearch-seed#[F397FD839CADB029]-SendThread(127.0.0.1:63715), state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)         at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)         at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)         at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)    4) Thread[id=3035, name=zkCallback-358-thread-2, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)    5) Thread[id=3036, name=zkCallback-358-thread-3, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)    6) Thread[id=3037, name=zkCallback-358-thread-4, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 6 threads leaked from SUITE scope at org.apache.solr.cloud.CollectionsAPIDistributedZkTest: 
   1) Thread[id=3007, name=zkCallback-358-thread-1, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=3006, name=TEST-CollectionsAPIDistributedZkTest.testDistribSearch-seed#[F397FD839CADB029]-EventThread, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
   3) Thread[id=3005, name=TEST-CollectionsAPIDistributedZkTest.testDistribSearch-seed#[F397FD839CADB029]-SendThread(127.0.0.1:63715), state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
        at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
   4) Thread[id=3035, name=zkCallback-358-thread-2, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
   5) Thread[id=3036, name=zkCallback-358-thread-3, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
   6) Thread[id=3037, name=zkCallback-358-thread-4, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([F397FD839CADB029]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CollectionsAPIDistributedZkTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=3007, name=zkCallback-358-thread-1, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=3005, name=TEST-CollectionsAPIDistributedZkTest.testDistribSearch-seed#[F397FD839CADB029]-SendThread(127.0.0.1:63715), state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)         at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)         at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)         at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)    3) Thread[id=3035, name=zkCallback-358-thread-2, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)    4) Thread[id=3036, name=zkCallback-358-thread-3, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)    5) Thread[id=3037, name=zkCallback-358-thread-4, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=3007, name=zkCallback-358-thread-1, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=3005, name=TEST-CollectionsAPIDistributedZkTest.testDistribSearch-seed#[F397FD839CADB029]-SendThread(127.0.0.1:63715), state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
        at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
   3) Thread[id=3035, name=zkCallback-358-thread-2, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
   4) Thread[id=3036, name=zkCallback-358-thread-3, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
   5) Thread[id=3037, name=zkCallback-358-thread-4, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([F397FD839CADB029]:0)




Build Log:
[...truncated 8804 lines...]
   [junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
   [junit4]   2> Creating dataDir: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-F397FD839CADB029-001\init-core-data-001
   [junit4]   2> 1026417 T2819 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (true)
   [junit4]   2> 1026417 T2819 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /go_/uw
   [junit4]   2> 1026426 T2819 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 1026426 T2819 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1026427 T2820 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1026526 T2819 oasc.ZkTestServer.run start zk server on port:63715
   [junit4]   2> 1026526 T2819 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1026527 T2819 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1026540 T2827 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1258093f name:ZooKeeperConnection Watcher:127.0.0.1:63715 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1026542 T2819 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1026542 T2819 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1026542 T2819 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1026551 T2819 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1026554 T2819 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1026557 T2830 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@eef47b8 name:ZooKeeperConnection Watcher:127.0.0.1:63715/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1026557 T2819 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1026557 T2819 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1026559 T2819 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 1026564 T2819 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 1026568 T2819 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 1026571 T2819 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 1026574 T2819 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1026575 T2819 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1026580 T2819 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 1026580 T2819 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 1026586 T2819 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1026586 T2819 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1026590 T2819 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1026591 T2819 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1026594 T2819 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1026595 T2819 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1026598 T2819 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1026599 T2819 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 1026603 T2819 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1026604 T2819 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1026608 T2819 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1026609 T2819 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1026612 T2819 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1026613 T2819 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1026617 T2819 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1026617 T2819 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1026620 T2819 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1026621 T2819 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1026627 T2821 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x14a66efd1cd0001, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 1026628 T2819 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1026630 T2819 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1026635 T2833 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6035f93c name:ZooKeeperConnection Watcher:127.0.0.1:63715/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1026635 T2819 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1026635 T2819 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> Using new style solr.xml
   [junit4]   2> 1028393 T2819 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1028404 T2819 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1028414 T2819 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:63725
   [junit4]   2> 1028419 T2819 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1028420 T2819 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1028420 T2819 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-F397FD839CADB029-001\tempDir-002
   [junit4]   2> 1028420 T2819 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-F397FD839CADB029-001\tempDir-002\'
   [junit4]   2> 1028481 T2819 oasc.ConfigSolr.fromFile Loading container configuration from C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-F397FD839CADB029-001\tempDir-002\solr.xml
   [junit4]   2> 1028491 T2819 oasc.CorePropertiesLocator.<init> Config-defined core root directory: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-F397FD839CADB029-001\tempDir-002
   [junit4]   2> 1028491 T2819 oasc.CoreContainer.<init> New CoreContainer 174502297
   [junit4]   2> 1028491 T2819 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-F397FD839CADB029-001\tempDir-002\]
   [junit4]   2> 1028492 T2819 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1028492 T2819 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1028493 T2819 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1028493 T2819 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1028493 T2819 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 1028493 T2819 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1028493 T2819 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1028493 T2819 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1028493 T2819 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1028493 T2819 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1028498 T2819 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 1028500 T2819 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1028500 T2819 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1028500 T2819 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1028500 T2819 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:63715/solr
   [junit4]   2> 1028500 T2819 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1028500 T2819 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1028505 T2819 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1028508 T2845 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@68f754f6 name:ZooKeeperConnection Watcher:127.0.0.1:63715 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1028509 T2819 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1028510 T2819 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1028516 T2819 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1028518 T2848 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1250093a name:ZooKeeperConnection Watcher:127.0.0.1:63715/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1028519 T2819 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1028526 T2819 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1028532 T2819 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1028538 T2819 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 1028544 T2819 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 1028550 T2819 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 1028557 T2819 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1028559 T2819 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:63725_go_%2Fuw
   [junit4]   2> 1028561 T2819 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:63725_go_%2Fuw
   [junit4]   2> 1028566 T2819 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 1028569 T2819 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1028573 T2819 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1028576 T2819 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:63725_go_%2Fuw
   [junit4]   2> 1028576 T2819 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1028580 T2819 oasc.Overseer.start Overseer (id=92999922517606404-127.0.0.1:63725_go_%2Fuw-n_0000000000) starting
   [junit4]   2> 1028583 T2819 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1028599 T2819 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=10000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 1028600 T2850 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1028602 T2819 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1028606 T2819 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1028609 T2819 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1028616 T2849 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1028617 T2819 oasc.CorePropertiesLocator.discover Looking for core definitions underneath C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-F397FD839CADB029-001\tempDir-002
   [junit4]   2> 1029863 T2819 oasc.CorePropertiesLocator.discoverUnder Found core conf in C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-F397FD839CADB029-001\tempDir-002\conf\
   [junit4]   2> 1030071 T2819 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1030073 T2819 oass.SolrDispatchFilter.init user.dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0
   [junit4]   2> 1030073 T2819 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1030081 T2819 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1030082 T2819 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1030086 T2855 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@217e5b74 name:ZooKeeperConnection Watcher:127.0.0.1:63715/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1030087 T2819 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1030087 T2819 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1030090 T2819 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1030099 T2819 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1030099 T2819 oasc.AbstractFullDistribZkTestBase.createJettys Creating collection1 with stateFormat=2
   [junit4]   2> 1030099 T2819 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1030101 T2819 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1030104 T2858 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a4f1840 name:ZooKeeperConnection Watcher:127.0.0.1:63715/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1030105 T2819 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1030105 T2819 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1030109 T2848 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1030117 T2849 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 0, message = {
   [junit4]   2> 	  "operation":"create",
   [junit4]   2> 	  "name":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "stateFormat":2} current state version: 0
   [junit4]   2> 1030117 T2849 oasco.ClusterStateMutator.createCollection building a new cName: collection1
   [junit4]   2> 1030119 T2849 oasco.ZkStateWriter.writePendingUpdates going to create_collection /collections/collection1/state.json
   [junit4]   2> 1030121 T2855 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1030122 T2848 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1031769 T2819 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 1031770 T2819 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1031780 T2819 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1031783 T2819 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:63740
   [junit4]   2> 1031789 T2819 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1031790 T2819 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1031790 T2819 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-F397FD839CADB029-001\tempDir-003
   [junit4]   2> 1031791 T2819 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-F397FD839CADB029-001\tempDir-003\'
   [junit4]   2> 1031841 T2819 oasc.ConfigSolr.fromFile Loading container configuration from C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-F397FD839CADB029-001\tempDir-003\solr.xml
   [junit4]   2> 1031850 T2819 oasc.CorePropertiesLocator.<init> Config-defined core root directory: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-F397FD839CADB029-001\tempDir-003
   [junit4]   2> 1031851 T2819 oasc.CoreContainer.<init> New CoreContainer 52998163
   [junit4]   2> 1031851 T2819 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-F397FD839CADB029-001\tempDir-003\]
   [junit4]   2> 1031852 T2819 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1031852 T2819 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1031853 T2819 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1031853 T2819 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1031853 T2819 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 1031853 T2819 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1031853 T2819 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1031853 T2819 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1031853 T2819 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1031853 T2819 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1031858 T2819 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 1031860 T2819 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1031860 T2819 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1031860 T2819 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1031860 T2819 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:63715/solr
   [junit4]   2> 1031861 T2819 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1031862 T2819 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1031864 T2819 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1031868 T2870 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@58d929f9 name:ZooKeeperConnection Watcher:127.0.0.1:63715 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1031869 T2819 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1031870 T2819 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1031875 T2819 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1031877 T2873 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@652ea778 name:ZooKeeperConnection Watcher:127.0.0.1:63715/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1031878 T2819 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1031891 T2819 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1032888 T2819 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:63740_go_%2Fuw
   [junit4]   2> 1032890 T2819 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:63740_go_%2Fuw
   [junit4]   2> 1032908 T2819 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1032915 T2819 oasc.CorePropertiesLocator.discover Looking for core definitions underneath C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-F397FD839CADB029-001\tempDir-003
   [junit4]   2> 1033578 T2819 oasc.CorePropertiesLocator.discoverUnder Found core conf in C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-F397FD839CADB029-001\tempDir-003\conf\
   [junit4]   2> 1033700 T2819 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1033705 T2819 oass.SolrDispatchFilter.init user.dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0
   [junit4]   2> 1033705 T2819 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1035671 T2819 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 1035671 T2819 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1035684 T2819 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1035689 T2819 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:63749
   [junit4]   2> 1035694 T2819 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1035694 T2819 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1035695 T2819 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-F397FD839CADB029-001\tempDir-004
   [junit4]   2> 1035695 T2819 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-F397FD839CADB029-001\tempDir-004\'
   [junit4]   2> 1035750 T2819 oasc.ConfigSolr.fromFile Loading container configuration from C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-F397FD839CADB029-001\tempDir-004\solr.xml
   [junit4]   2> 1035769 T2819 oasc.CorePropertiesLocator.<init> Config-defined core root directory: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-F397FD839CADB029-001\tempDir-004
   [junit4]   2> 1035770 T2819 oasc.CoreContainer.<init> New CoreContainer 1437769454
   [junit4]   2> 1035770 T2819 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-F397FD839CADB029-001\tempDir-004\]
   [junit4]   2> 1035771 T2819 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1035771 T2819 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1035772 T2819 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1035772 T2819 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1035772 T2819 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 1035774 T2819 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1035774 T2819 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1035775 T2819 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1035775 T2819 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1035776 T2819 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1035780 T2819 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 1035786 T2819 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1035787 T2819 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1035787 T2819 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1035788 T2819 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:63715/solr
   [junit4]   2> 1035788 T2819 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1035788 T2819 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1035790 T2819 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1035795 T2886 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1462d6e6 name:ZooKeeperConnection Watcher:127.0.0.1:63715 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1035796 T2819 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1035797 T2819 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1035800 T2821 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x14a66efd1cd0009, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 1035802 T2819 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1035805 T2889 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@37b8fa89 name:ZooKeeperConnection Watcher:127.0.0.1:63715/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1035805 T2819 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1035817 T2819 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1036903 T2819 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:63749_go_%2Fuw
   [junit4]   2> 1036906 T2819 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:63749_go_%2Fuw
   [junit4]   2> 1036920 T2819 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1036930 T2819 oasc.CorePropertiesLocator.discover Looking for core definitions underneath C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-F397FD839CADB029-001\tempDir-004
   [junit4]   2> 1038128 T2819 oasc.CorePropertiesLocator.discoverUnder Found core conf in C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-F397FD839CADB029-001\tempDir-004\conf\
   [junit4]   2> 1038340 T2819 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1038342 T2819 oass.SolrDispatchFilter.init user.dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0
   [junit4]   2> 1038342 T2819 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1040187 T2819 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 1040188 T2819 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1040194 T2819 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1040197 T2819 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:63758
   [junit4]   2> 1040203 T2819 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1040203 T2819 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1040203 T2819 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-F397FD839CADB029-001\tempDir-005
   [junit4]   2> 1040203 T2819 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-F397FD839CADB029-001\tempDir-005\'
   [junit4]   2> 1040238 T2819 oasc.ConfigSolr.fromFile Loading container configuration from C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-F397FD839CADB029-001\tempDir-005\solr.xml
   [junit4]   2> 1040247 T2819 oasc.CorePropertiesLocator.<init> Config-defined core root directory: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-F397FD839CADB029-001\tempDir-005
   [junit4]   2> 1040248 T2819 oasc.CoreContainer.<init> New CoreContainer 1499352349
   [junit4]   2> 1040250 T2819 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-F397FD839CADB029-001\tempDir-005\]
   [junit4]   2> 1040251 T2819 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1040251 T2819 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1040251 T2819 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1040251 T2819 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1040251 T2819 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 1040251 T2819 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1040251 T2819 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1040251 T2819 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1040253 T2819 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1040253 T2819 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1040258 T2819 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 1040264 T2819 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1040264 T2819 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1040264 T2819 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1040264 T2819 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:63715/solr
   [junit4]   2> 1040264 T2819 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1040264 T2819 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1040267 T2819 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1040270 T2902 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2816f29e name:ZooKeeperConnection Watcher:127.0.0.1:63715 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1040272 T2819 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1040272 T2819 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1040277 T2819 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1040281 T2905 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5ff638fd name:ZooKeeperConnection Watcher:127.0.0.1:63715/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1040281 T2819 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1040287 T2819 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1041375 T2819 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:63758_go_%2Fuw
   [junit4]   2> 1041377 T2819 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:63758_go_%2Fuw
   [junit4]   2> 1041385 T2819 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1041389 T2819 oasc.CorePropertiesLocator.discover Looking for core definitions underneath C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-F397FD839CADB029-001\tempDir-005
   [junit4]   2> 1042624 T2819 oasc.CorePropertiesLocator.discoverUnder Found core conf in C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-F397FD839CADB029-001\tempDir-005\conf\
   [junit4]   2> 1042839 T2819 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1042841 T2819 oass.SolrDispatchFilter.init user.dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0
   [junit4]   2> 1042841 T2819 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1044705 T2819 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 4
   [junit4]   2> 1044707 T2819 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1044720 T2819 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1044724 T2819 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:63767
   [junit4]   2> 1044728 T2819 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1044728 T2819 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1044728 T2819 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-F397FD839CADB029-001\tempDir-006
   [junit4]   2> 1044728 T2819 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-F397FD839CADB029-001\tempDir-006\'
   [junit4]   2> 1044768 T2819 oasc.ConfigSolr.fromFile Loading container configuration from C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-F397FD839CADB029-001\tempDir-006\solr.xml
   [junit4]   2> 1044779 T2819 oasc.CorePropertiesLocator.<init> Config-defined core root directory: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-F397FD839CADB029-001\tempDir-006
   [junit4]   2> 1044779 T2819 oasc.CoreContainer.<init> New CoreContainer 736664320
   [junit4]   2> 1044779 T2819 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-F397FD839CADB029-001\tempDir-006\]
   [junit4]   2> 1044780 T2819 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1044781 T2819 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1044781 T2819 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1044781 T2819 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1044781 T2819 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 1044781 T2819 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1044781 T2819 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1044781 T2819 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1044781 T2819 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1044781 T2819 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1044788 T2819 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 1044798 T2819 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1044798 T2819 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1044798 T2819 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1044798 T2819 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:63715/solr
   [junit4]   2> 1044798 T2819 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1044798 T2819 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1044801 T2819 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1044808 T2918 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@58bb952e name:ZooKeeperConnection Watcher:127.0.0.1:63715 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1044810 T2819 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1044810 T2819 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1044820 T2821 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x14a66efd1cd000d, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 1044825 T2819 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1044829 T2921 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5aecadcb name:ZooKeeperConnection Watcher:127.0.0.1:63715/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1044829 T2819 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1044847 T2819 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1046044 T2819 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:63767_go_%2Fuw
   [junit4]   2> 1046047 T2819 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:63767_go_%2Fuw
   [junit4]   2> 1046062 T2819 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1046069 T2819 oasc.CorePropertiesLocator.discover Looking for core definitions underneath C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-F397FD839CADB029-001\tempDir-006
   [junit4]   2> 1046829 T2819 oasc.CorePropertiesLocator.discoverUnder Found core conf in C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-F397FD839CADB029-001\tempDir-006\conf\
   [junit4]   2> 1046974 T2819 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1046976 T2819 oass.SolrDispatchFilter.init user.dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0
   [junit4]   2> 1046977 T2819 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1047052 T2867 oasha.CollectionsHandler.handleCreateAction Creating Collection : replicationFactor=2&collection.configName=conf1&router.field=myOwnField&name=solrj_collection&action=CREATE&numShards=2&wt=javabin&version=2
   [junit4]   2> 1047054 T2848 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/collection-queue-work state SyncConnected
   [junit4]   2> 1047056 T2850 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"create",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"solrj_collection",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "collection.configName":"conf1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "router.field":"myOwnField"}
   [junit4]   2> 1047059 T2923 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : create , {
   [junit4]   2> 	  "operation":"create",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"solrj_collection",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "collection.configName":"conf1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "router.field":"myOwnField"}
   [junit4]   2> 1047064 T2923 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/solrj_collection 
   [junit4]   2> 1047066 T2923 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection
   [junit4]   2> 1047074 T2848 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1047078 T2849 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"create",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"solrj_collection",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "collection.configName":"conf1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "router.field":"myOwnField"} current state version: 1
   [junit4]   2> 1047078 T2849 oasco.ClusterStateMutator.createCollection building a new cName: solrj_collection
   [junit4]   2> 1047083 T2873 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1047084 T2848 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1047084 T2921 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1047085 T2905 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1047085 T2855 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1047085 T2889 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1047313 T2923 oasc.OverseerCollectionProcessor.createCollection Creating SolrCores for new collection solrj_collection, shardNames [shard1, shard2] , replicationFactor : 2
   [junit4]   2> 1047313 T2923 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard1_replica1 as part of slice shard1 of collection solrj_collection on 127.0.0.1:63740_go_%2Fuw
   [junit4]   2> 1047315 T2923 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard1_replica2 as part of slice shard1 of collection solrj_collection on 127.0.0.1:63725_go_%2Fuw
   [junit4]   2> 1047334 T2923 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard2_replica1 as part of slice shard2 of collection solrj_collection on 127.0.0.1:63758_go_%2Fuw
   [junit4]   2> 1047341 T2923 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard2_replica2 as part of slice shard2 of collection solrj_collection on 127.0.0.1:63767_go_%2Fuw
   [junit4]   2> 1047379 T2865 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&collection.configName=conf1&name=solrj_collection_shard1_replica1&action=CREATE&numShards=2&collection=solrj_collection&shard=shard1&wt=javabin&version=2
   [junit4]   2> 1047429 T2865 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica1 state=down collection=solrj_collection
   [junit4]   2> 1047435 T2898 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&collection.configName=conf1&name=solrj_collection_shard2_replica1&action=CREATE&numShards=2&collection=solrj_collection&shard=shard2&wt=javabin&version=2
   [junit4]   2> 1047437 T2910 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&collection.configName=conf1&name=solrj_collection_shard2_replica2&action=CREATE&numShards=2&collection=solrj_collection&shard=shard2&wt=javabin&version=2
   [junit4]   2> 1047443 T2865 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1047444 T2848 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1047449 T2839 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&collection.configName=conf1&name=solrj_collection_shard1_replica2&action=CREATE&numShards=2&collection=solrj_collection&shard=shard1&wt=javabin&version=2
   [junit4]   2> 1047452 T2898 oasc.ZkController.publish publishing core=solrj_collection_shard2_replica1 state=down collection=solrj_collection
   [junit4]   2> 1047453 T2849 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"solrj_collection_shard1_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:63740/go_/uw",
   [junit4]   2> 	  "node_name":"127.0.0.1:63740_go_%2Fuw",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "operation":"state"} current state version: 2
   [junit4]   2> 1047454 T2898 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1047454 T2910 oasc.ZkController.publish publishing core=solrj_collection_shard2_replica2 state=down collection=solrj_collection
   [junit4]   2> 1047457 T2910 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1047458 T2849 oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"solrj_collection_shard1_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:63740/go_/uw",
   [junit4]   2> 	  "node_name":"127.0.0.1:63740_go_%2Fuw",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 1047458 T2839 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica2 state=down collection=solrj_collection
   [junit4]   2> 1047460 T2839 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1047464 T2849 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 3, message = {
   [junit4]   2> 	  "core":"solrj_collection_shard2_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:63758/go_/uw",
   [junit4]   2> 	  "node_name":"127.0.0.1:63758_go_%2Fuw",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "operation":"state"} current state version: 2
   [junit4]   2> 1047466 T2849 oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"solrj_collection_shard2_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:63758/go_/uw",
   [junit4]   2> 	  "node_name":"127.0.0.1:63758_go_%2Fuw",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 1047472 T2848 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1047475 T2849 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 2, message = {
   [junit4]   2> 	  "core":"solrj_collection_shard2_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:63767/go_/uw",
   [junit4]   2> 	  "node_name":"127.0.0.1:63767_go_%2Fuw",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "operation":"state"} current state version: 2
   [junit4]   2> 1047477 T2849 oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"solrj_collection_shard2_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:63767/go_/uw",
   [junit4]   2> 	  "node_name":"127.0.0.1:63767_go_%2Fuw",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 1047481 T2848 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1047483 T2849 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 2, message = {
   [junit4]   2> 	  "core":"solrj_collection_shard1_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:63725/go_/uw",
   [junit4]   2> 	  "node_name":"127.0.0.1:63725_go_%2Fuw",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "operation":"state"} current state version: 2
   [junit4]   2> 1047486 T2849 oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"solrj_collection_shard1_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:63725/go_/uw",
   [junit4]   2> 	  "node_name":"127.0.0.1:63725_go_%2Fuw",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 1047609 T2873 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1047609 T2848 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1047611 T2921 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1047611 T2889 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1047612 T2855 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1047611 T2905 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1048629 T2865 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard1_replica1
   [junit4]   2> 1048629 T2865 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 1048630 T2865 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1048631 T2865 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 1048634 T2865 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1048634 T2865 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-F397FD839CADB029-001\tempDir-003\solrj_collection_shard1_replica1\'
   [junit4]   2> 1048640 T2898 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard2_replica1
   [junit4]   2> 1048640 T2898 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 1048643 T2898 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1048644 T2898 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 1048644 T2910 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard2_replica2
   [junit4]   2> 1048644 T2910 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 1048647 T2839 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard1_replica2
   [junit4]   2> 1048647 T2839 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 1048648 T2910 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1048648 T2910 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 1048648 T2898 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1048649 T2898 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-F397FD839CADB029-001\tempDir-005\solrj_collection_shard2_replica1\'
   [junit4]   2> 1048674 T2839 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1048674 T2839 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 1048675 T2910 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1048675 T2910 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-F397FD839CADB029-001\tempDir-006\solrj_collection_shard2_replica2\'
   [junit4]   2> 1048690 T2865 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 1048692 T2839 

[...truncated too long message...]

roup=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >    5) Thread[id=3036, name=zkCallback-358-thread-3, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >    6) Thread[id=3037, name=zkCallback-358-thread-4, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([F397FD839CADB029]:0)Throwable #3: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=3007, name=zkCallback-358-thread-1, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >    2) Thread[id=3005, name=TEST-CollectionsAPIDistributedZkTest.testDistribSearch-seed#[F397FD839CADB029]-SendThread(127.0.0.1:63715), state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    >         at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
   [junit4]    >         at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
   [junit4]    >         at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
   [junit4]    >         at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
   [junit4]    >         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   [junit4]    >         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
   [junit4]    >         at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
   [junit4]    >    3) Thread[id=3035, name=zkCallback-358-thread-2, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >    4) Thread[id=3036, name=zkCallback-358-thread-3, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >    5) Thread[id=3037, name=zkCallback-358-thread-4, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([F397FD839CADB029]:0)
   [junit4] Completed in 113.06s, 1 test, 2 failures, 2 errors <<< FAILURES!

[...truncated 1150 lines...]
BUILD FAILED
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:519: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:467: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:61: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\extra-targets.xml:39: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build.xml:187: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\common-build.xml:510: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:1349: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:956: There were test failures: 452 suites, 1849 tests, 3 suite-level errors, 1 failure, 1443 ignored (17 assumptions)

Total time: 47 minutes 8 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.8.0_20 -XX:+UseCompressedOops -XX:+UseParallelGC (asserts: true)
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



Re: [JENKINS] Lucene-Solr-trunk-Windows (32bit/jdk1.8.0_20) - Build # 4501 - Still Failing!

Posted by Adrien Grand <jp...@gmail.com>.
This failure does not reproduce for me, does it for anyone?

On Sat, Dec 20, 2014 at 1:32 AM, Policeman Jenkins Server
<je...@thetaphi.de> wrote:
> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Windows/4501/
> Java: 32bit/jdk1.8.0_20 -server -XX:+UseSerialGC (asserts: true)
>
> 1 tests failed.
> FAILED:  org.apache.lucene.facet.TestRandomSamplingFacetsCollector.testRandomSampling
>
> Error Message:
> 54
>
> Stack Trace:
> java.lang.ArrayIndexOutOfBoundsException: 54
>         at __randomizedtesting.SeedInfo.seed([36B2EF8F9284ED2B:DCAA869223D31A9C]:0)
>         at org.apache.lucene.facet.TestRandomSamplingFacetsCollector.testRandomSampling(TestRandomSamplingFacetsCollector.java:133)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:483)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
>         at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>         at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>         at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>         at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>         at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>         at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>         at java.lang.Thread.run(Thread.java:745)
>
>
>
>
> Build Log:
> [...truncated 5831 lines...]
>    [junit4] Suite: org.apache.lucene.facet.TestRandomSamplingFacetsCollector
>    [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestRandomSamplingFacetsCollector -Dtests.method=testRandomSampling -Dtests.seed=36B2EF8F9284ED2B -Dtests.slow=true -Dtests.locale=es_GT -Dtests.timezone=America/Araguaina -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
>    [junit4] ERROR   1.08s | TestRandomSamplingFacetsCollector.testRandomSampling <<<
>    [junit4]    > Throwable #1: java.lang.ArrayIndexOutOfBoundsException: 54
>    [junit4]    >        at __randomizedtesting.SeedInfo.seed([36B2EF8F9284ED2B:DCAA869223D31A9C]:0)
>    [junit4]    >        at org.apache.lucene.facet.TestRandomSamplingFacetsCollector.testRandomSampling(TestRandomSamplingFacetsCollector.java:133)
>    [junit4]    >        at java.lang.Thread.run(Thread.java:745)
>    [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50): {$full_path$=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), $facets=PostingsFormat(name=LuceneFixedGap), $payloads$=Lucene50(blocksize=128), EvenOdd=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128)))}, docValues:{$facets=DocValuesFormat(name=Asserting)}, sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {EvenOdd=IB SPL-L3(800.0)}, locale=es_GT, timezone=America/Araguaina
>    [junit4]   2> NOTE: Windows 7 6.1 x86/Oracle Corporation 1.8.0_20 (32-bit)/cpus=2,threads=1,free=16838256,total=58535936
>    [junit4]   2> NOTE: All tests run in this JVM: [TestDrillDownQuery, TestTaxonomyFacetCounts, TestTaxonomyFacetSumValueSource, TestSearcherTaxonomyManager, TestRandomSamplingFacetsCollector]
>    [junit4] Completed in 1.14s, 1 test, 1 error <<< FAILURES!
>
> [...truncated 63 lines...]
> BUILD FAILED
> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:519: The following error occurred while executing this line:
> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:467: The following error occurred while executing this line:
> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:61: The following error occurred while executing this line:
> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\extra-targets.xml:39: The following error occurred while executing this line:
> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build.xml:456: The following error occurred while executing this line:
> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:2070: The following error occurred while executing this line:
> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\module-build.xml:58: The following error occurred while executing this line:
> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:1349: The following error occurred while executing this line:
> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:956: There were test failures: 23 suites, 154 tests, 1 error
>
> Total time: 23 minutes 43 seconds
> Build step 'Invoke Ant' marked build as failure
> [description-setter] Description set: Java: 32bit/jdk1.8.0_20 -server -XX:+UseSerialGC (asserts: true)
> Archiving artifacts
> Recording test results
> Email was triggered for: Failure - Any
> Sending email for trigger: Failure - Any
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org



-- 
Adrien

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org