You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2014/11/21 20:01:46 UTC

[JENKINS] Lucene-Solr-Tests-trunk-Java8 - Build # 5 - Still Failing

Build: https://builds.apache.org/job/Lucene-Solr-Tests-trunk-Java8/5/

2 tests failed.
REGRESSION:  org.apache.solr.TestDistributedGrouping.testDistribSearch

Error Message:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[https://[ff01::114]:33332, https://[ff01::083]:33332, https://127.0.0.1:37975, https://[ff01::213]:33332]

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[https://[ff01::114]:33332, https://[ff01::083]:33332, https://127.0.0.1:37975, https://[ff01::213]:33332]
	at __randomizedtesting.SeedInfo.seed([103A566D0461F871:91DCD875733E984D]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:569)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
	at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:91)
	at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
	at org.apache.solr.BaseDistributedSearchTestCase.queryServer(BaseDistributedSearchTestCase.java:512)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:560)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:542)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:521)
	at org.apache.solr.TestDistributedGrouping.doTest(TestDistributedGrouping.java:206)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:875)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.handler.TestSolrConfigHandlerConcurrent.testDistribSearch

Error Message:
[[size property not set, expected = 11, actual null, initialSize property not set, expected = 12, actual null, autowarmCount property not set, expected = 13, actual null], [], [], []]

Stack Trace:
java.lang.AssertionError: [[size property not set, expected = 11, actual null, initialSize property not set, expected = 12, actual null, autowarmCount property not set, expected = 13, actual null], [], [], []]
	at __randomizedtesting.SeedInfo.seed([103A566D0461F871:91DCD875733E984D]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.handler.TestSolrConfigHandlerConcurrent.doTest(TestSolrConfigHandlerConcurrent.java:112)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 13391 lines...]
   [junit4] Suite: org.apache.solr.handler.TestSolrConfigHandlerConcurrent
   [junit4]   2> Creating dataDir: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-103A566D0461F871-001/init-core-data-001
   [junit4]   2> 409504 T2325 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 409505 T2325 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /orln/
   [junit4]   2> 409508 T2325 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 409509 T2325 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 409510 T2326 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 409610 T2325 oasc.ZkTestServer.run start zk server on port:47646
   [junit4]   2> 409611 T2325 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 409612 T2325 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 409616 T2333 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3d627ba3 name:ZooKeeperConnection Watcher:127.0.0.1:47646 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 409617 T2325 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 409617 T2325 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 409617 T2325 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 409620 T2325 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 409621 T2325 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 409622 T2336 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2458f3dd name:ZooKeeperConnection Watcher:127.0.0.1:47646/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 409622 T2325 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 409623 T2325 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 409623 T2325 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 409625 T2325 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 409626 T2325 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 409627 T2325 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 409629 T2325 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 409630 T2325 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 409632 T2325 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 409632 T2325 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 409734 T2325 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 409735 T2325 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 409736 T2325 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 409737 T2325 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 409738 T2325 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 409739 T2325 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 409740 T2325 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 409741 T2325 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 409742 T2325 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 409743 T2325 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 409745 T2325 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 409745 T2325 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 409747 T2325 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 409747 T2325 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 409749 T2325 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 409750 T2325 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 409751 T2325 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 409751 T2325 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 410973 T2325 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 410994 T2325 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:37737
   [junit4]   2> 410994 T2325 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 410994 T2325 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 410995 T2325 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-103A566D0461F871-001/tempDir-002
   [junit4]   2> 410995 T2325 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-103A566D0461F871-001/tempDir-002/'
   [junit4]   2> 411025 T2325 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-103A566D0461F871-001/tempDir-002/solr.xml
   [junit4]   2> 411038 T2325 oasc.CoreContainer.<init> New CoreContainer 1421426377
   [junit4]   2> 411039 T2325 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-103A566D0461F871-001/tempDir-002/]
   [junit4]   2> 411039 T2325 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 411040 T2325 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 411040 T2325 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 411040 T2325 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 411040 T2325 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 411040 T2325 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 411041 T2325 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 411041 T2325 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 411041 T2325 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 411041 T2325 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 411042 T2325 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 411042 T2325 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 411042 T2325 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 411043 T2325 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 411043 T2325 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:47646/solr
   [junit4]   2> 411043 T2325 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 411044 T2325 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 411044 T2325 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 411046 T2348 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5640bd9 name:ZooKeeperConnection Watcher:127.0.0.1:47646 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 411047 T2325 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 411047 T2325 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 411049 T2325 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 411050 T2351 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@286c5bed name:ZooKeeperConnection Watcher:127.0.0.1:47646/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 411051 T2325 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 411052 T2325 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 411053 T2325 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 411055 T2325 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 411056 T2325 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 411058 T2325 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 411060 T2325 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 411060 T2325 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37737_orln
   [junit4]   2> 411061 T2325 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:37737_orln
   [junit4]   2> 411062 T2325 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 411063 T2325 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 411064 T2325 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 411065 T2325 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:37737_orln
   [junit4]   2> 411066 T2325 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 411067 T2325 oasc.Overseer.start Overseer (id=92837767038042115-127.0.0.1:37737_orln-n_0000000000) starting
   [junit4]   2> 411068 T2325 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 411072 T2325 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 411073 T2353 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 411073 T2325 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 411075 T2325 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 411075 T2325 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 411077 T2352 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 411079 T2355 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 411080 T2355 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 411080 T2351 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 411080 T2355 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 411082 T2352 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:37737/orln",
   [junit4]   2> 	  "node_name":"127.0.0.1:37737_orln",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 411082 T2352 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 411082 T2352 oasc.Overseer$ClusterStateUpdater.createCollection state version control_collection 1
   [junit4]   2> 411082 T2352 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 411084 T2351 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 412081 T2355 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 412082 T2355 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 412082 T2355 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 412083 T2355 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 412083 T2355 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 412083 T2355 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-103A566D0461F871-001/tempDir-002/collection1/'
   [junit4]   2> 412085 T2355 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-103A566D0461F871-001/tempDir-002/collection1/lib/.svn/' to classloader
   [junit4]   2> 412086 T2355 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-103A566D0461F871-001/tempDir-002/collection1/lib/README' to classloader
   [junit4]   2> 412087 T2355 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-103A566D0461F871-001/tempDir-002/collection1/lib/classes/' to classloader
   [junit4]   2> 412119 T2355 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 412120 T2355 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 412134 T2355 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 412145 T2355 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 412246 T2355 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 412252 T2355 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 412366 T2355 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 412376 T2355 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 412378 T2355 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 412385 T2355 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 412388 T2355 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 412390 T2355 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 412391 T2355 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 412392 T2355 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 412392 T2355 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 412392 T2355 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 412393 T2355 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 412393 T2355 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 412393 T2355 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 412393 T2355 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-103A566D0461F871-001/tempDir-002/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-103A566D0461F871-001/tempDir-001/control/data/
   [junit4]   2> 412394 T2355 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@330fda4c
   [junit4]   2> 412394 T2355 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-103A566D0461F871-001/tempDir-001/control/data
   [junit4]   2> 412395 T2355 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-103A566D0461F871-001/tempDir-001/control/data/index/
   [junit4]   2> 412395 T2355 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-103A566D0461F871-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 412395 T2355 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-103A566D0461F871-001/tempDir-001/control/data/index
   [junit4]   2> 412396 T2355 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=30, maxMergeAtOnceExplicit=41, maxMergedSegmentMB=64.537109375, floorSegmentMB=1.693359375, forceMergeDeletesPctAllowed=3.111209946074762, segmentsPerTier=41.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 412397 T2355 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@17bfecc0 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@3d315b92),segFN=segments_1,generation=1}
   [junit4]   2> 412397 T2355 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 412400 T2355 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 412400 T2355 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 412401 T2355 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 412401 T2355 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 412401 T2355 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 412401 T2355 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 412402 T2355 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 412402 T2355 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 412402 T2355 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 412403 T2355 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 412404 T2355 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 412404 T2355 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 412404 T2355 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 412405 T2355 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 412405 T2355 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 412405 T2355 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 412406 T2355 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 412406 T2355 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 412406 T2355 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 412407 T2355 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 412407 T2355 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 412417 T2355 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 412418 T2355 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 412420 T2355 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 412421 T2355 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 412425 T2355 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 412425 T2355 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 412425 T2355 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=16, maxMergeAtOnceExplicit=15, maxMergedSegmentMB=60.369140625, floorSegmentMB=1.87109375, forceMergeDeletesPctAllowed=16.64506491494333, segmentsPerTier=38.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3115152765900482
   [junit4]   2> 412426 T2355 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@17bfecc0 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@3d315b92),segFN=segments_1,generation=1}
   [junit4]   2> 412426 T2355 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 412427 T2355 oass.SolrIndexSearcher.<init> Opening Searcher@48d7329e[collection1] main
   [junit4]   2> 412427 T2355 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 412428 T2355 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 412428 T2355 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 412428 T2355 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 412428 T2355 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 412429 T2355 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 412429 T2355 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 412429 T2355 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 412430 T2355 oascc.SolrZkClient.makePath makePath: /configs/conf1/_rest_managed.json
   [junit4]   2> 412431 T2355 oasr.ManagedResourceStorage$ZooKeeperStorageIO$1.close Wrote 38 bytes to new znode /configs/conf1/_rest_managed.json
   [junit4]   2> 412432 T2355 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 412432 T2355 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 412434 T2356 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@48d7329e[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 412435 T2355 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 412436 T2359 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:37737/orln collection:control_collection shard:shard1
   [junit4]   2> 412436 T2325 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J3
   [junit4]   2> 412436 T2325 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 412436 T2359 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 412437 T2325 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 412438 T2325 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 412440 T2362 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7e3464f1 name:ZooKeeperConnection Watcher:127.0.0.1:47646/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 412440 T2325 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 412440 T2325 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 412441 T2325 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 412441 T2359 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 412443 T2351 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 412443 T2359 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 412443 T2359 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C136 name=collection1 org.apache.solr.core.SolrCore@3982b644 url=http://127.0.0.1:37737/orln/collection1 node=127.0.0.1:37737_orln C136_STATE=coll:control_collection core:collection1 props:{core=collection1, base_url=http://127.0.0.1:37737/orln, node_name=127.0.0.1:37737_orln, state=down}
   [junit4]   2> 412443 T2359 C136 P37737 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:37737/orln/collection1/
   [junit4]   2> 412444 T2359 C136 P37737 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 412444 T2325 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 412444 T2359 C136 P37737 oasc.SyncStrategy.syncToMe http://127.0.0.1:37737/orln/collection1/ has no replicas
   [junit4]   2> 412445 T2325 oasc.AbstractFullDistribZkTestBase.createJettys Creating collection1 with stateFormat=2
   [junit4]   2> 412445 T2359 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:37737/orln/collection1/ shard1
   [junit4]   2> 412446 T2325 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 412446 T2359 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 412447 T2325 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 412448 T2365 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5249c5fa name:ZooKeeperConnection Watcher:127.0.0.1:47646/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 412448 T2325 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 412449 T2325 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 412450 T2351 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 412453 T2352 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: collection1
   [junit4]   2> 412454 T2352 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 412454 T2352 oasc.Overseer$ClusterStateUpdater.createCollection state version collection1 2
   [junit4]   2> 412455 T2351 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 412455 T2352 oasc.Overseer$ClusterStateUpdater.updateZkStates going to create_collection /collections/collection1/state.json
   [junit4]   2> 412456 T2351 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 412456 T2362 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 412502 T2359 oasc.ZkController.register We are http://127.0.0.1:37737/orln/collection1/ and leader is http://127.0.0.1:37737/orln/collection1/
   [junit4]   2> 412503 T2359 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:37737/orln
   [junit4]   2> 412503 T2359 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 412503 T2359 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 412503 T2359 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 412504 T2351 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 412506 T2352 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:37737/orln",
   [junit4]   2> 	  "node_name":"127.0.0.1:37737_orln",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 412609 T2362 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 412609 T2351 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 413587 T2325 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 413587 T2325 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 413590 T2325 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:35189
   [junit4]   2> 413590 T2325 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 413591 T2325 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 413591 T2325 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-103A566D0461F871-001/tempDir-003
   [junit4]   2> 413591 T2325 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-103A566D0461F871-001/tempDir-003/'
   [junit4]   2> 413620 T2325 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-103A566D0461F871-001/tempDir-003/solr.xml
   [junit4]   2> 413633 T2325 oasc.CoreContainer.<init> New CoreContainer 751864104
   [junit4]   2> 413634 T2325 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-103A566D0461F871-001/tempDir-003/]
   [junit4]   2> 413634 T2325 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 413634 T2325 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 413635 T2325 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 413635 T2325 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 413635 T2325 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 413635 T2325 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 413635 T2325 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 413636 T2325 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 413636 T2325 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 413636 T2325 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 413637 T2325 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 413637 T2325 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 413637 T2325 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 413637 T2325 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 413638 T2325 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:47646/solr
   [junit4]   2> 413638 T2325 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 413638 T2325 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 413639 T2325 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 413641 T2377 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3b3bc7d name:ZooKeeperConnection Watcher:127.0.0.1:47646 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 413641 T2325 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 413642 T2325 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 413644 T2325 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 413645 T2380 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4ed9fdbd name:ZooKeeperConnection Watcher:127.0.0.1:47646/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 413645 T2325 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 413649 T2325 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 414653 T2325 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35189_orln
   [junit4]   2> 414654 T2325 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:35189_orln
   [junit4]   2> 414656 T2325 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 414662 T2381 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 414662 T2381 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 414663 T2351 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 414663 T2381 oasc.ZkController.preRegister Registering watch for external collection collection1
   [junit4]   2> 414663 T2381 oascc.ZkStateReader.addZkWatch addZkWatch collection1
   [junit4]   2> 414664 T2381 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/collection1/state.json from ZooKeeper... 
   [junit4]   2> 414665 T2381 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 0 
   [junit4]   2> 414665 T2381 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 414668 T2352 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:35189/orln",
   [junit4]   2> 	  "node_name":"127.0.0.1:35189_orln",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 414670 T2352 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 414671 T2352 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 414673 T2351 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 414674 T2352 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
   [junit4]   2> 414674 T2380 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 414675 T2380 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 1 
   [junit4]   2> 415666 T2381 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 415667 T2381 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 415667 T2381 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 415667 T2381 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 415668 T2381 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 415668 T2381 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-103A566D0461F871-001/tempDir-003/collection1/'
   [junit4]   2> 415670 T2381 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-103A566D0461F871-001/tempDir-003/collection1/lib/classes/' to classloader
   [junit4]   2> 415670 T2381 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-103A566D0461F871-001/tempDir-003/collection1/lib/README' to classloader
   [junit4]   2> 415671 T2381 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-103A566D0461F871-001/tempDir-003/collection1/lib/.svn/' to classloader
   [junit4]   2> 415702 T2381 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 415704 T2381 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 415718 T2381 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 415729 T2381 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 415831 T2381 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 415837 T2381 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 415952 T2381 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 415962 T2381 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 415964 T2381 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 415971 T2381 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 415974 T2381 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 415976 T2381 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 415977 T2381 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 415978 T2381 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 415978 T2381 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 415978 T2381 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 415979 T2381 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 415979 T2381 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 415979 T2381 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 415979 T2381 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-103A566D0461F871-001/tempDir-003/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-103A566D0461F871-001/tempDir-001/jetty1/
   [junit4]   2> 415980 T2381 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@330fda4c
   [junit4]   2> 415981 T2381 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-103A566D0461F871-001/tempDir-001/jetty1
   [junit4]   2> 415981 T2381 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-103A566D0461F871-001/tempDir-001/jetty1/index/
   [junit4]   2> 415981 T2381 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-103A566D0461F871-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 415981 T2381 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-103A566D0461F871-001/tempDir-001/jetty1/index
   [junit4]   2> 415982 T2381 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=30, maxMergeAtOnceExplicit=41, maxMergedSegmentMB=64.537109375, floorSegmentMB=1.693359375, forceMergeDeletesPctAllowed=3.111209946074762, segmentsPerTier=41.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 415983 T2381 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1e365fa1 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@66e9bbe),segFN=segments_1,generation=1}
   [junit4]   2> 415983 T2381 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 415987 T2381 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 415987 T2381 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 415987 T2381 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 415988 T2381 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 415988 T2381 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 415988 T2381 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 415989 T2381 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 415989 T2381 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 415989 T2381 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 415990 T2381 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 415990 T2381 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 415990 T2381 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 415991 T2381 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 415991 T2381 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 415992 T2381 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 415992 T2381 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 415992 T2381 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 415993 T2381 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 415993 T2381 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 415993 T2381 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 415994 T2381 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 416004 T2381 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 416005 T2381 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 416006 T2381 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 416008 T2381 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 416011 T2381 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 416011 T2381 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 416012 T2381 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=16, maxMergeAtOnceExplicit=15, maxMergedSegmentMB=60.369140625, floorSegmentMB=1.87109375, forceMergeDeletesPctAllowed=16.64506491494333, segmentsPerTier=38.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3115152765900482
   [junit4]   2> 416012 T2381 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1e365fa1 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@66e9bbe),segFN=segments_1,generation=1}
   [junit4]   2> 416013 T2381 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 416013 T2381 oass.SolrIndexSearcher.<init> Opening Searcher@1e3921a8[collection1] main
   [junit4]   2> 416013 T2381 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 416014 T2381 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 416014 T2381 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 416014 T2381 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 416015 T2381 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 416015 T2381 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 416016 T2381 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
   [junit4]   2> 416016 T2381 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 416016 T2381 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
   [junit4]   2> 416016 T2381 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 416019 T2382 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1e3921a8[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 416020 T2381 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 416020 T2385 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:35189/orln collection:collection1 shard:shard2
   [junit4]   2> 416021 T2325 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J3
   [junit4]   2> 416021 T2325 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 416021 T2385 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 416025 T2385 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 416026 T2351 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 416026 T2385 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 416026 T2385 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C137 name=collection1 org.apache.solr.core.SolrCore@4e55ac0b url=http://127.0.0.1:35189/orln/collection1 node=127.0.0.1:35189_orln C137_STATE=coll:collection1 core:collection1 props:{core=collection1, base_url=http://127.0.0.1:35189/orln, node_name=127.0.0.1:35189_orln, state=down}
   [junit4]   2> 416026 T2385 C137 P35189 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:35189/orln/collection1/
   [junit4]   2> 416027 T2385 C137 P35189 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 416027 T2385 C137 P35189 oasc.SyncStrategy.syncToMe http://127.0.0.1:35189/orln/collection1/ has no replicas
   [junit4]   2> 416027 T2385 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:35189/orln/collection1/ shard2
   [junit4]   2> 416027 T2385 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 416030 T2352 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
   [junit4]   2> 416030 T2380 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 416031 T2380 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 2 
   [junit4]   2> 416035 T2351 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 416035 T2352 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
   [junit4]   2> 416035 T2380 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 416036 T2380 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 3 
   [junit4]   2> 416081 T2385 oasc.ZkController.register We are http://127.0.0.1:35189/orln/collection1/ and leader is http://127.0.0.1:35189/orln/collection1/
   [junit4]   2> 416082 T2385 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:35189/orln
   [junit4]   2> 416082 T2385 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 416082 T2385 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 416082 T2385 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 416083 T2351 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 416085 T2385 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 3 
   [junit4]   2> 416085 T2352 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:35189/orln",
   [junit4]   2> 	  "node_name":"127.0.0.1:35189_orln",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 416088 T2352 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
   [junit4]   2> 416089 T2380 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 416089 T2380 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 4 
   [junit4]   2> 417139 T2325 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 417140 T2325 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 417142 T2325 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:28438
   [junit4]   2> 417143 T2325 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 417143 T2325 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 417143 T2325 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-103A566D0461F871-001/tempDir-004
   [junit4]   2> 417143 T2325 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-103A566D0461F871-001/tempDir-004/'
   [junit4]   2> 417173 T2325 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-103A566D0461F871-001/tempDir-004/solr.xml
   [junit4]   2> 417186 T2325 oasc.CoreContainer.<init> New CoreContainer 967093745
   [junit4]   2> 417187 T2325 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-103A566D0461F871-001/tempDir-004/]
   [junit4]   2> 417187 T2325 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 417187 T2325 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 417188 T2325 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 417188 T2325 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 417188 T2325 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 417188 T2325 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 417189 T2325 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 417189 T2325 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 417189 T2325 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 417189 T2325 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 417190 T2325 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 417190 T2325 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 417190 T2325 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 417190 T2325 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 417191 T2325 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:47646/solr
   [junit4]   2> 417191 T2325 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 417191 T2325 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 417192 T2325 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 417194 T2397 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d035dfa name:ZooKeeperConnection Watcher:127.0.0.1:47646 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 417195 T2325 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 417195 T2325 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 417197 T2325 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 417199 T2400 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5f3d26d0 name:ZooKeeperConnection Watcher:127.0.0.1:47646/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 417199 T2325 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 417203 T2325 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 418206 T2325 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:28438_orln
   [junit4]   2> 418207 T2325 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:28438_orln
   [junit4]   2> 418210 T2325 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 418215 T2401 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 418215 T2401 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 418216 T2351 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 418216 T2401 oasc.ZkController.preRegister Registering watch for external collection collection1
   [junit4]   2> 418217 T2401 oascc.ZkStateReader.addZkWatch addZkWatch collection1
   [junit4]   2> 418217 T2401 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/collection1/state.json from ZooKeeper... 
   [junit4]   2> 418218 T2401 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 4 
   [junit4]   2> 418218 T2401 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 418218 T2352 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:28438/orln",
   [junit4]   2> 	  "node_name":"127.0.0.1:28438_orln",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 418220 T2352 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 418220 T2352 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 418223 T2352 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
   [junit4]   2> 418224 T2400 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 418224 T2380 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 418224 T2400 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 5 
   [junit4]   2> 418225 T2380 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 5 
   [junit4]   2> 419218 T2401 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 419219 T2401 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 419219 T2401 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 419220 T2401 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 419220 T2401 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 419220 T2401 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-103A566D0461F871-001/tempDir-004/collection1/'
   [junit4]   2> 419222 T2401 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-103A566D0461F871-001/tempDir-004/collection1/lib/README' to classloader
   [junit4]   2> 419223 T2401 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-103A566D0461F871-001/tempDir-004/collection1/lib/.svn/' to classloader
   [junit4]   2> 419224 T2401 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-103A566D0461F871-001/tempDir-004/collection1/lib/classes/' to classloader
   [junit4]   2> 419256 T2401 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 419258 T2401 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 419273 T2401 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 419285 T2401 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 419386 T2401 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 419392 T2401 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 419507 T2401 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 419517 T2401 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 419518 T2401 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 419526 T2401 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 419529 T2401 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 419531 T2401 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 419532 T2401 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 419533 T2401 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 419533 T2401 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 419533 T2401 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 419534 T2401 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 419534 T2401 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 419534 T2401 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 419534 T2401 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-103A566D0461F871-001/tempDir-004/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-103A566D0461F871-001/tempDir-001/jetty2/
   [junit4]   2> 419535 T2401 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@330fda4c
   [junit4]   2> 419536 T2401 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-103A566D0461F871-001/tempDir-001/jetty2
   [junit4]   2> 419536 T2401 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-103A566D0461F871-001/tempDir-001/jetty2/index/
   [junit4]   2> 419536 T2401 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-103A566D0461F871-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 419537 T2401 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-103A566D0461F871-001/tempDir-001/jetty2/index
   [junit4]   2> 419537 T2401 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=30, maxMergeAtOnceExplicit=41, maxMergedSegmentMB=64.537109375, floorSegmentMB=1.693359375, forceMergeDeletesPctAllowed=3.111209946074762, segmentsPerTier=41.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 419538 T2401 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@4ad441db lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@505677e5),segFN=segments_1,generation=1}
   [junit4]   2> 419538 T2401 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 419542 T2401 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 419543 T2401 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 419543 T2401 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 419543 T2401 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 419543 T2401 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 419544 T2401 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 419544 T2401 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 419544 T2401 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 419545 T2401 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 419545 T2401 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 419546 T2401 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 419546 T2401 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 419546 T2401 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 419547 T2401 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 419548 T2401 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 419548 T2401 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 419548 T2401 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 419548 T2401 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 419549 T2401 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 419549 T2401 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 419550 T2401 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 419560 T2401 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 419562 T2401 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 419563 T2401 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 419564 T2401 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 419567 T2401 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 419568 T2401 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 419569 T2401 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=16, maxMergeAtOnceExplicit=15, maxMergedSegmentMB=60.369140625, floorSegmentMB=1.87109375, forceMergeDeletesPctAllowed=16.64506491494333, segmentsPerTier=38.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3115152765900482
   [junit4]   2> 419569 T2401 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@4ad441db lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@505677e5),segFN=segments_1,generation=1}
   [junit4]   2> 419570 T2401 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 419570 T2401 oass.SolrIndexSearcher.<init> Opening Searcher@62dc7557[collection1] main
   [junit4]   2> 419570 T2401 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 419571 T2401 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 419571 T2401 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 419572 T2401 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 419572 T2401 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 419572 T2401 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 419573 T2401 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
   [junit4]   2> 419573 T2401 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 419573 T2401 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
   [junit4]   2> 419573 T2401 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 419577 T2402 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@62dc7557[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 419577 T2401 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 419578 T2405 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:28438/orln collection:co

[...truncated too long message...]

-Tests-trunk-Java8/solr/core/lib/org.restlet.ext.servlet-2.1.1.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/lib/protobuf-java-2.5.0.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/lib/slf4j-log4j12-1.7.6.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/lib/spatial4j-0.4.1.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/solrj/lib/commons-io-2.4.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/solrj/lib/httpclient-4.3.1.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/solrj/lib/httpcore-4.3.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/solrj/lib/httpmime-4.3.1.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/solrj/lib/noggit-0.6.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/solrj/lib/slf4j-api-1.7.6.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/solrj/lib/wstx-asl-3.2.7.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/solrj/lib/zookeeper-3.4.6.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/server/lib/jetty-continuation-8.1.10.v20130312.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/server/lib/jetty-deploy-8.1.10.v20130312.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/server/lib/jetty-http-8.1.10.v20130312.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/server/lib/jetty-io-8.1.10.v20130312.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/server/lib/jetty-jmx-8.1.10.v20130312.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/server/lib/jetty-security-8.1.10.v20130312.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/server/lib/jetty-server-8.1.10.v20130312.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/server/lib/jetty-servlet-8.1.10.v20130312.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/server/lib/jetty-util-8.1.10.v20130312.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/server/lib/jetty-webapp-8.1.10.v20130312.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/server/lib/jetty-xml-8.1.10.v20130312.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/server/lib/servlet-api-3.0.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/example/example-DIH/solr/db/lib/derby-10.9.1.0.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/example/example-DIH/solr/db/lib/hsqldb-1.8.0.10.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/lucene/build/core/classes/java:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/lucene/test-framework/lib/junit-4.10.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/lucene/test-framework/lib/randomizedtesting-runner-2.1.9.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/test-lib/cglib-nodep-2.2.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/test-lib/commons-collections-3.2.1.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/test-lib/easymock-3.0.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/test-lib/hadoop-common-2.2.0-tests.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/test-lib/hadoop-hdfs-2.2.0-tests.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/test-lib/javax.servlet-api-3.0.1.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/test-lib/jcl-over-slf4j-1.7.6.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/test-lib/jersey-core-1.9.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/test-lib/jetty-6.1.26.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/test-lib/jetty-util-6.1.26.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/test-lib/objenesis-1.2.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/lucene/build/analysis/icu/lucene-analyzers-icu-6.0.0-SNAPSHOT.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/contrib/solr-analysis-extras/classes/java:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/contrib/analysis-extras/lib/icu4j-54.1.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-launcher.jar:/home/jenkins/.ant/lib/ivy-2.3.0.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jsch.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jai.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-regexp.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-netrexx.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-antlr.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-junit4.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-junit.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-bsf.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-log4j.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-bcel.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-resolver.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-swing.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-oro.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-xalan2.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jmf.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jdepend.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-testutil.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-commons-logging.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-commons-net.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-javamail.jar:/usr/local/openjdk8/lib/tools.jar:/home/jenkins/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.1.9.jar com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe -eventsfile /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/temp/junit4-J2-20141121_173810_288.events @/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/temp/junit4-J2-20141121_173810_288.suites
   [junit4] ERROR: JVM J2 ended with an exception: Forked process returned with error code: 1. Very likely a JVM crash.  Process output piped in logs above.
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1481)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:133)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:964)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:961)
   [junit4] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4] 	at java.lang.Thread.run(Thread.java:745)

BUILD FAILED
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/build.xml:515: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/build.xml:463: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/build.xml:61: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/extra-targets.xml:39: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build.xml:186: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/common-build.xml:508: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/lucene/common-build.xml:1345: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/lucene/common-build.xml:952: At least one slave process threw an exception, first: Forked process returned with error code: 134. Very likely a JVM crash.  Process output piped in logs above.

Total time: 92 minutes 30 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-Tests-trunk-Java8 - Build # 10 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-trunk-Java8/10/

1 tests failed.
FAILED:  org.apache.solr.cloud.TestSolrCloudWithKerberos.testKerberizedSolr

Error Message:
No live SolrServers available to handle this request:[http://127.0.0.1:39003, http://127.0.0.1:51276, http://127.0.0.1:33817, http://127.0.0.1:40748, http://127.0.0.1:39669]

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:39003, http://127.0.0.1:51276, http://127.0.0.1:33817, http://127.0.0.1:40748, http://127.0.0.1:39669]
	at __randomizedtesting.SeedInfo.seed([7F80B71B49A3C81:ACE1DDB6FEFBFCB6]:0)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:355)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1086)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:856)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:799)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:135)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:152)
	at org.apache.solr.cloud.TestSolrCloudWithKerberos.testKerberizedSolr(TestSolrCloudWithKerberos.java:159)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:960)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:935)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:39669: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 500 </title>
</head>
<body>
<h2>HTTP ERROR: 500</h2>
<p>Problem accessing /admin/collections. Reason:
<pre>    org.apache.solr.common.SolrException: Error during request authentication, </pre></p>
<hr /><i><small>Powered by Jetty://</small></i>
</body>
</html>

	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:529)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:235)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:227)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:376)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:328)
	... 47 more




Build Log:
[...truncated 10521 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestSolrCloudWithKerberos
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/init-core-data-001
   [junit4]   2> 1478245 T8713 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 1483607 T8713 oadsc.DefaultDirectoryService.showSecurityWarnings WARN You didn't change the admin password of directory service instance 'DefaultKrbServer'.  Please update the admin password as soon as possible to prevent a possible security breach.
   [junit4]   2> 1484334 T8713 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 1484334 T8722 oasc.ZkTestServer$2$1.setClientPort client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1484335 T8722 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1484435 T8713 oasc.ZkTestServer.run start zk server on port:40569
   [junit4]   2> 1484437 T8729 oascc.ConnectionManager.process WARN zkClient received AuthFailed
   [junit4]   2> 1484447 T8732 oascc.ConnectionManager.process WARN zkClient received AuthFailed
   [junit4]   2> 1484452 T8713 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1484454 T8713 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 1484456 T8713 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1484458 T8713 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1484459 T8713 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1484461 T8713 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1484462 T8713 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1484464 T8713 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1484465 T8713 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1484467 T8713 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1484468 T8713 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1484472 T8736 oascc.ConnectionManager.process WARN zkClient received AuthFailed
   [junit4]   2> 1484812 T8713 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/control-001/cores/collection1
   [junit4]   2> 1484814 T8713 oejs.Server.doStart jetty-9.2.10.v20150310
   [junit4]   2> 1484816 T8713 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@4436ea90{/,null,AVAILABLE}
   [junit4]   2> 1484817 T8713 oejs.AbstractConnector.doStart Started ServerConnector@3a49cb8c{HTTP/1.1}{127.0.0.1:40748}
   [junit4]   2> 1484817 T8713 oejs.Server.doStart Started @1486683ms
   [junit4]   2> 1484817 T8713 oascse.JettySolrRunner$1.lifeCycleStarted Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/tempDir-002/control/data, hostContext=/, hostPort=40748, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/control-001/cores}
   [junit4]   2> 1484818 T8713 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@c387f44
   [junit4]   2> 1484818 T8713 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/control-001/'
   [junit4]   2> 1485974 T8713 oasc.SolrXmlConfig.fromFile Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/control-001/solr.xml
   [junit4]   2> 1485980 T8713 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/control-001/cores
   [junit4]   2> 1485981 T8713 oasc.CoreContainer.<init> New CoreContainer 222157298
   [junit4]   2> 1485981 T8713 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/control-001/]
   [junit4]   2> 1485981 T8713 oasc.CoreContainer.load loading shared library: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/control-001/lib
   [junit4]   2> 1485982 T8713 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/control-001/lib).
   [junit4]   2> 1485998 T8713 oashc.HttpShardHandlerFactory.init created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 1486001 T8713 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1486002 T8713 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1486002 T8713 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1486003 T8713 oasc.CoreContainer.load Node Name: 127.0.0.1
   [junit4]   2> 1486003 T8713 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:40569/solr
   [junit4]   2> 1486003 T8713 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1486004 T8751 oascc.ConnectionManager.process WARN zkClient received AuthFailed
   [junit4]   2> 1486015 T8755 n:127.0.0.1:40748_ oascc.ConnectionManager.process WARN zkClient received AuthFailed
   [junit4]   2> 1486025 T8713 n:127.0.0.1:40748_ oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40748_
   [junit4]   2> 1486030 T8713 n:127.0.0.1:40748_ oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1486031 T8713 n:127.0.0.1:40748_ oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:40748_
   [junit4]   2> 1486033 T8713 n:127.0.0.1:40748_ oasc.Overseer.start Overseer (id=93883821892239364-127.0.0.1:40748_-n_0000000000) starting
   [junit4]   2> 1486040 T8713 n:127.0.0.1:40748_ oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 1486041 T8758 n:127.0.0.1:40748_ oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1486042 T8757 n:127.0.0.1:40748_ oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1486045 T8713 n:127.0.0.1:40748_ oasc.CoreContainer.initializeAuthenticationPlugin Authentication plugin class obtained from ZK: org.apache.solr.security.KerberosPlugin
   [junit4]   2> 1486054 T8713 n:127.0.0.1:40748_ oass.KerberosPlugin.init Params: {token.valid=30, kerberos.principal=HTTP/127.0.0.1, cookie.domain=127.0.0.1, kerberos.keytab=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/tempDir-001/minikdc/keytabs, cookie.path=/, type=kerberos, kerberos.name.rules=DEFAULT}
   [junit4]   2> 1486075 T8713 n:127.0.0.1:40748_ oasc.CoreContainer.initializeAuthenticationPlugin Reconfiguring the shard handler factory and update shard handler.
   [junit4]   2> 1486076 T8713 n:127.0.0.1:40748_ oashc.HttpShardHandlerFactory.reconfigureHttpClient Reconfiguring the default client with: org.apache.solr.client.solrj.impl.Krb5HttpClientConfigurer@5c575b56
   [junit4]   2> 1486077 T8713 n:127.0.0.1:40748_ oascsi.Krb5HttpClientConfigurer.configure Setting up SPNego auth with config: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/tempDir-001/minikdc/jaas-client.conf
   [junit4]   2> 1486080 T8713 n:127.0.0.1:40748_ oasu.UpdateShardHandler.reconfigureHttpClient Reconfiguring the default client with: org.apache.solr.client.solrj.impl.Krb5HttpClientConfigurer@5c575b56
   [junit4]   2> 1486080 T8713 n:127.0.0.1:40748_ oascsi.Krb5HttpClientConfigurer.configure Setting up SPNego auth with config: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/tempDir-001/minikdc/jaas-client.conf
   [junit4]   2> 1486083 T8713 n:127.0.0.1:40748_ oasc.CorePropertiesLocator.discover Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/control-001/cores
   [junit4]   2> 1486084 T8713 n:127.0.0.1:40748_ oasc.CoreDescriptor.<init> CORE DESCRIPTOR: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/control-001/cores/collection1, collection=control_collection, absoluteInstDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/control-001/cores/collection1/, coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 1486084 T8713 n:127.0.0.1:40748_ oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/control-001/cores/collection1/
   [junit4]   2> 1486085 T8713 n:127.0.0.1:40748_ oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1486086 T8761 n:127.0.0.1:40748_ c:control_collection x:collection1 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 1486086 T8761 n:127.0.0.1:40748_ c:control_collection x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1486087 T8755 n:127.0.0.1:40748_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1486087 T8761 n:127.0.0.1:40748_ oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1486088 T8757 n:127.0.0.1:40748_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:40748",
   [junit4]   2> 	  "node_name":"127.0.0.1:40748_",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"} current state version: 0
   [junit4]   2> 1486088 T8757 n:127.0.0.1:40748_ oasco.ReplicaMutator.updateState Update state numShards=1 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:40748",
   [junit4]   2> 	  "node_name":"127.0.0.1:40748_",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 1486088 T8757 n:127.0.0.1:40748_ oasco.ClusterStateMutator.createCollection building a new cName: control_collection
   [junit4]   2> 1486089 T8757 n:127.0.0.1:40748_ oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1487087 T8761 n:127.0.0.1:40748_ oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1487088 T8761 n:127.0.0.1:40748_ oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 1487091 T8761 n:127.0.0.1:40748_ oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1487092 T8761 n:127.0.0.1:40748_ oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/control-001/cores/collection1/'
   [junit4]   2> 1487113 T8761 n:127.0.0.1:40748_ oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 1487120 T8761 n:127.0.0.1:40748_ oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 1487139 T8761 n:127.0.0.1:40748_ oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 1487148 T8761 n:127.0.0.1:40748_ oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1487149 T8761 n:127.0.0.1:40748_ oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 1487155 T8761 n:127.0.0.1:40748_ oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1487275 T8761 n:127.0.0.1:40748_ oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1487281 T8761 n:127.0.0.1:40748_ oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1487283 T8761 n:127.0.0.1:40748_ oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1487292 T8761 n:127.0.0.1:40748_ oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1487294 T8761 n:127.0.0.1:40748_ oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1487296 T8761 n:127.0.0.1:40748_ oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1487296 T8761 n:127.0.0.1:40748_ oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1487297 T8761 n:127.0.0.1:40748_ oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1487297 T8761 n:127.0.0.1:40748_ oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1487298 T8761 n:127.0.0.1:40748_ oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1487298 T8761 n:127.0.0.1:40748_ oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1487298 T8761 n:127.0.0.1:40748_ oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 1487298 T8761 n:127.0.0.1:40748_ x:collection1 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1487299 T8761 n:127.0.0.1:40748_ x:collection1 oasc.SolrCore.<init> [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/control-001/cores/collection1/], dataDir=[null]
   [junit4]   2> 1487299 T8761 n:127.0.0.1:40748_ x:collection1 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@17cfcf30
   [junit4]   2> 1487299 T8761 n:127.0.0.1:40748_ x:collection1 oasc.CachingDirectoryFactory.get return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/control-001/cores/collection1/data
   [junit4]   2> 1487300 T8761 n:127.0.0.1:40748_ x:collection1 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/control-001/cores/collection1/data/index/
   [junit4]   2> 1487301 T8761 n:127.0.0.1:40748_ x:collection1 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/control-001/cores/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1487301 T8761 n:127.0.0.1:40748_ x:collection1 oasc.CachingDirectoryFactory.get return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/control-001/cores/collection1/data/index
   [junit4]   2> 1487301 T8761 n:127.0.0.1:40748_ x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=33, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3006284492664655]
   [junit4]   2> 1487302 T8761 n:127.0.0.1:40748_ x:collection1 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@3229d82c lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@477df76e),segFN=segments_1,generation=1}
   [junit4]   2> 1487302 T8761 n:127.0.0.1:40748_ x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1487306 T8761 n:127.0.0.1:40748_ x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1487306 T8761 n:127.0.0.1:40748_ x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1487306 T8761 n:127.0.0.1:40748_ x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1487307 T8761 n:127.0.0.1:40748_ x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1487307 T8761 n:127.0.0.1:40748_ x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1487307 T8761 n:127.0.0.1:40748_ x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1487307 T8761 n:127.0.0.1:40748_ x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1487308 T8761 n:127.0.0.1:40748_ x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1487308 T8761 n:127.0.0.1:40748_ x:collection1 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1487310 T8761 n:127.0.0.1:40748_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1487311 T8761 n:127.0.0.1:40748_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1487312 T8761 n:127.0.0.1:40748_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1487313 T8761 n:127.0.0.1:40748_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1487316 T8761 n:127.0.0.1:40748_ x:collection1 oasc.RequestHandlers.initHandlersFromConfig Registered paths: /admin/mbeans,standard,/update/csv,/update/json/docs,/admin/luke,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/schema,/admin/plugins,/admin/logging,/update/json,/admin/threads,/admin/ping,/update,/admin/file
   [junit4]   2> 1487317 T8761 n:127.0.0.1:40748_ x:collection1 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 1487322 T8761 n:127.0.0.1:40748_ x:collection1 oasu.UpdateHandler.<init> Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1487323 T8761 n:127.0.0.1:40748_ x:collection1 oasu.UpdateLog.init Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1487325 T8761 n:127.0.0.1:40748_ x:collection1 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1487325 T8761 n:127.0.0.1:40748_ x:collection1 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1487326 T8761 n:127.0.0.1:40748_ x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=40, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 1487327 T8761 n:127.0.0.1:40748_ x:collection1 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@3229d82c lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@477df76e),segFN=segments_1,generation=1}
   [junit4]   2> 1487327 T8761 n:127.0.0.1:40748_ x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1487327 T8761 n:127.0.0.1:40748_ x:collection1 oass.SolrIndexSearcher.<init> Opening Searcher@6edfb126[collection1] main
   [junit4]   2> 1487328 T8761 n:127.0.0.1:40748_ x:collection1 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1487329 T8761 n:127.0.0.1:40748_ x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1487329 T8761 n:127.0.0.1:40748_ x:collection1 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 1487329 T8761 n:127.0.0.1:40748_ x:collection1 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1487330 T8761 n:127.0.0.1:40748_ x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1487330 T8761 n:127.0.0.1:40748_ x:collection1 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1487330 T8761 n:127.0.0.1:40748_ x:collection1 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 1487331 T8761 n:127.0.0.1:40748_ x:collection1 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 1487331 T8763 n:127.0.0.1:40748_ x:collection1 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6edfb126[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1487332 T8761 n:127.0.0.1:40748_ x:collection1 oasc.ZkController.getConfDirListeners watch zkdir /configs/conf1
   [junit4]   2> 1487332 T8761 n:127.0.0.1:40748_ x:collection1 oasu.UpdateLog.onFirstSearcher On first searcher opened, looking up max value of version field
   [junit4]   2> 1487332 T8761 n:127.0.0.1:40748_ x:collection1 oasu.VersionInfo.getMaxVersionFromIndex Refreshing highest value of _version_ for 65536 version buckets from index
   [junit4]   2> 1487333 T8761 n:127.0.0.1:40748_ x:collection1 oasu.VersionInfo.getMaxVersionFromIndex No terms found for _version_, cannot seed version bucket highest value from index
   [junit4]   2> 1487333 T8761 n:127.0.0.1:40748_ x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion Could not find max version in index or recent updates, using new clock 1502141153418412032
   [junit4]   2> 1487335 T8761 n:127.0.0.1:40748_ x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion Took 3 ms to seed version buckets with highest version 1502141153418412032
   [junit4]   2> 1487336 T8761 n:127.0.0.1:40748_ x:collection1 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1487341 T8766 n:127.0.0.1:40748_ c:control_collection s:shard1 x:collection1 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:40748 collection:control_collection shard:shard1
   [junit4]   2> 1487341 T8713 n:127.0.0.1:40748_ oascsi.Krb5HttpClientConfigurer.configure Setting up SPNego auth with config: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/tempDir-001/minikdc/jaas-client.conf
   [junit4]   2> 1487342 T8713 n:127.0.0.1:40748_ oass.SolrDispatchFilter.init user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2
   [junit4]   2> 1487342 T8713 n:127.0.0.1:40748_ oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1487343 T8713 oascsi.Krb5HttpClientConfigurer.configure Setting up SPNego auth with config: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/tempDir-001/minikdc/jaas-client.conf
   [junit4]   2> 1487344 T8713 oascsi.Krb5HttpClientConfigurer.configure Setting up SPNego auth with config: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/tempDir-001/minikdc/jaas-client.conf
   [junit4]   2> 1487347 T8769 oascc.ConnectionManager.process WARN zkClient received AuthFailed
   [junit4]   2> 1487352 T8766 n:127.0.0.1:40748_ c:control_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 1487354 T8755 n:127.0.0.1:40748_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1487354 T8766 n:127.0.0.1:40748_ c:control_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1487355 T8757 n:127.0.0.1:40748_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection"} current state version: 1
   [junit4]   2> 1487355 T8766 n:127.0.0.1:40748_ c:control_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C1209 name=collection1 org.apache.solr.core.SolrCore@c68063c url=http://127.0.0.1:40748/collection1 node=127.0.0.1:40748_ C1209_STATE=coll:control_collection core:collection1 props:{core=collection1, base_url=http://127.0.0.1:40748, node_name=127.0.0.1:40748_, state=down}
   [junit4]   2> 1487356 T8766 n:127.0.0.1:40748_ c:control_collection s:shard1 x:collection1 C1209 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:40748/collection1/
   [junit4]   2> 1487356 T8766 n:127.0.0.1:40748_ c:control_collection s:shard1 x:collection1 C1209 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1487357 T8766 n:127.0.0.1:40748_ c:control_collection s:shard1 x:collection1 C1209 oasc.SyncStrategy.syncToMe http://127.0.0.1:40748/collection1/ has no replicas
   [junit4]   2> 1487357 T8766 n:127.0.0.1:40748_ c:control_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:40748/collection1/ shard1
   [junit4]   2> 1487360 T8755 n:127.0.0.1:40748_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1487361 T8713 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1487361 T8757 n:127.0.0.1:40748_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:40748",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "state":"active"} current state version: 1
   [junit4]   2> 1487512 T8766 n:127.0.0.1:40748_ c:control_collection s:shard1 x:collection1 oasc.ZkController.register We are http://127.0.0.1:40748/collection1/ and leader is http://127.0.0.1:40748/collection1/
   [junit4]   2> 1487512 T8766 n:127.0.0.1:40748_ c:control_collection s:shard1 x:collection1 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:40748
   [junit4]   2> 1487513 T8766 n:127.0.0.1:40748_ c:control_collection s:shard1 x:collection1 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1487513 T8766 n:127.0.0.1:40748_ c:control_collection s:shard1 x:collection1 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 1487513 T8766 n:127.0.0.1:40748_ c:control_collection s:shard1 x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1487514 T8755 n:127.0.0.1:40748_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1487515 T8757 n:127.0.0.1:40748_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:40748",
   [junit4]   2> 	  "node_name":"127.0.0.1:40748_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"} current state version: 2
   [junit4]   2> 1487516 T8757 n:127.0.0.1:40748_ oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:40748",
   [junit4]   2> 	  "node_name":"127.0.0.1:40748_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 1487719 T8713 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/shard-1-001/cores/collection1
   [junit4]   2> 1487721 T8713 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/shard-1-001
   [junit4]   2> 1487722 T8713 oejs.Server.doStart jetty-9.2.10.v20150310
   [junit4]   2> 1487727 T8713 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@2eab3622{/,null,AVAILABLE}
   [junit4]   2> 1487728 T8713 oejs.AbstractConnector.doStart Started ServerConnector@31002725{HTTP/1.1}{127.0.0.1:39003}
   [junit4]   2> 1487729 T8713 oejs.Server.doStart Started @1489595ms
   [junit4]   2> 1487729 T8713 oascse.JettySolrRunner$1.lifeCycleStarted Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/tempDir-002/jetty1, solrconfig=solrconfig.xml, hostContext=/, hostPort=39003, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/shard-1-001/cores}
   [junit4]   2> 1487729 T8713 oascsi.Krb5HttpClientConfigurer.configure Setting up SPNego auth with config: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/tempDir-001/minikdc/jaas-client.conf
   [junit4]   2> 1487730 T8713 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@c387f44
   [junit4]   2> 1487730 T8713 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/shard-1-001/'
   [junit4]   2> 1487745 T8713 oasc.SolrXmlConfig.fromFile Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/shard-1-001/solr.xml
   [junit4]   2> 1487750 T8713 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/shard-1-001/cores
   [junit4]   2> 1487751 T8713 oasc.CoreContainer.<init> New CoreContainer 454062642
   [junit4]   2> 1487751 T8713 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/shard-1-001/]
   [junit4]   2> 1487751 T8713 oasc.CoreContainer.load loading shared library: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/shard-1-001/lib
   [junit4]   2> 1487751 T8713 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/shard-1-001/lib).
   [junit4]   2> 1487758 T8713 oashc.HttpShardHandlerFactory.init created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 1487759 T8713 oascsi.Krb5HttpClientConfigurer.configure Setting up SPNego auth with config: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/tempDir-001/minikdc/jaas-client.conf
   [junit4]   2> 1487760 T8713 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1487761 T8713 oascsi.Krb5HttpClientConfigurer.configure Setting up SPNego auth with config: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/tempDir-001/minikdc/jaas-client.conf
   [junit4]   2> 1487761 T8713 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1487761 T8713 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1487762 T8713 oasc.CoreContainer.load Node Name: 127.0.0.1
   [junit4]   2> 1487762 T8713 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:40569/solr
   [junit4]   2> 1487762 T8713 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1487764 T8783 oascc.ConnectionManager.process WARN zkClient received AuthFailed
   [junit4]   2> 1487767 T8786 n:127.0.0.1:39003_ oascc.ConnectionManager.process WARN zkClient received AuthFailed
   [junit4]   2> 1488783 T8713 n:127.0.0.1:39003_ oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39003_
   [junit4]   2> 1488786 T8713 n:127.0.0.1:39003_ oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1488788 T8713 n:127.0.0.1:39003_ oasc.CoreContainer.initializeAuthenticationPlugin Authentication plugin class obtained from ZK: org.apache.solr.security.KerberosPlugin
   [junit4]   2> 1488788 T8713 n:127.0.0.1:39003_ oass.KerberosPlugin.init Params: {token.valid=30, kerberos.principal=HTTP/127.0.0.1, cookie.domain=127.0.0.1, kerberos.keytab=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/tempDir-001/minikdc/keytabs, cookie.path=/, type=kerberos, kerberos.name.rules=DEFAULT}
   [junit4]   2> 1488790 T8713 n:127.0.0.1:39003_ oasc.CoreContainer.initializeAuthenticationPlugin Reconfiguring the shard handler factory and update shard handler.
   [junit4]   2> 1488790 T8713 n:127.0.0.1:39003_ oashc.HttpShardHandlerFactory.reconfigureHttpClient Reconfiguring the default client with: org.apache.solr.client.solrj.impl.Krb5HttpClientConfigurer@4334d2
   [junit4]   2> 1488791 T8713 n:127.0.0.1:39003_ oascsi.Krb5HttpClientConfigurer.configure Setting up SPNego auth with config: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/tempDir-001/minikdc/jaas-client.conf
   [junit4]   2> 1488791 T8713 n:127.0.0.1:39003_ oasu.UpdateShardHandler.reconfigureHttpClient Reconfiguring the default client with: org.apache.solr.client.solrj.impl.Krb5HttpClientConfigurer@4334d2
   [junit4]   2> 1488791 T8713 n:127.0.0.1:39003_ oascsi.Krb5HttpClientConfigurer.configure Setting up SPNego auth with config: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/tempDir-001/minikdc/jaas-client.conf
   [junit4]   2> 1488793 T8713 n:127.0.0.1:39003_ oasc.CorePropertiesLocator.discover Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/shard-1-001/cores
   [junit4]   2> 1488794 T8713 n:127.0.0.1:39003_ oasc.CoreDescriptor.<init> CORE DESCRIPTOR: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/shard-1-001/cores/collection1, collection=collection1, absoluteInstDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/shard-1-001/cores/collection1/, coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 1488794 T8713 n:127.0.0.1:39003_ oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/shard-1-001/cores/collection1/
   [junit4]   2> 1488795 T8713 n:127.0.0.1:39003_ oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1488799 T8788 n:127.0.0.1:39003_ c:collection1 x:collection1 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 1488799 T8788 n:127.0.0.1:39003_ c:collection1 x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1488800 T8788 n:127.0.0.1:39003_ oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1488800 T8755 n:127.0.0.1:40748_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1488801 T8757 n:127.0.0.1:40748_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:39003",
   [junit4]   2> 	  "node_name":"127.0.0.1:39003_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"} current state version: 3
   [junit4]   2> 1488801 T8757 n:127.0.0.1:40748_ oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:39003",
   [junit4]   2> 	  "node_name":"127.0.0.1:39003_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 1488802 T8757 n:127.0.0.1:40748_ oasco.ClusterStateMutator.createCollection building a new cName: collection1
   [junit4]   2> 1488802 T8757 n:127.0.0.1:40748_ oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 1489800 T8788 n:127.0.0.1:39003_ oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1489801 T8788 n:127.0.0.1:39003_ oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 1489801 T8788 n:127.0.0.1:39003_ oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1489802 T8788 n:127.0.0.1:39003_ oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/shard-1-001/cores/collection1/'
   [junit4]   2> 1489814 T8788 n:127.0.0.1:39003_ oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 1489820 T8788 n:127.0.0.1:39003_ oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 1489826 T8788 n:127.0.0.1:39003_ oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 1489841 T8788 n:127.0.0.1:39003_ oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1489842 T8788 n:127.0.0.1:39003_ oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 1489854 T8788 n:127.0.0.1:39003_ oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1490019 T8788 n:127.0.0.1:39003_ oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1490025 T8788 n:127.0.0.1:39003_ oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1490026 T8788 n:127.0.0.1:39003_ oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1490034 T8788 n:127.0.0.1:39003_ oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1490036 T8788 n:127.0.0.1:39003_ oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1490038 T8788 n:127.0.0.1:39003_ oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1490039 T8788 n:127.0.0.1:39003_ oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1490039 T8788 n:127.0.0.1:39003_ oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1490039 T8788 n:127.0.0.1:39003_ oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1490040 T8788 n:127.0.0.1:39003_ oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1490040 T8788 n:127.0.0.1:39003_ oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1490040 T8788 n:127.0.0.1:39003_ oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 1490040 T8788 n:127.0.0.1:39003_ x:collection1 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1490041 T8788 n:127.0.0.1:39003_ x:collection1 oasc.SolrCore.<init> [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/shard-1-001/cores/collection1/], dataDir=[null]
   [junit4]   2> 1490041 T8788 n:127.0.0.1:39003_ x:collection1 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@17cfcf30
   [junit4]   2> 1490041 T8788 n:127.0.0.1:39003_ x:collection1 oasc.CachingDirectoryFactory.get return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/shard-1-001/cores/collection1/data
   [junit4]   2> 1490042 T8788 n:127.0.0.1:39003_ x:collection1 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/shard-1-001/cores/collection1/data/index/
   [junit4]   2> 1490042 T8788 n:127.0.0.1:39003_ x:collection1 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/shard-1-001/cores/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1490042 T8788 n:127.0.0.1:39003_ x:collection1 oasc.CachingDirectoryFactory.get return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/shard-1-001/cores/collection1/data/index
   [junit4]   2> 1490043 T8788 n:127.0.0.1:39003_ x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=33, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3006284492664655]
   [junit4]   2> 1490043 T8788 n:127.0.0.1:39003_ x:collection1 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@7211656a lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@293e526a),segFN=segments_1,generation=1}
   [junit4]   2> 1490043 T8788 n:127.0.0.1:39003_ x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1490047 T8788 n:127.0.0.1:39003_ x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1490048 T8788 n:127.0.0.1:39003_ x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1490048 T8788 n:127.0.0.1:39003_ x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1490048 T8788 n:127.0.0.1:39003_ x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1490048 T8788 n:127.0.0.1:39003_ x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1490048 T8788 n:127.0.0.1:39003_ x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1490049 T8788 n:127.0.0.1:39003_ x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1490049 T8788 n:127.0.0.1:39003_ x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1490049 T8788 n:127.0.0.1:39003_ x:collection1 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1490051 T8788 n:127.0.0.1:39003_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1490052 T8788 n:127.0.0.1:39003_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1490053 T8788 n:127.0.0.1:39003_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1490054 T8788 n:127.0.0.1:39003_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1490058 T8788 n:127.0.0.1:39003_ x:collection1 oasc.RequestHandlers.initHandlersFromConfig Registered paths: /admin/mbeans,standard,/update/csv,/update/json/docs,/admin/luke,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/schema,/admin/plugins,/admin/logging,/update/json,/admin/threads,/admin/ping,/update,/admin/file
   [junit4]   2> 1490059 T8788 n:127.0.0.1:39003_ x:collection1 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 1490060 T8788 n:127.0.0.1:39003_ x:collection1 oasu.UpdateHandler.<init> Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1490060 T8788 n:127.0.0.1:39003_ x:collection1 oasu.UpdateLog.init Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1490061 T8788 n:127.0.0.1:39003_ x:collection1 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1490061 T8788 n:127.0.0.1:39003_ x:collection1 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1490062 T8788 n:127.0.0.1:39003_ x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=40, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 1490062 T8788 n:127.0.0.1:39003_ x:collection1 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@7211656a lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@293e526a),segFN=segments_1,generation=1}
   [junit4]   2> 1490062 T8788 n:127.0.0.1:39003_ x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1490063 T8788 n:127.0.0.1:39003_ x:collection1 oass.SolrIndexSearcher.<init> Opening Searcher@365f21f0[collection1] main
   [junit4]   2> 1490070 T8788 n:127.0.0.1:39003_ x:collection1 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1490071 T8788 n:127.0.0.1:39003_ x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1490071 T8788 n:127.0.0.1:39003_ x:collection1 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 1490071 T8788 n:127.0.0.1:39003_ x:collection1 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1490072 T8788 n:127.0.0.1:39003_ x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1490072 T8788 n:127.0.0.1:39003_ x:collection1 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1490072 T8788 n:127.0.0.1:39003_ x:collection1 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 1490072 T8788 n:127.0.0.1:39003_ x:collection1 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 1490073 T8788 n:127.0.0.1:39003_ x:collection1 oasc.ZkController.getConfDirListeners watch zkdir /configs/conf1
   [junit4]   2> 1490074 T8789 n:127.0.0.1:39003_ x:collection1 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@365f21f0[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1490074 T8788 n:127.0.0.1:39003_ x:collection1 oasu.UpdateLog.onFirstSearcher On first searcher opened, looking up max value of version field
   [junit4]   2> 1490074 T8788 n:127.0.0.1:39003_ x:collection1 oasu.VersionInfo.getMaxVersionFromIndex Refreshing highest value of _version_ for 65536 version buckets from index
   [junit4]   2> 1490074 T8788 n:127.0.0.1:39003_ x:collection1 oasu.VersionInfo.getMaxVersionFromIndex No terms found for _version_, cannot seed version bucket highest value from index
   [junit4]   2> 1490074 T8788 n:127.0.0.1:39003_ x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion Could not find max version in index or recent updates, using new clock 1502141156292558848
   [junit4]   2> 1490077 T8788 n:127.0.0.1:39003_ x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion Took 3 ms to seed version buckets with highest version 1502141156292558848
   [junit4]   2> 1490077 T8788 n:127.0.0.1:39003_ x:collection1 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1490078 T8792 n:127.0.0.1:39003_ c:collection1 s:shard2 x:collection1 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:39003 collection:collection1 shard:shard2
   [junit4]   2> 1490078 T8713 n:127.0.0.1:39003_ oascsi.Krb5HttpClientConfigurer.configure Setting up SPNego auth with config: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/tempDir-001/minikdc/jaas-client.conf
   [junit4]   2> 1490079 T8713 n:127.0.0.1:39003_ oass.SolrDispatchFilter.init user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2
   [junit4]   2> 1490079 T8713 n:127.0.0.1:39003_ oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1490080 T8713 oascsi.Krb5HttpClientConfigurer.configure Setting up SPNego auth with config: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/tempDir-001/minikdc/jaas-client.conf
   [junit4]   2> 1490082 T8792 n:127.0.0.1:39003_ c:collection1 s:shard2 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 1490084 T8755 n:127.0.0.1:40748_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1490084 T8792 n:127.0.0.1:39003_ c:collection1 s:shard2 x:collection1 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1490084 T8792 n:127.0.0.1:39003_ c:collection1 s:shard2 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C1210 name=collection1 org.apache.solr.core.SolrCore@680e69fb url=http://127.0.0.1:39003/collection1 node=127.0.0.1:39003_ C1210_STATE=coll:collection1 core:collection1 props:{core=collection1, base_url=http://127.0.0.1:39003, node_name=127.0.0.1:39003_, state=down}
   [junit4]   2> 1490084 T8792 n:127.0.0.1:39003_ c:collection1 s:shard2 x:collection1 C1210 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:39003/collection1/
   [junit4]   2> 1490084 T8757 n:127.0.0.1:40748_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1"} current state version: 4
   [junit4]   2> 1490085 T8792 n:127.0.0.1:39003_ c:collection1 s:shard2 x:collection1 C1210 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1490085 T8792 n:127.0.0.1:39003_ c:collection1 s:shard2 x:collection1 C1210 oasc.SyncStrategy.syncToMe http://127.0.0.1:39003/collection1/ has no replicas
   [junit4]   2> 1490085 T8792 n:127.0.0.1:39003_ c:collection1 s:shard2 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:39003/collection1/ shard2
   [junit4]   2> 1490088 T8755 n:127.0.0.1:40748_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1490089 T8757 n:127.0.0.1:40748_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:39003",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "state":"active"} current state version: 4
   [junit4]   2> 1490240 T8792 n:127.0.0.1:39003_ c:collection1 s:shard2 x:collection1 oasc.ZkController.register We are http://127.0.0.1:39003/collection1/ and leader is http://127.0.0.1:39003/collection1/
   [junit4]   2> 1490240 T8792 n:127.0.0.1:39003_ c:collection1 s:shard2 x:collection1 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:39003
   [junit4]   2> 1490240 T8792 n:127.0.0.1:39003_ c:collection1 s:shard2 x:collection1 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1490240 T8792 n:127.0.0.1:39003_ c:collection1 s:shard2 x:collection1 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 1490241 T8792 n:127.0.0.1:39003_ c:collection1 s:shard2 x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1490241 T8755 n:127.0.0.1:40748_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1490242 T8757 n:127.0.0.1:40748_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:39003",
   [junit4]   2> 	  "node_name":"127.0.0.1:39003_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"} current state version: 5
   [junit4]   2> 1490243 T8757 n:127.0.0.1:40748_ oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:39003",
   [junit4]   2> 	  "node_name":"127.0.0.1:39003_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 1490432 T8713 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/shard-2-001/cores/collection1
   [junit4]   2> 1490433 T8713 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/shard-2-001
   [junit4]   2> 1490434 T8713 oejs.Server.doStart jetty-9.2.10.v20150310
   [junit4]   2> 1490436 T8713 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@5f1593e7{/,null,AVAILABLE}
   [junit4]   2> 1490437 T8713 oejs.AbstractConnector.doStart Started ServerConnector@4a36cc46{HTTP/1.1}{127.0.0.1:33817}
   [junit4]   2> 1490437 T8713 oejs.Server.doStart Started @1492303ms
   [junit4]   2> 1490438 T8713 oascse.JettySolrRunner$1.lifeCycleStarted Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/tempDir-002/jetty2, solrconfig=solrconfig.xml, hostContext=/, hostPort=33817, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/shard-2-001/cores}
   [junit4]   2> 1490438 T8713 oascsi.Krb5HttpClientConfigurer.configure Setting up SPNego auth with config: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/tempDir-001/minikdc/jaas-client.conf
   [junit4]   2> 1490438 T8713 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@c387f44
   [junit4]   2> 1490439 T8713 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/shard-2-001/'
   [junit4]   2> 1490454 T8713 oasc.SolrXmlConfig.fromFile Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/shard-2-001/solr.xml
   [junit4]   2> 1490475 T8713 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/shard-2-001/cores
   [junit4]   2> 1490476 T8713 oasc.CoreContainer.<init> New CoreContainer 6697740
   [junit4]   2> 1490477 T8713 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/shard-2-001/]
   [junit4]   2> 1490477 T8713 oasc.CoreContainer.load loading shared library: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/shard-2-001/lib
   [junit4]   2> 1490479 T8713 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/shard-2-001/lib).
   [junit4]   2> 1490486 T8713 oashc.HttpShardHandlerFactory.init created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 1490487 T8713 oascsi.Krb5HttpClientConfigurer.configure Setting up SPNego auth with config: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/tempDir-001/minikdc/jaas-client.conf
   [junit4]   2> 1490488 T8713 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1490488 T8713 oascsi.Krb5HttpClientConfigurer.configure Setting up SPNego auth with config: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/tempDir-001/minikdc/jaas-client.conf
   [junit4]   2> 1490489 T8713 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1490489 T8713 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1490489 T8713 oasc.CoreContainer.load Node Name: 127.0.0.1
   [junit4]   2> 1490489 T8713 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:40569/solr
   [junit4]   2> 1490490 T8713 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1490495 T8806 oascc.ConnectionManager.process WARN zkClient received AuthFailed
   [junit4]   2> 1490508 T8810 n:127.0.0.1:33817_ oascc.ConnectionManager.process WARN zkClient received AuthFailed
   [junit4]   2> 1491530 T8713 n:127.0.0.1:33817_ oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33817_
   [junit4]   2> 1491534 T8713 n:127.0.0.1:33817_ oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1491535 T8713 n:127.0.0.1:33817_ oasc.CoreContainer.initializeAuthenticationPlugin Authentication plugin class obtained from ZK: org.apache.solr.security.KerberosPlugin
   [junit4]   2> 1491536 T8713 n:127.0.0.1:33817_ oass.KerberosPlugin.init Params: {token.valid=30, kerberos.principal=HTTP/127.0.0.1, cookie.domain=127.0.0.1, kerberos.keytab=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/tempDir-001/minikdc/keytabs, cookie.path=/, type=kerberos, kerberos.name.rules=DEFAULT}
   [junit4]   2> 1491538 T8713 n:127.0.0.1:33817_ oasc.CoreContainer.initializeAuthenticationPlugin Reconfiguring the shard handler factory and update shard handler.
   [junit4]   2> 1491538 T8713 n:127.0.0.1:33817_ oashc.HttpShardHandlerFactory.reconfigureHttpClient Reconfiguring the default client with: org.apache.solr.client.solrj.impl.Krb5HttpClientConfigurer@3c3729d7
   [junit4]   2> 1491539 T8713 n:127.0.0.1:33817_ oascsi.Krb5HttpClientConfigurer.configure Setting up SPNego auth with config: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/tempDir-001/minikdc/jaas-client.conf
   [junit4]   2> 1491539 T8713 n:127.0.0.1:33817_ oasu.UpdateShardHandler.reconfigureHttpClient Reconfiguring the default client with: org.apache.solr.client.solrj.impl.Krb5HttpClientConfigurer@3c3729d7
   [junit4]   2> 1491539 T8713 n:127.0.0.1:33817_ oascsi.Krb5HttpClientConfigurer.configure Setting up SPNego auth with config: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/tempDir-001/minikdc/jaas-client.conf
   [junit4]   2> 1491540 T8713 n:127.0.0.1:33817_ oasc.CorePropertiesLocator.discover Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/shard-2-001/cores
   [junit4]   2> 1491541 T8713 n:127.0.0.1:33817_ oasc.CoreDescriptor.<init> CORE DESCRIPTOR: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/shard-2-001/cores/collection1, collection=collection1, absoluteInstDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/shard-2-001/cores/collection1/, coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 1491542 T8713 n:127.0.0.1:33817_ oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/shard-2-001/cores/collection1/
   [junit4]   2> 1491542 T8713 n:127.0.0.1:33817_ oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1491543 T8813 n:127.0.0.1:33817_ c:collection1 x:collection1 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 1491543 T8813 n:127.0.0.1:33817_ c:collection1 x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1491544 T8755 n:127.0.0.1:40748_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1491544 T8813 n:127.0.0.1:33817_ oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1491545 T8757 n:127.0.0.1:40748_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:33817",
   [junit4]   2> 	  "node_name":"127.0.0.1:33817_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"} current state version: 6
   [junit4]   2> 1491545 T8757 n:127.0.0.1:40748_ oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:33817",
   [junit4]   2> 	  "node_name":"127.0.0.1:33817_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 1491545 T8757 n:127.0.0.1:40748_ oasco.ReplicaMutator.updateState Collection already exists with numShards=2
   [junit4]   2> 1491546 T8757 n:127.0.0.1:40748_ oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1492544 T8813 n:127.0.0.1:33817_ oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1492545 T8813 n:127.0.0.1:33817_ oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 1492545 T8813 n:127.0.0.1:33817_ oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1492546 T8813 n:127.0.0.1:33817_ oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/shard-2-001/cores/collection1/'
   [junit4]   2> 1492558 T8813 n:127.0.0.1:33817_ oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 1492562 T8813 n:127.0.0.1:33817_ oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 1492566 T8813 n:127.0.0.1:33817_ oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 1492575 T8813 n:127.0.0.1:33817_ oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1492576 T8813 n:127.0.0.1:33817_ oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 1492581 T8813 n:127.0.0.1:33817_ oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1492685 T8813 n:127.0.0.1:33817_ oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1492690 T8813 n:127.0.0.1:33817_ oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1492692 T8813 n:127.0.0.1:33817_ oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1492700 T8813 n:127.0.0.1:33817_ oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1492702 T8813 n:127.0.0.1:33817_ oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1492703 T8813 n:127.0.0.1:33817_ oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1492704 T8813 n:127.0.0.1:33817_ oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1492704 T8813 n:127.0.0.1:33817_ oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1492705 T8813 n:127.0.0.1:33817_ oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1492705 T8813 n:127.0.0.1:33817_ oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1492706 T8813 n:127.0.0.1:33817_ oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1492706 T8813 n:127.0.0.1:33817_ oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 1492706 T8813 n:127.0.0.1:33817_ x:collection1 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1492706 T8813 n:127.0.0.1:33817_ x:collection1 oasc.SolrCore.<init> [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/shard-2-001/cores/collection1/], dataDir=[null]
   [junit4]   2> 1492707 T8813 n:127.0.0.1:33817_ x:collection1 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@17cfcf30
   [junit4]   2> 1492707 T8813 n:127.0.0.1:33817_ x:collection1 oasc.CachingDirectoryFactory.get return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/shard-2-001/cores/collection1/data
   [junit4]   2> 1492707 T8813 n:127.0.0.1:33817_ x:collection1 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/shard-2-001/cores/collection1/data/index/
   [junit4]   2> 1492707 T8813 n:127.0.0.1:33817_ x:collection1 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/shard-2-001/cores/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1492708 T8813 n:127.0.0.1:33817_ x:collection1 oasc.CachingDirectoryFactory.get return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001/shard-2-001/cores/collection1/data/index
   [junit4]   2> 1492708 T8813 n:127.0.0.1:33817_ x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=33, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3006284492664655]
   [junit4]   2> 1492708 T8813 n:127.0.0.1:33817_ x:collection1 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@d394c40 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@abd78ba),segFN=segments_1,generation=1}
   [junit4]   2> 1492709 T8813 n:127.0.0.1:33817_ x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1492712 T8813 n:127.0.0.1:33817_ x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1492713 T8813 n:127.0.0.1:33817_ x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1492713 T8813 n:127.0.0.1:33817_ x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1492713 T8813 n:127.0.0.1:33817_ x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1492713 T8813 n:127.0.0.1:33817_ x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1492713 T8813 n:127.0.0.1:33817_ x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1492714 T8813 n:127.0.0.1:33817_ x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1492714 T8813 n:127.0.0.1:33817_ x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit

[...truncated too long message...]

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 1509722 T8713 oejs.AbstractConnector.doStop Stopped ServerConnector@4dab548e{HTTP/1.1}{127.0.0.1:0}
   [junit4]   2> 1509722 T8713 oejsh.ContextHandler.doStop Stopped o.e.j.s.ServletContextHandler@61c3db9c{/,null,UNAVAILABLE}
   [junit4]   2> 1509725 T8713 c:control_collection s:shard1 x:collection1 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:40569 40569
   [junit4]   2> 1509998 T8722 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:40569 40569
   [junit4]   2> 1509999 T8722 oasc.ZkTestServer$ZKServerMain.runFromConfig WARN Watch limit violations: 
   [junit4]   2> 	Maximum concurrent create/delete watches above limit:
   [junit4]   2> 	
   [junit4]   2> 		7	/solr/aliases.json
   [junit4]   2> 		7	/solr/clusterstate.json
   [junit4]   2> 		5	/solr/configs/conf1
   [junit4]   2> 	
   [junit4]   2> 	Maximum concurrent children watches above limit:
   [junit4]   2> 	
   [junit4]   2> 		7	/solr/live_nodes
   [junit4]   2> 		5	/solr/overseer/collection-queue-work
   [junit4]   2> 		4	/solr/overseer/queue
   [junit4]   2> 	
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestSolrCloudWithKerberos -Dtests.method=testKerberizedSolr -Dtests.seed=7F80B71B49A3C81 -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=es_BO -Dtests.timezone=Canada/Newfoundland -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   31.8s J2 | TestSolrCloudWithKerberos.testKerberizedSolr <<<
   [junit4]    > Throwable #1: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:39003, http://127.0.0.1:51276, http://127.0.0.1:33817, http://127.0.0.1:40748, http://127.0.0.1:39669]
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([7F80B71B49A3C81:ACE1DDB6FEFBFCB6]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:355)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1086)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:856)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:799)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:135)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:152)
   [junit4]    > 	at org.apache.solr.cloud.TestSolrCloudWithKerberos.testKerberizedSolr(TestSolrCloudWithKerberos.java:159)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:960)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:935)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]    > Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:39669: Expected mime type application/octet-stream but got text/html. <html>
   [junit4]    > <head>
   [junit4]    > <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
   [junit4]    > <title>Error 500 </title>
   [junit4]    > </head>
   [junit4]    > <body>
   [junit4]    > <h2>HTTP ERROR: 500</h2>
   [junit4]    > <p>Problem accessing /admin/collections. Reason:
   [junit4]    > <pre>    org.apache.solr.common.SolrException: Error during request authentication, </pre></p>
   [junit4]    > <hr /><i><small>Powered by Jetty://</small></i>
   [junit4]    > </body>
   [junit4]    > </html>
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:529)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:235)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:227)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:376)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:328)
   [junit4]    > 	... 47 more
   [junit4]   2> 1510012 T8713 c:control_collection s:shard1 x:collection1 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 7F80B71B49A3C81-001
   [junit4]   2> 31774 T8712 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50), sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=es_BO, timezone=Canada/Newfoundland
   [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 1.8.0_25 (64-bit)/cpus=4,threads=1,free=171080232,total=513802240
   [junit4]   2> NOTE: All tests run in this JVM: [QueryParsingTest, SuggestComponentTest, OutputWriterTest, TestBinaryResponseWriter, HdfsUnloadDistributedZkTest, OutOfBoxZkACLAndCredentialsProvidersTest, DeleteShardTest, MultiThreadedOCPTest, TestStandardQParsers, SoftAutoCommitTest, XmlUpdateRequestHandlerTest, BasicDistributedZk2Test, DirectUpdateHandlerTest, SolrInfoMBeanTest, LoggingHandlerTest, TestManagedSchemaFieldResource, TestFieldCollectionResource, TestFastOutputStream, TestFieldTypeResource, RegexBoostProcessorTest, HdfsSyncSliceTest, TestPseudoReturnFields, DistributedQueryElevationComponentTest, TestSolrJ, TestSort, CircularListTest, SolrIndexSplitterTest, StatsComponentTest, TestSolrXml, TestSolrQueryParserDefaultOperatorResource, TestStressRecovery, TestQuerySenderNoQuery, DocValuesTest, TestSolr4Spatial2, SolrTestCaseJ4Test, FacetPivotSmallTest, SearchHandlerTest, TestManagedStopFilterFactory, TestShortCircuitedRequests, FileBasedSpellCheckerTest, TestMissingGroups, AsyncMigrateRouteKeyTest, DistribJoinFromCollectionTest, AutoCommitTest, CachingDirectoryFactoryTest, StressHdfsTest, QueryEqualityTest, TestSolrConfigHandler, RecoveryZkTest, MergeStrategyTest, TestRecovery, SolrCoreCheckLockOnStartupTest, TestDynamicFieldCollectionResource, TestExactSharedStatsCache, XsltUpdateRequestHandlerTest, TestMiniSolrCloudClusterSSL, TestCloudInspectUtil, AliasIntegrationTest, TestPartialUpdateDeduplication, TestCopyFieldCollectionResource, ResponseLogComponentTest, PluginInfoTest, DistributedIntervalFacetingTest, CustomCollectionTest, TestSearcherReuse, OverriddenZkACLAndCredentialsProvidersTest, DistanceUnitsTest, TestSolrQueryParser, VMParamsZkACLAndCredentialsProvidersTest, TestGroupingSearch, TestPerFieldSimilarity, HdfsRecoverLeaseTest, TestLeaderElectionZkExpiry, TestSystemIdResolver, TestMiniSolrCloudCluster, PolyFieldTest, SOLR749Test, TestZkChroot, CollectionsAPIDistributedZkTest, OverseerTest, ClusterStateUpdateTest, LeaderElectionTest, ShardRoutingCustomTest, TestFaceting, TestHashPartitioner, TestStressReorder, TestReload, TestStressVersions, SpellCheckComponentTest, BadIndexSchemaTest, TestBadConfig, CurrencyFieldXmlFileTest, AnalysisAfterCoreReloadTest, SolrRequestParserTest, TestTrie, StatelessScriptUpdateProcessorFactoryTest, DisMaxRequestHandlerTest, StandardRequestHandlerTest, TestQueryTypes, DocumentBuilderTest, TestCollationField, ReturnFieldsTest, MBeansHandlerTest, BinaryUpdateRequestHandlerTest, TestQuerySenderListener, TestSolrIndexConfig, TestMergePolicyConfig, SampleTest, TestDocSet, NumericFieldsTest, DirectSolrConnectionTest, NotRequiredUniqueKeyTest, SpellPossibilityIteratorTest, TestXIncludeConfig, TestSweetSpotSimilarityFactory, TestFastWriter, ScriptEngineTest, PrimUtilsTest, SpellingQueryConverterTest, DOMUtilTest, RAMDirectoryFactoryTest, ClusterStateTest, TestUtils, ZkNodePropsTest, TestRTGBase, TestCursorMarkWithoutUniqueKey, TestDistributedMissingSort, TestSimpleTrackingShardHandler, BaseCdcrDistributedZkTest, CdcrVersionReplicationTest, DistribCursorPagingTest, DistributedVersionInfoTest, OverseerStatusTest, ReplicationFactorTest, RollingRestartTest, SimpleCollectionCreateDeleteTest, SolrCloudExampleTest, TestRebalanceLeaders, TestSolrCloudWithKerberos]
   [junit4] Completed [433/502] on J2 in 32.55s, 1 test, 1 error <<< FAILURES!

[...truncated 222 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/build.xml:526: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/build.xml:474: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/build.xml:61: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/extra-targets.xml:39: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build.xml:229: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/common-build.xml:512: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/lucene/common-build.xml:1415: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/lucene/common-build.xml:973: There were test failures: 502 suites, 1984 tests, 1 error, 35 ignored (25 assumptions)

Total time: 50 minutes 23 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-Tests-trunk-Java8 - Build # 9 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-trunk-Java8/9/

1 tests failed.
FAILED:  org.apache.solr.cloud.TestSolrCloudWithKerberos.testKerberizedSolr

Error Message:
No live SolrServers available to handle this request:[http://127.0.0.1:43820, http://127.0.0.1:45701, http://127.0.0.1:45570, http://127.0.0.1:45919, http://127.0.0.1:41620]

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:43820, http://127.0.0.1:45701, http://127.0.0.1:45570, http://127.0.0.1:45919, http://127.0.0.1:41620]
	at __randomizedtesting.SeedInfo.seed([9840BAED8B010DDA:33596C2AC160CDED]:0)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:355)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1086)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:856)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:799)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:135)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:152)
	at org.apache.solr.cloud.TestSolrCloudWithKerberos.testKerberizedSolr(TestSolrCloudWithKerberos.java:159)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:960)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:935)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:41620: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 500 </title>
</head>
<body>
<h2>HTTP ERROR: 500</h2>
<p>Problem accessing /admin/collections. Reason:
<pre>    org.apache.solr.common.SolrException: Error during request authentication, </pre></p>
<hr /><i><small>Powered by Jetty://</small></i>
</body>
</html>

	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:529)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:235)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:227)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:376)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:328)
	... 47 more




Build Log:
[...truncated 10484 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestSolrCloudWithKerberos
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 9840BAED8B010DDA-001/init-core-data-001
   [junit4]   2> 1424189 T8006 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 1427307 T8006 oadsc.DefaultDirectoryService.showSecurityWarnings WARN You didn't change the admin password of directory service instance 'DefaultKrbServer'.  Please update the admin password as soon as possible to prevent a possible security breach.
   [junit4]   2> 1427585 T8006 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 1427586 T8014 oasc.ZkTestServer$2$1.setClientPort client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1427586 T8014 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1427686 T8006 oasc.ZkTestServer.run start zk server on port:57183
   [junit4]   2> 1427687 T8006 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1427688 T8006 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1427688 T8019 oaz.ClientCnxn$SendThread.startConnect WARN SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 9840BAED8B010DDA-001/tempDir-001/minikdc/jaas-client.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
   [junit4]   2> 1427689 T8021 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3e0b0c29 name:ZooKeeperConnection Watcher:127.0.0.1:57183 got event WatchedEvent state:AuthFailed type:None path:null path:null type:None
   [junit4]   2> 1427689 T8021 oascc.ConnectionManager.process WARN zkClient received AuthFailed
   [junit4]   2> 1427691 T8021 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3e0b0c29 name:ZooKeeperConnection Watcher:127.0.0.1:57183 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1427691 T8006 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1427691 T8006 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1427691 T8006 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1427700 T8006 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1427703 T8022 oaz.ClientCnxn$SendThread.startConnect WARN SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 9840BAED8B010DDA-001/tempDir-001/minikdc/jaas-client.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
   [junit4]   2> 1427707 T8006 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1427707 T8024 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@14207114 name:ZooKeeperConnection Watcher:127.0.0.1:57183/solr got event WatchedEvent state:AuthFailed type:None path:null path:null type:None
   [junit4]   2> 1427708 T8024 oascc.ConnectionManager.process WARN zkClient received AuthFailed
   [junit4]   2> 1427707 T8025 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@14207114 name:ZooKeeperConnection Watcher:127.0.0.1:57183/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1427708 T8006 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1427708 T8006 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1427709 T8006 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 1427710 T8006 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 1427712 T8006 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 1427713 T8006 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 1427715 T8006 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1427715 T8006 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1427717 T8006 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 1427718 T8006 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 1427719 T8006 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1427720 T8006 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1427721 T8006 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1427722 T8006 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1427723 T8006 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1427724 T8006 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1427725 T8006 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1427725 T8006 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 1427727 T8006 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1427727 T8006 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1427729 T8006 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1427729 T8006 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1427731 T8006 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1427732 T8006 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1427733 T8006 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1427733 T8006 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1427735 T8006 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1427735 T8006 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1427737 T8006 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1427739 T8026 oaz.ClientCnxn$SendThread.startConnect WARN SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 9840BAED8B010DDA-001/tempDir-001/minikdc/jaas-client.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
   [junit4]   2> 1427740 T8006 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1427743 T8028 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@55bce6c1 name:ZooKeeperConnection Watcher:127.0.0.1:57183/solr got event WatchedEvent state:AuthFailed type:None path:null path:null type:None
   [junit4]   2> 1427743 T8028 oascc.ConnectionManager.process WARN zkClient received AuthFailed
   [junit4]   2> 1427743 T8029 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@55bce6c1 name:ZooKeeperConnection Watcher:127.0.0.1:57183/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1427743 T8006 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1427744 T8006 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1428085 T8006 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 9840BAED8B010DDA-001/control-001/cores/collection1
   [junit4]   2> 1428087 T8006 oejs.Server.doStart jetty-9.2.10.v20150310
   [junit4]   2> 1428089 T8006 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@193b859c{/,null,AVAILABLE}
   [junit4]   2> 1428089 T8006 oejs.AbstractConnector.doStart Started ServerConnector@3d1ae0e4{HTTP/1.1}{127.0.0.1:41620}
   [junit4]   2> 1428090 T8006 oejs.Server.doStart Started @1430179ms
   [junit4]   2> 1428090 T8006 oascse.JettySolrRunner$1.lifeCycleStarted Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 9840BAED8B010DDA-001/tempDir-002/control/data, hostContext=/, hostPort=41620, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 9840BAED8B010DDA-001/control-001/cores}
   [junit4]   2> 1428090 T8006 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@c387f44
   [junit4]   2> 1428091 T8006 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 9840BAED8B010DDA-001/control-001/'
   [junit4]   2> 1428106 T8006 oasc.SolrXmlConfig.fromFile Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 9840BAED8B010DDA-001/control-001/solr.xml
   [junit4]   2> 1428113 T8006 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 9840BAED8B010DDA-001/control-001/cores
   [junit4]   2> 1428113 T8006 oasc.CoreContainer.<init> New CoreContainer 992021330
   [junit4]   2> 1428113 T8006 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 9840BAED8B010DDA-001/control-001/]
   [junit4]   2> 1428114 T8006 oasc.CoreContainer.load loading shared library: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 9840BAED8B010DDA-001/control-001/lib
   [junit4]   2> 1428114 T8006 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 9840BAED8B010DDA-001/control-001/lib).
   [junit4]   2> 1428121 T8006 oashc.HttpShardHandlerFactory.init created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 1428123 T8006 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1428123 T8006 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1428123 T8006 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1428124 T8006 oasc.CoreContainer.load Node Name: 127.0.0.1
   [junit4]   2> 1428124 T8006 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:57183/solr
   [junit4]   2> 1428124 T8006 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1428125 T8006 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1428125 T8006 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1428125 T8041 oaz.ClientCnxn$SendThread.startConnect WARN SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 9840BAED8B010DDA-001/tempDir-001/minikdc/jaas-client.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
   [junit4]   2> 1428126 T8043 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@32746ce8 name:ZooKeeperConnection Watcher:127.0.0.1:57183 got event WatchedEvent state:AuthFailed type:None path:null path:null type:None
   [junit4]   2> 1428127 T8044 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@32746ce8 name:ZooKeeperConnection Watcher:127.0.0.1:57183 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1428127 T8006 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1428128 T8006 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1428127 T8043 oascc.ConnectionManager.process WARN zkClient received AuthFailed
   [junit4]   2> 1428129 T8045 n:127.0.0.1:41620_ oaz.ClientCnxn$SendThread.startConnect WARN SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 9840BAED8B010DDA-001/tempDir-001/minikdc/jaas-client.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
   [junit4]   2> 1428129 T8006 n:127.0.0.1:41620_ oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1428130 T8047 n:127.0.0.1:41620_ oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6c41d756 name:ZooKeeperConnection Watcher:127.0.0.1:57183/solr got event WatchedEvent state:AuthFailed type:None path:null path:null type:None
   [junit4]   2> 1428130 T8047 n:127.0.0.1:41620_ oascc.ConnectionManager.process WARN zkClient received AuthFailed
   [junit4]   2> 1428130 T8048 n:127.0.0.1:41620_ oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6c41d756 name:ZooKeeperConnection Watcher:127.0.0.1:57183/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1428131 T8006 n:127.0.0.1:41620_ oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1428132 T8006 n:127.0.0.1:41620_ oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1428136 T8006 n:127.0.0.1:41620_ oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1428138 T8006 n:127.0.0.1:41620_ oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 1428140 T8006 n:127.0.0.1:41620_ oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 1428142 T8006 n:127.0.0.1:41620_ oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 1428144 T8006 n:127.0.0.1:41620_ oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1428145 T8006 n:127.0.0.1:41620_ oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1428147 T8006 n:127.0.0.1:41620_ oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1428148 T8006 n:127.0.0.1:41620_ oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41620_
   [junit4]   2> 1428148 T8006 n:127.0.0.1:41620_ oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:41620_
   [junit4]   2> 1428150 T8006 n:127.0.0.1:41620_ oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 1428151 T8006 n:127.0.0.1:41620_ oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1428152 T8006 n:127.0.0.1:41620_ oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1428154 T8006 n:127.0.0.1:41620_ oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:41620_
   [junit4]   2> 1428154 T8006 n:127.0.0.1:41620_ oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1428155 T8006 n:127.0.0.1:41620_ oasc.Overseer.start Overseer (id=93882404124884996-127.0.0.1:41620_-n_0000000000) starting
   [junit4]   2> 1428157 T8006 n:127.0.0.1:41620_ oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1428162 T8006 n:127.0.0.1:41620_ oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 1428163 T8006 n:127.0.0.1:41620_ oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1428163 T8050 n:127.0.0.1:41620_ oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1428163 T8049 n:127.0.0.1:41620_ oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1428166 T8006 n:127.0.0.1:41620_ oasc.CoreContainer.initializeAuthenticationPlugin Authentication plugin class obtained from ZK: org.apache.solr.security.KerberosPlugin
   [junit4]   2> 1428173 T8006 n:127.0.0.1:41620_ oass.KerberosPlugin.init Params: {token.valid=30, kerberos.principal=HTTP/127.0.0.1, cookie.domain=127.0.0.1, kerberos.keytab=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 9840BAED8B010DDA-001/tempDir-001/minikdc/keytabs, cookie.path=/, type=kerberos, kerberos.name.rules=DEFAULT}
   [junit4]   2> 1428185 T8006 n:127.0.0.1:41620_ oasc.CoreContainer.initializeAuthenticationPlugin Reconfiguring the shard handler factory and update shard handler.
   [junit4]   2> 1428185 T8006 n:127.0.0.1:41620_ oashc.HttpShardHandlerFactory.reconfigureHttpClient Reconfiguring the default client with: org.apache.solr.client.solrj.impl.Krb5HttpClientConfigurer@5ec6ebfa
   [junit4]   2> 1428185 T8006 n:127.0.0.1:41620_ oascsi.Krb5HttpClientConfigurer.configure Setting up SPNego auth with config: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 9840BAED8B010DDA-001/tempDir-001/minikdc/jaas-client.conf
   [junit4]   2> 1428188 T8006 n:127.0.0.1:41620_ oasu.UpdateShardHandler.reconfigureHttpClient Reconfiguring the default client with: org.apache.solr.client.solrj.impl.Krb5HttpClientConfigurer@5ec6ebfa
   [junit4]   2> 1428189 T8006 n:127.0.0.1:41620_ oascsi.Krb5HttpClientConfigurer.configure Setting up SPNego auth with config: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 9840BAED8B010DDA-001/tempDir-001/minikdc/jaas-client.conf
   [junit4]   2> 1428190 T8006 n:127.0.0.1:41620_ oasc.CorePropertiesLocator.discover Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 9840BAED8B010DDA-001/control-001/cores
   [junit4]   2> 1428191 T8006 n:127.0.0.1:41620_ oasc.CoreDescriptor.<init> CORE DESCRIPTOR: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 9840BAED8B010DDA-001/control-001/cores/collection1, collection=control_collection, absoluteInstDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 9840BAED8B010DDA-001/control-001/cores/collection1/, coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 1428191 T8006 n:127.0.0.1:41620_ oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 9840BAED8B010DDA-001/control-001/cores/collection1/
   [junit4]   2> 1428192 T8006 n:127.0.0.1:41620_ oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1428192 T8053 n:127.0.0.1:41620_ c:control_collection x:collection1 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 1428193 T8053 n:127.0.0.1:41620_ c:control_collection x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1428193 T8048 n:127.0.0.1:41620_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1428193 T8053 n:127.0.0.1:41620_ oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1428194 T8049 n:127.0.0.1:41620_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:41620",
   [junit4]   2> 	  "node_name":"127.0.0.1:41620_",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"} current state version: 0
   [junit4]   2> 1428195 T8049 n:127.0.0.1:41620_ oasco.ReplicaMutator.updateState Update state numShards=1 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:41620",
   [junit4]   2> 	  "node_name":"127.0.0.1:41620_",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 1428195 T8049 n:127.0.0.1:41620_ oasco.ClusterStateMutator.createCollection building a new cName: control_collection
   [junit4]   2> 1428195 T8049 n:127.0.0.1:41620_ oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1428196 T8048 n:127.0.0.1:41620_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1429194 T8053 n:127.0.0.1:41620_ oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1429194 T8053 n:127.0.0.1:41620_ oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 1429195 T8053 n:127.0.0.1:41620_ oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1429195 T8053 n:127.0.0.1:41620_ oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 1429196 T8053 n:127.0.0.1:41620_ oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1429196 T8053 n:127.0.0.1:41620_ oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 9840BAED8B010DDA-001/control-001/cores/collection1/'
   [junit4]   2> 1429208 T8053 n:127.0.0.1:41620_ oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 1429222 T8053 n:127.0.0.1:41620_ oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 1429227 T8053 n:127.0.0.1:41620_ oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 1429236 T8053 n:127.0.0.1:41620_ oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1429238 T8053 n:127.0.0.1:41620_ oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 1429242 T8053 n:127.0.0.1:41620_ oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1429364 T8053 n:127.0.0.1:41620_ oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1429371 T8053 n:127.0.0.1:41620_ oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1429372 T8053 n:127.0.0.1:41620_ oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1429379 T8053 n:127.0.0.1:41620_ oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1429381 T8053 n:127.0.0.1:41620_ oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1429383 T8053 n:127.0.0.1:41620_ oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1429384 T8053 n:127.0.0.1:41620_ oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1429394 T8053 n:127.0.0.1:41620_ oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1429395 T8053 n:127.0.0.1:41620_ oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1429396 T8053 n:127.0.0.1:41620_ oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1429396 T8053 n:127.0.0.1:41620_ oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1429396 T8053 n:127.0.0.1:41620_ oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 1429397 T8053 n:127.0.0.1:41620_ x:collection1 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1429397 T8053 n:127.0.0.1:41620_ x:collection1 oasc.SolrCore.<init> [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 9840BAED8B010DDA-001/control-001/cores/collection1/], dataDir=[null]
   [junit4]   2> 1429397 T8053 n:127.0.0.1:41620_ x:collection1 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@513effe
   [junit4]   2> 1429398 T8053 n:127.0.0.1:41620_ x:collection1 oasc.CachingDirectoryFactory.get return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 9840BAED8B010DDA-001/control-001/cores/collection1/data
   [junit4]   2> 1429398 T8053 n:127.0.0.1:41620_ x:collection1 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 9840BAED8B010DDA-001/control-001/cores/collection1/data/index/
   [junit4]   2> 1429398 T8053 n:127.0.0.1:41620_ x:collection1 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 9840BAED8B010DDA-001/control-001/cores/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1429398 T8053 n:127.0.0.1:41620_ x:collection1 oasc.CachingDirectoryFactory.get return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 9840BAED8B010DDA-001/control-001/cores/collection1/data/index
   [junit4]   2> 1429399 T8053 n:127.0.0.1:41620_ x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=10, maxMergeAtOnceExplicit=40, maxMergedSegmentMB=46.45703125, floorSegmentMB=0.359375, forceMergeDeletesPctAllowed=4.1970158864613, segmentsPerTier=34.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1429399 T8053 n:127.0.0.1:41620_ x:collection1 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@4c2fe812 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@36d9f691),segFN=segments_1,generation=1}
   [junit4]   2> 1429400 T8053 n:127.0.0.1:41620_ x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1429403 T8053 n:127.0.0.1:41620_ x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1429403 T8053 n:127.0.0.1:41620_ x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1429403 T8053 n:127.0.0.1:41620_ x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1429404 T8053 n:127.0.0.1:41620_ x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1429404 T8053 n:127.0.0.1:41620_ x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1429404 T8053 n:127.0.0.1:41620_ x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1429404 T8053 n:127.0.0.1:41620_ x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1429404 T8053 n:127.0.0.1:41620_ x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1429405 T8053 n:127.0.0.1:41620_ x:collection1 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1429407 T8053 n:127.0.0.1:41620_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1429408 T8053 n:127.0.0.1:41620_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1429409 T8053 n:127.0.0.1:41620_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1429410 T8053 n:127.0.0.1:41620_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1429412 T8053 n:127.0.0.1:41620_ x:collection1 oasc.RequestHandlers.initHandlersFromConfig Registered paths: /admin/mbeans,standard,/update/csv,/update/json/docs,/admin/luke,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/schema,/admin/plugins,/admin/logging,/update/json,/admin/threads,/admin/ping,/update,/admin/file
   [junit4]   2> 1429413 T8053 n:127.0.0.1:41620_ x:collection1 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 1429414 T8053 n:127.0.0.1:41620_ x:collection1 oasu.UpdateHandler.<init> Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1429414 T8053 n:127.0.0.1:41620_ x:collection1 oasu.UpdateLog.init Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1429415 T8053 n:127.0.0.1:41620_ x:collection1 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1429415 T8053 n:127.0.0.1:41620_ x:collection1 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1429416 T8053 n:127.0.0.1:41620_ x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=27, maxMergeAtOnceExplicit=46, maxMergedSegmentMB=61.61328125, floorSegmentMB=1.8525390625, forceMergeDeletesPctAllowed=10.319260276732745, segmentsPerTier=46.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1429416 T8053 n:127.0.0.1:41620_ x:collection1 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@4c2fe812 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@36d9f691),segFN=segments_1,generation=1}
   [junit4]   2> 1429417 T8053 n:127.0.0.1:41620_ x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1429417 T8053 n:127.0.0.1:41620_ x:collection1 oass.SolrIndexSearcher.<init> Opening Searcher@91b28d[collection1] main
   [junit4]   2> 1429417 T8053 n:127.0.0.1:41620_ x:collection1 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 1429418 T8053 n:127.0.0.1:41620_ x:collection1 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1429418 T8053 n:127.0.0.1:41620_ x:collection1 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1429419 T8053 n:127.0.0.1:41620_ x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1429419 T8053 n:127.0.0.1:41620_ x:collection1 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 1429419 T8053 n:127.0.0.1:41620_ x:collection1 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1429420 T8053 n:127.0.0.1:41620_ x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1429420 T8053 n:127.0.0.1:41620_ x:collection1 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1429420 T8053 n:127.0.0.1:41620_ x:collection1 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 1429420 T8053 n:127.0.0.1:41620_ x:collection1 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 1429421 T8053 n:127.0.0.1:41620_ x:collection1 oasc.ZkController.getConfDirListeners watch zkdir /configs/conf1
   [junit4]   2> 1429421 T8054 n:127.0.0.1:41620_ x:collection1 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@91b28d[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1429421 T8053 n:127.0.0.1:41620_ x:collection1 oasu.UpdateLog.onFirstSearcher On first searcher opened, looking up max value of version field
   [junit4]   2> 1429422 T8053 n:127.0.0.1:41620_ x:collection1 oasu.VersionInfo.getMaxVersionFromIndex Refreshing highest value of _version_ for 65536 version buckets from index
   [junit4]   2> 1429422 T8053 n:127.0.0.1:41620_ x:collection1 oasu.VersionInfo.getMaxVersionFromIndex No terms found for _version_, cannot seed version bucket highest value from index
   [junit4]   2> 1429422 T8053 n:127.0.0.1:41620_ x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion Could not find max version in index or recent updates, using new clock 1502118467918102528
   [junit4]   2> 1429424 T8053 n:127.0.0.1:41620_ x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion Took 2 ms to seed version buckets with highest version 1502118467918102528
   [junit4]   2> 1429424 T8053 n:127.0.0.1:41620_ x:collection1 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1429425 T8057 n:127.0.0.1:41620_ c:control_collection s:shard1 x:collection1 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:41620 collection:control_collection shard:shard1
   [junit4]   2> 1429426 T8006 n:127.0.0.1:41620_ oascsi.Krb5HttpClientConfigurer.configure Setting up SPNego auth with config: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 9840BAED8B010DDA-001/tempDir-001/minikdc/jaas-client.conf
   [junit4]   2> 1429426 T8006 n:127.0.0.1:41620_ oass.SolrDispatchFilter.init user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2
   [junit4]   2> 1429426 T8006 n:127.0.0.1:41620_ oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1429427 T8006 oascsi.Krb5HttpClientConfigurer.configure Setting up SPNego auth with config: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 9840BAED8B010DDA-001/tempDir-001/minikdc/jaas-client.conf
   [junit4]   2> 1429428 T8006 oascsi.Krb5HttpClientConfigurer.configure Setting up SPNego auth with config: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 9840BAED8B010DDA-001/tempDir-001/minikdc/jaas-client.conf
   [junit4]   2> 1429428 T8006 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1429429 T8058 oaz.ClientCnxn$SendThread.startConnect WARN SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 9840BAED8B010DDA-001/tempDir-001/minikdc/jaas-client.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
   [junit4]   2> 1429430 T8057 n:127.0.0.1:41620_ c:control_collection s:shard1 x:collection1 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 1429430 T8006 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1429433 T8060 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2f1e0b6 name:ZooKeeperConnection Watcher:127.0.0.1:57183/solr got event WatchedEvent state:AuthFailed type:None path:null path:null type:None
   [junit4]   2> 1429433 T8060 oascc.ConnectionManager.process WARN zkClient received AuthFailed
   [junit4]   2> 1429434 T8057 n:127.0.0.1:41620_ c:control_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 1429435 T8048 n:127.0.0.1:41620_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1429436 T8049 n:127.0.0.1:41620_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection"} current state version: 1
   [junit4]   2> 1429436 T8061 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2f1e0b6 name:ZooKeeperConnection Watcher:127.0.0.1:57183/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1429437 T8006 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1429438 T8006 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1429438 T8006 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1429438 T8057 n:127.0.0.1:41620_ c:control_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1429439 T8057 n:127.0.0.1:41620_ c:control_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C1286 name=collection1 org.apache.solr.core.SolrCore@371c3455 url=http://127.0.0.1:41620/collection1 node=127.0.0.1:41620_ C1286_STATE=coll:control_collection core:collection1 props:{core=collection1, base_url=http://127.0.0.1:41620, node_name=127.0.0.1:41620_, state=down}
   [junit4]   2> 1429439 T8057 n:127.0.0.1:41620_ c:control_collection s:shard1 x:collection1 C1286 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:41620/collection1/
   [junit4]   2> 1429440 T8006 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1429440 T8006 oasc.AbstractFullDistribZkTestBase.createJettys Creating collection1 with stateFormat=2
   [junit4]   2> 1429441 T8006 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1429441 T8057 n:127.0.0.1:41620_ c:control_collection s:shard1 x:collection1 C1286 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1429441 T8062 oaz.ClientCnxn$SendThread.startConnect WARN SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 9840BAED8B010DDA-001/tempDir-001/minikdc/jaas-client.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
   [junit4]   2> 1429441 T8006 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1429441 T8057 n:127.0.0.1:41620_ c:control_collection s:shard1 x:collection1 C1286 oasc.SyncStrategy.syncToMe http://127.0.0.1:41620/collection1/ has no replicas
   [junit4]   2> 1429442 T8057 n:127.0.0.1:41620_ c:control_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:41620/collection1/ shard1
   [junit4]   2> 1429443 T8057 n:127.0.0.1:41620_ c:control_collection s:shard1 x:collection1 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 1429445 T8048 n:127.0.0.1:41620_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1429446 T8049 n:127.0.0.1:41620_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:41620",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "state":"active"} current state version: 1
   [junit4]   2> 1429448 T8065 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@29f79975 name:ZooKeeperConnection Watcher:127.0.0.1:57183/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1429448 T8006 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1429448 T8006 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1429450 T8048 n:127.0.0.1:41620_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1429450 T8049 n:127.0.0.1:41620_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"create",
   [junit4]   2> 	  "name":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "stateFormat":"2"} current state version: 1
   [junit4]   2> 1429450 T8049 n:127.0.0.1:41620_ oasco.ClusterStateMutator.createCollection building a new cName: collection1
   [junit4]   2> 1429451 T8061 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1429451 T8048 n:127.0.0.1:41620_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1429452 T8064 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@29f79975 name:ZooKeeperConnection Watcher:127.0.0.1:57183/solr got event WatchedEvent state:AuthFailed type:None path:null path:null type:None
   [junit4]   2> 1429452 T8064 oascc.ConnectionManager.process Client->ZooKeeper status change trigger but we are already closed
   [junit4]   2> 1429498 T8057 n:127.0.0.1:41620_ c:control_collection s:shard1 x:collection1 oasc.ZkController.register We are http://127.0.0.1:41620/collection1/ and leader is http://127.0.0.1:41620/collection1/
   [junit4]   2> 1429498 T8057 n:127.0.0.1:41620_ c:control_collection s:shard1 x:collection1 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:41620
   [junit4]   2> 1429498 T8057 n:127.0.0.1:41620_ c:control_collection s:shard1 x:collection1 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1429499 T8057 n:127.0.0.1:41620_ c:control_collection s:shard1 x:collection1 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 1429499 T8057 n:127.0.0.1:41620_ c:control_collection s:shard1 x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1429500 T8048 n:127.0.0.1:41620_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1429501 T8049 n:127.0.0.1:41620_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:41620",
   [junit4]   2> 	  "node_name":"127.0.0.1:41620_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"} current state version: 2
   [junit4]   2> 1429501 T8049 n:127.0.0.1:41620_ oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:41620",
   [junit4]   2> 	  "node_name":"127.0.0.1:41620_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 1429502 T8049 n:127.0.0.1:41620_ oasco.ZkStateWriter.writePendingUpdates going to create_collection /collections/collection1/state.json
   [junit4]   2> 1429503 T8061 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1429504 T8048 n:127.0.0.1:41620_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1429608 T8061 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1429608 T8048 n:127.0.0.1:41620_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1429801 T8006 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 9840BAED8B010DDA-001/shard-1-001/cores/collection1
   [junit4]   2> 1429802 T8006 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 9840BAED8B010DDA-001/shard-1-001
   [junit4]   2> 1429803 T8006 oejs.Server.doStart jetty-9.2.10.v20150310
   [junit4]   2> 1429805 T8006 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@6c58ada5{/,null,AVAILABLE}
   [junit4]   2> 1429805 T8006 oejs.AbstractConnector.doStart Started ServerConnector@29768aa3{HTTP/1.1}{127.0.0.1:43820}
   [junit4]   2> 1429805 T8006 oejs.Server.doStart Started @1431895ms
   [junit4]   2> 1429806 T8006 oascse.JettySolrRunner$1.lifeCycleStarted Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 9840BAED8B010DDA-001/tempDir-002/jetty1, solrconfig=solrconfig.xml, hostContext=/, hostPort=43820, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 9840BAED8B010DDA-001/shard-1-001/cores}
   [junit4]   2> 1429806 T8006 oascsi.Krb5HttpClientConfigurer.configure Setting up SPNego auth with config: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 9840BAED8B010DDA-001/tempDir-001/minikdc/jaas-client.conf
   [junit4]   2> 1429807 T8006 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@c387f44
   [junit4]   2> 1429807 T8006 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 9840BAED8B010DDA-001/shard-1-001/'
   [junit4]   2> 1429821 T8006 oasc.SolrXmlConfig.fromFile Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 9840BAED8B010DDA-001/shard-1-001/solr.xml
   [junit4]   2> 1429827 T8006 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 9840BAED8B010DDA-001/shard-1-001/cores
   [junit4]   2> 1429827 T8006 oasc.CoreContainer.<init> New CoreContainer 1300125204
   [junit4]   2> 1429828 T8006 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 9840BAED8B010DDA-001/shard-1-001/]
   [junit4]   2> 1429828 T8006 oasc.CoreContainer.load loading shared library: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 9840BAED8B010DDA-001/shard-1-001/lib
   [junit4]   2> 1429828 T8006 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 9840BAED8B010DDA-001/shard-1-001/lib).
   [junit4]   2> 1429835 T8006 oashc.HttpShardHandlerFactory.init created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 1429837 T8006 oascsi.Krb5HttpClientConfigurer.configure Setting up SPNego auth with config: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 9840BAED8B010DDA-001/tempDir-001/minikdc/jaas-client.conf
   [junit4]   2> 1429837 T8006 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1429838 T8006 oascsi.Krb5HttpClientConfigurer.configure Setting up SPNego auth with config: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 9840BAED8B010DDA-001/tempDir-001/minikdc/jaas-client.conf
   [junit4]   2> 1429838 T8006 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1429838 T8006 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1429839 T8006 oasc.CoreContainer.load Node Name: 127.0.0.1
   [junit4]   2> 1429839 T8006 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:57183/solr
   [junit4]   2> 1429839 T8006 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1429840 T8006 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1429840 T8006 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1429840 T8077 oaz.ClientCnxn$SendThread.startConnect WARN SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 9840BAED8B010DDA-001/tempDir-001/minikdc/jaas-client.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
   [junit4]   2> 1429841 T8079 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7eb5a740 name:ZooKeeperConnection Watcher:127.0.0.1:57183 got event WatchedEvent state:AuthFailed type:None path:null path:null type:None
   [junit4]   2> 1429841 T8079 oascc.ConnectionManager.process WARN zkClient received AuthFailed
   [junit4]   2> 1429842 T8079 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7eb5a740 name:ZooKeeperConnection Watcher:127.0.0.1:57183 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1429842 T8006 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1429842 T8006 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1429844 T8006 n:127.0.0.1:43820_ oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1429844 T8080 n:127.0.0.1:43820_ oaz.ClientCnxn$SendThread.startConnect WARN SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 9840BAED8B010DDA-001/tempDir-001/minikdc/jaas-client.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
   [junit4]   2> 1429847 T8082 n:127.0.0.1:43820_ oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2b6e04ab name:ZooKeeperConnection Watcher:127.0.0.1:57183/solr got event WatchedEvent state:AuthFailed type:None path:null path:null type:None
   [junit4]   2> 1429847 T8082 n:127.0.0.1:43820_ oascc.ConnectionManager.process WARN zkClient received AuthFailed
   [junit4]   2> 1429847 T8083 n:127.0.0.1:43820_ oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2b6e04ab name:ZooKeeperConnection Watcher:127.0.0.1:57183/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1429847 T8006 n:127.0.0.1:43820_ oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1429851 T8006 n:127.0.0.1:43820_ oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1430856 T8006 n:127.0.0.1:43820_ oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43820_
   [junit4]   2> 1430857 T8006 n:127.0.0.1:43820_ oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:43820_
   [junit4]   2> 1430860 T8006 n:127.0.0.1:43820_ oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1430862 T8006 n:127.0.0.1:43820_ oasc.CoreContainer.initializeAuthenticationPlugin Authentication plugin class obtained from ZK: org.apache.solr.security.KerberosPlugin
   [junit4]   2> 1430862 T8006 n:127.0.0.1:43820_ oass.KerberosPlugin.init Params: {token.valid=30, kerberos.principal=HTTP/127.0.0.1, cookie.domain=127.0.0.1, kerberos.keytab=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 9840BAED8B010DDA-001/tempDir-001/minikdc/keytabs, cookie.path=/, type=kerberos, kerberos.name.rules=DEFAULT}
   [junit4]   2> 1430864 T8006 n:127.0.0.1:43820_ oasc.CoreContainer.initializeAuthenticationPlugin Reconfiguring the shard handler factory and update shard handler.
   [junit4]   2> 1430865 T8006 n:127.0.0.1:43820_ oashc.HttpShardHandlerFactory.reconfigureHttpClient Reconfiguring the default client with: org.apache.solr.client.solrj.impl.Krb5HttpClientConfigurer@370cbbe
   [junit4]   2> 1430865 T8006 n:127.0.0.1:43820_ oascsi.Krb5HttpClientConfigurer.configure Setting up SPNego auth with config: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 9840BAED8B010DDA-001/tempDir-001/minikdc/jaas-client.conf
   [junit4]   2> 1430865 T8006 n:127.0.0.1:43820_ oasu.UpdateShardHandler.reconfigureHttpClient Reconfiguring the default client with: org.apache.solr.client.solrj.impl.Krb5HttpClientConfigurer@370cbbe
   [junit4]   2> 1430866 T8006 n:127.0.0.1:43820_ oascsi.Krb5HttpClientConfigurer.configure Setting up SPNego auth with config: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 9840BAED8B010DDA-001/tempDir-001/minikdc/jaas-client.conf
   [junit4]   2> 1430867 T8006 n:127.0.0.1:43820_ oasc.CorePropertiesLocator.discover Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 9840BAED8B010DDA-001/shard-1-001/cores
   [junit4]   2> 1430868 T8006 n:127.0.0.1:43820_ oasc.CoreDescriptor.<init> CORE DESCRIPTOR: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 9840BAED8B010DDA-001/shard-1-001/cores/collection1, collection=collection1, absoluteInstDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 9840BAED8B010DDA-001/shard-1-001/cores/collection1/, coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 1430868 T8006 n:127.0.0.1:43820_ oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 9840BAED8B010DDA-001/shard-1-001/cores/collection1/
   [junit4]   2> 1430868 T8006 n:127.0.0.1:43820_ oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1430871 T8085 n:127.0.0.1:43820_ c:collection1 x:collection1 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 1430871 T8085 n:127.0.0.1:43820_ c:collection1 x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1430872 T8048 n:127.0.0.1:41620_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1430873 T8085 n:127.0.0.1:43820_ oasc.ZkController.preRegister Registering watch for external collection collection1
   [junit4]   2> 1430873 T8085 n:127.0.0.1:43820_ oascc.ZkStateReader.addZkWatch addZkWatch collection1
   [junit4]   2> 1430873 T8049 n:127.0.0.1:41620_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:43820",
   [junit4]   2> 	  "node_name":"127.0.0.1:43820_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"} current state version: 4
   [junit4]   2> 1430874 T8085 n:127.0.0.1:43820_ oascc.ZkStateReader.addZkWatch Updating collection state at /collections/collection1/state.json from ZooKeeper... 
   [junit4]   2> 1430874 T8049 n:127.0.0.1:41620_ oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:43820",
   [junit4]   2> 	  "node_name":"127.0.0.1:43820_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 1430874 T8049 n:127.0.0.1:41620_ oasco.ReplicaMutator.updateState Collection already exists with numShards=2
   [junit4]   2> 1430875 T8049 n:127.0.0.1:41620_ oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 1430875 T8085 n:127.0.0.1:43820_ oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 0 
   [junit4]   2> 1430875 T8085 n:127.0.0.1:43820_ oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1430977 T8049 n:127.0.0.1:41620_ oasco.ZkStateWriter.writePendingUpdates going to update_collection /collections/collection1/state.json version: 0
   [junit4]   2> 1430978 T8083 n:127.0.0.1:43820_ oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json for collection collection1 has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1430979 T8083 n:127.0.0.1:43820_ oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 1 
   [junit4]   2> 1431876 T8085 n:127.0.0.1:43820_ oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1431876 T8085 n:127.0.0.1:43820_ oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 1431877 T8085 n:127.0.0.1:43820_ oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1431877 T8085 n:127.0.0.1:43820_ oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1431878 T8085 n:127.0.0.1:43820_ oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1431878 T8085 n:127.0.0.1:43820_ oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 9840BAED8B010DDA-001/shard-1-001/cores/collection1/'
   [junit4]   2> 1431890 T8085 n:127.0.0.1:43820_ oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 1431895 T8085 n:127.0.0.1:43820_ oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 1431899 T8085 n:127.0.0.1:43820_ oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 1431920 T8085 n:127.0.0.1:43820_ oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1431921 T8085 n:127.0.0.1:43820_ oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 1431925 T8085 n:127.0.0.1:43820_ oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1432023 T8085 n:127.0.0.1:43820_ oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1432029 T8085 n:127.0.0.1:43820_ oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1432030 T8085 n:127.0.0.1:43820_ oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1432038 T8085 n:127.0.0.1:43820_ oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1432041 T8085 n:127.0.0.1:43820_ oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1432043 T8085 n:127.0.0.1:43820_ oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1432043 T8085 n:127.0.0.1:43820_ oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1432044 T8085 n:127.0.0.1:43820_ oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1432044 T8085 n:127.0.0.1:43820_ oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1432045 T8085 n:127.0.0.1:43820_ oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1432045 T8085 n:127.0.0.1:43820_ oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1432045 T8085 n:127.0.0.1:43820_ oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 1432045 T8085 n:127.0.0.1:43820_ x:collection1 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1432045 T8085 n:127.0.0.1:43820_ x:collection1 oasc.SolrCore.<init> [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 9840BAED8B010DDA-001/shard-1-001/cores/collection1/], dataDir=[null]
   [junit4]   2> 1432046 T8085 n:127.0.0.1:43820_ x:collection1 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@513effe
   [junit4]   2> 1432046 T8085 n:127.0.0.1:43820_ x:collection1 oasc.CachingDirectoryFactory.get return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 9840BAED8B010DDA-001/shard-1-001/cores/collection1/data
   [junit4]   2> 1432046 T8085 n:127.0.0.1:43820_ x:collection1 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 9840BAED8B010DDA-001/shard-1-001/cores/collection1/data/index/
   [junit4]   2> 1432047 T8085 n:127.0.0.1:43820_ x:collection1 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 9840BAED8B010DDA-001/shard-1-001/cores/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1432047 T8085 n:127.0.0.1:43820_ x:collection1 oasc.CachingDirectoryFactory.get return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 9840BAED8B010DDA-001/shard-1-001/cores/collection1/data/index
   [junit4]   2> 1432047 T8085 n:127.0.0.1:43820_ x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=10, maxMergeAtOnceExplicit=40, maxMergedSegmentMB=46.45703125, floorSegmentMB=0.359375, forceMergeDeletesPctAllowed=4.1970158864613, segmentsPerTier=34.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1432048 T8085 n:127.0.0.1:43820_ x:collection1 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@4f84f3ea lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4fb3db81),segFN=segments_1,generation=1}
   [junit4]   2> 1432048 T8085 n:127.0.0.1:43820_ x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1432052 T8085 n:127.0.0.1:43820_ x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1432052 T8085 n:127.0.0.1:43820_ x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1432052 T8085 n:127.0.0.1:43820_ x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1432052 T8085 n:127.0.0.1:43820_ x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1432053 T8085 n:127.0.0.1:43820_ x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1432053 T8085 n:127.0.0.1:43820_ x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1432053 T8085 n:127.0.0.1:43820_ x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1432053 T8085 n:127.0.0.1:43820_ x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1432053 T8085 n:127.0.0.1:43820_ x:collection1 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1432055 T8085 n:127.0.0.1:43820_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1432056 T8085 n:127.0.0.1:43820_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1432057 T8085 n:127.0.0.1:43820_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1432058 T8085 n:127.0.0.1:43820_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1432062 T8085 n:127.0.0.1:43820_ x:collection1 oasc.RequestHandlers.initHandlersFromConfig Registered paths: /admin/mbeans,standard,/update/csv,/update/json/docs,/admin/luke,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/schema,/admin/plugins,/admin/logging,/update/json,/admin/threads,/admin/ping,/update,/admin/file
   [junit4]   2> 1432062 T8085 n:127.0.0.1:43820_ x:collection1 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 1432063 T8085 n:127.0.0.1:43820_ x:collection1 oasu.UpdateHandler.<init> Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1432063 T8085 n:127.0.0.1:43820_ x:collection1 oasu.UpdateLog.init Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1432073 T8085 n:127.0.0.1:43820_ x:collection1 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1432074 T8085 n:127.0.0.1:43820_ x:collection1 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1432074 T8085 n:127.0.0.1:43820_ x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=27, maxMergeAtOnceExplicit=46, maxMergedSegmentMB=61.61328125, floorSegmentMB=1.8525390625, forceMergeDeletesPctAllowed=10.319260276732745, segmentsPerTier=46.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1432075 T8085 n:127.0.0.1:43820_ x:collection1 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@4f84f3ea lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4fb3db81),segFN=segments_1,generation=1}
   [junit4]   2> 1432075 T8085 n:127.0.0.1:43820_ x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1432075 T8085 n:127.0.0.1:43820_ x:collection1 oass.SolrIndexSearcher.<init> Opening Searcher@444ca24[collection1] main
   [junit4]   2> 1432075 T8085 n:127.0.0.1:43820_ x:collection1 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1432079 T8085 n:127.0.0.1:43820_ x:collection1 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1432080 T8085 n:127.0.0.1:43820_ x:collection1 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1432080 T8085 n:127.0.0.1:43820_ x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1432080 T8085 n:127.0.0.1:43820_ x:collection1 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 1432080 T8085 n:127.0.0.1:43820_ x:collection1 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1432081 T8085 n:127.0.0.1:43820_ x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1432081 T8085 n:127.0.0.1:43820_ x:collection1 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1432081 T8085 n:127.0.0.1:43820_ x:collection1 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 1432081 T8085 n:127.0.0.1:43820_ x:collection1 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 1432082 T8086 n:127.0.0.1:43820_ x:collection1 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@444ca24[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1432082 T8085 n:127.0.0.1:43820_ x:collection1 oasc.ZkController.getConfDirListeners watch zkdir /configs/conf1
   [junit4]   2> 1432083 T8085 n:127.0.0.1:43820_ x:collection1 oasu.UpdateLog.onFirstSearcher On first searcher opened, looking up max value of version field
   [junit4]   2> 1432083 T8085 n:127.0.0.1:43820_ x:collection1 oasu.VersionInfo.getMaxVersionFromIndex Refreshing highest value of _version_ for 65536 version buckets from index
   [junit4]   2> 1432083 T8085 n:127.0.0.1:43820_ x:collection1 oasu.VersionInfo.getMaxVersionFromIndex No terms found for _version_, cannot seed version bucket highest value from index
   [junit4]   2> 1432083 T8085 n:127.0.0.1:43820_ x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion Could not find max version in index or recent updates, using new clock 1502118470708363264
   [junit4]   2> 1432086 T8085 n:127.0.0.1:43820_ x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion Took 3 ms to seed version buckets with highest version 1502118470708363264
   [junit4]   2> 1432086 T8085 n:127.0.0.1:43820_ x:collection1 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1432087 T8089 n:127.0.0.1:43820_ c:collection1 s:shard2 x:collection1 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:43820 collection:collection1 shard:shard2
   [junit4]   2> 1432088 T8006 n:127.0.0.1:43820_ oascsi.Krb5HttpClientConfigurer.configure Setting up SPNego auth with config: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 9840BAED8B010DDA-001/tempDir-001/minikdc/jaas-client.conf
   [junit4]   2> 1432088 T8006 n:127.0.0.1:43820_ oass.SolrDispatchFilter.init user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2
   [junit4]   2> 1432088 T8006 n:127.0.0.1:43820_ oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1432089 T8006 oascsi.Krb5HttpClientConfigurer.configure Setting up SPNego auth with config: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 9840BAED8B010DDA-001/tempDir-001/minikdc/jaas-client.conf
   [junit4]   2> 1432092 T8089 n:127.0.0.1:43820_ c:collection1 s:shard2 x:collection1 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 1432096 T8089 n:127.0.0.1:43820_ c:collection1 s:shard2 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 1432097 T8048 n:127.0.0.1:41620_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1432097 T8089 n:127.0.0.1:43820_ c:collection1 s:shard2 x:collection1 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1432097 T8089 n:127.0.0.1:43820_ c:collection1 s:shard2 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C1287 name=collection1 org.apache.solr.core.SolrCore@aaa5bd9 url=http://127.0.0.1:43820/collection1 node=127.0.0.1:43820_ C1287_STATE=coll:collection1 core:collection1 props:{core=collection1, base_url=http://127.0.0.1:43820, node_name=127.0.0.1:43820_, state=down}
   [junit4]   2> 1432098 T8089 n:127.0.0.1:43820_ c:collection1 s:shard2 x:collection1 C1287 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:43820/collection1/
   [junit4]   2> 1432098 T8049 n:127.0.0.1:41620_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1"} current state version: 4
   [junit4]   2> 1432098 T8089 n:127.0.0.1:43820_ c:collection1 s:shard2 x:collection1 C1287 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1432098 T8089 n:127.0.0.1:43820_ c:collection1 s:shard2 x:collection1 C1287 oasc.SyncStrategy.syncToMe http://127.0.0.1:43820/collection1/ has no replicas
   [junit4]   2> 1432098 T8089 n:127.0.0.1:43820_ c:collection1 s:shard2 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:43820/collection1/ shard2
   [junit4]   2> 1432099 T8089 n:127.0.0.1:43820_ c:collection1 s:shard2 x:collection1 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 1432099 T8049 n:127.0.0.1:41620_ oasco.ZkStateWriter.writePendingUpdates going to update_collection /collections/collection1/state.json version: 1
   [junit4]   2> 1432099 T8083 n:127.0.0.1:43820_ oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json for collection collection1 has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1432101 T8083 n:127.0.0

[...truncated too long message...]

jsh.ContextHandler.doStop Stopped o.e.j.s.ServletContextHandler@6649544c{/,null,UNAVAILABLE}
   [junit4]   2> 1448592 T8006 c:control_collection s:shard1 x:collection1 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:57183 57183
   [junit4]   2> 1448774 T8014 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:57183 57183
   [junit4]   2> 1448775 T8014 oasc.ZkTestServer$ZKServerMain.runFromConfig WARN Watch limit violations: 
   [junit4]   2> 	Maximum concurrent create/delete watches above limit:
   [junit4]   2> 	
   [junit4]   2> 		7	/solr/aliases.json
   [junit4]   2> 		7	/solr/clusterstate.json
   [junit4]   2> 		5	/solr/configs/conf1
   [junit4]   2> 		4	/solr/collections/collection1/state.json
   [junit4]   2> 	
   [junit4]   2> 	Maximum concurrent children watches above limit:
   [junit4]   2> 	
   [junit4]   2> 		7	/solr/live_nodes
   [junit4]   2> 		5	/solr/overseer/collection-queue-work
   [junit4]   2> 		4	/solr/overseer/queue
   [junit4]   2> 	
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestSolrCloudWithKerberos -Dtests.method=testKerberizedSolr -Dtests.seed=9840BAED8B010DDA -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=ar_TN -Dtests.timezone=Atlantic/Reykjavik -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   24.6s J2 | TestSolrCloudWithKerberos.testKerberizedSolr <<<
   [junit4]    > Throwable #1: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:43820, http://127.0.0.1:45701, http://127.0.0.1:45570, http://127.0.0.1:45919, http://127.0.0.1:41620]
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([9840BAED8B010DDA:33596C2AC160CDED]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:355)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1086)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:856)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:799)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:135)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:152)
   [junit4]    > 	at org.apache.solr.cloud.TestSolrCloudWithKerberos.testKerberizedSolr(TestSolrCloudWithKerberos.java:159)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:960)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:935)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]    > Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:41620: Expected mime type application/octet-stream but got text/html. <html>
   [junit4]    > <head>
   [junit4]    > <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
   [junit4]    > <title>Error 500 </title>
   [junit4]    > </head>
   [junit4]    > <body>
   [junit4]    > <h2>HTTP ERROR: 500</h2>
   [junit4]    > <p>Problem accessing /admin/collections. Reason:
   [junit4]    > <pre>    org.apache.solr.common.SolrException: Error during request authentication, </pre></p>
   [junit4]    > <hr /><i><small>Powered by Jetty://</small></i>
   [junit4]    > </body>
   [junit4]    > </html>
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:529)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:235)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:227)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:376)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:328)
   [junit4]    > 	... 47 more
   [junit4]   2> 1448785 T8006 c:control_collection s:shard1 x:collection1 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithKerberos 9840BAED8B010DDA-001
   [junit4]   2> 24601 T8005 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50), sim=DefaultSimilarity, locale=ar_TN, timezone=Atlantic/Reykjavik
   [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 1.8.0_25 (64-bit)/cpus=4,threads=1,free=160314800,total=494927872
   [junit4]   2> NOTE: All tests run in this JVM: [TestCloudInspectUtil, TestTrie, OverseerCollectionProcessorTest, CachingDirectoryFactoryTest, TestNoOpRegenerator, DocValuesMissingTest, OpenExchangeRatesOrgProviderTest, SuggesterTest, DirectUpdateHandlerOptimizeTest, TestClusterStateMutator, CursorPagingTest, TestInitParams, HardAutoCommitTest, TestFastOutputStream, TestRawResponseWriter, TestManagedSchemaFieldResource, TestPartialUpdateDeduplication, SolrIndexSplitterTest, TestStressRecovery, AsyncMigrateRouteKeyTest, TestFieldCollectionResource, PrimUtilsTest, ActionThrottleTest, TestExactSharedStatsCache, ResponseLogComponentTest, HdfsRecoverLeaseTest, DeleteShardTest, XmlUpdateRequestHandlerTest, TestSurroundQueryParser, TestSchemaSimilarityResource, TestStandardQParsers, TestQueryUtils, CdcrUpdateLogTest, TestReload, TestMiniSolrCloudClusterSSL, TestUtils, TestReplicationHandlerBackup, MergeStrategyTest, RankQueryTest, SimpleCollectionCreateDeleteTest, TestLuceneMatchVersion, TestQuerySenderNoQuery, TestManagedStopFilterFactory, DistributedQueryElevationComponentTest, BinaryUpdateRequestHandlerTest, OutOfBoxZkACLAndCredentialsProvidersTest, TestReqParamsAPI, TestMacros, PathHierarchyTokenizerFactoryTest, TermVectorComponentDistributedTest, DistributedIntervalFacetingTest, TestRestManager, BasicDistributedZk2Test, DistributedFacetPivotWhiteBoxTest, TestSearcherReuse, FileBasedSpellCheckerTest, DOMUtilTest, OverseerTest, XsltUpdateRequestHandlerTest, TestSolrConfigHandler, TestSolrXml, TestObjectReleaseTracker, TestDefaultSimilarityFactory, TestReRankQParserPlugin, SoftAutoCommitTest, TestMaxScoreQueryParser, RecoveryZkTest, QueryEqualityTest, TestCollationField, FacetPivotSmallTest, CircularListTest, TestNRTOpen, SolrInfoMBeanTest, TestIntervalFaceting, MultiThreadedOCPTest, FullSolrCloudDistribCmdsTest, SimpleFacetsTest, TestCSVResponseWriter, UnloadDistributedZkTest, BasicZkTest, TestRandomFaceting, LeaderElectionTest, TestZkChroot, TestRandomDVFaceting, AutoCommitTest, TestRecovery, TestHashPartitioner, ZkControllerTest, TestStressReorder, TestJoin, TestStressVersions, DistributedTermsComponentTest, TestRangeQuery, SolrCoreTest, StatsComponentTest, SpellCheckComponentTest, TestGroupingSearch, BadIndexSchemaTest, TestFiltering, DirectUpdateHandlerTest, ShowFileRequestHandlerTest, SimplePostToolTest, TestExtendedDismaxParser, SuggesterFSTTest, DocValuesTest, SuggesterWFSTTest, WordBreakSolrSpellCheckerTest, TestPseudoReturnFields, FieldMutatingUpdateProcessorTest, StatelessScriptUpdateProcessorFactoryTest, StandardRequestHandlerTest, PrimitiveFieldTypeTest, TermsComponentTest, TestIndexingPerformance, MoreLikeThisHandlerTest, TestSolrQueryParser, IndexSchemaRuntimeFieldTest, RegexBoostProcessorTest, IndexSchemaTest, JSONWriterTest, TestAnalyzedSuggestions, CSVRequestHandlerTest, TestBinaryResponseWriter, SOLR749Test, CopyFieldTest, TestMergePolicyConfig, TestDocSet, NumericFieldsTest, TestSolrCoreProperties, TestPhraseSuggestions, TestCharFilters, TestCodecSupport, TestXIncludeConfig, TestDFRSimilarityFactory, TestLMDirichletSimilarityFactory, TestFastWriter, ResourceLoaderTest, ChaosMonkeyNothingIsSafeTest, CloudExitableDirectoryReaderTest, ConcurrentDeleteAndCreateCollectionTest, CustomCollectionTest, DistribDocExpirationUpdateProcessorTest, LeaderInitiatedRecoveryOnCommitTest, ReplicationFactorTest, SSLMigrationTest, SaslZkACLProviderTest, TestLeaderElectionZkExpiry, TestMiniSolrCloudCluster, TestReplicaProperties, TestShortCircuitedRequests, TestSolrCloudWithKerberos]
   [junit4] Completed [417/502] on J2 in 25.37s, 1 test, 1 error <<< FAILURES!

[...truncated 275 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/build.xml:526: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/build.xml:474: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/build.xml:61: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/extra-targets.xml:39: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build.xml:229: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/common-build.xml:512: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/lucene/common-build.xml:1415: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/lucene/common-build.xml:973: There were test failures: 502 suites, 1984 tests, 1 error, 35 ignored (25 assumptions)

Total time: 49 minutes 36 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-Tests-trunk-Java8 - Build # 8 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-trunk-Java8/8/

4 tests failed.
FAILED:  org.apache.solr.cloud.TestSolrCloudWithKerberos.testKerberizedSolr

Error Message:
Error from server at http://127.0.0.1:47301/sa_xas: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 401 </title> </head> <body> <h2>HTTP ERROR: 401</h2> <p>Problem accessing /sa_xas/admin/collections. Reason: <pre>    Authentication required</pre></p> <hr /><i><small>Powered by Jetty://</small></i> </body> </html> 

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:47301/sa_xas: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 401 </title>
</head>
<body>
<h2>HTTP ERROR: 401</h2>
<p>Problem accessing /sa_xas/admin/collections. Reason:
<pre>    Authentication required</pre></p>
<hr /><i><small>Powered by Jetty://</small></i>
</body>
</html>

	at __randomizedtesting.SeedInfo.seed([7E178E8EB7F6CF8F:D50E5849FD970FB8]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:529)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:235)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:227)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:376)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:328)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1086)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:856)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:799)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:135)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:152)
	at org.apache.solr.cloud.TestSolrCloudWithKerberos.testKerberizedSolr(TestSolrCloudWithKerberos.java:157)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:960)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:935)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestSolrCloudWithKerberos

Error Message:
Some resources were not closed, shutdown, or released.

Stack Trace:
java.lang.AssertionError: Some resources were not closed, shutdown, or released.
	at __randomizedtesting.SeedInfo.seed([7E178E8EB7F6CF8F]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:234)
	at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:799)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestSolrCloudWithKerberos

Error Message:
5 threads leaked from SUITE scope at org.apache.solr.cloud.TestSolrCloudWithKerberos:     1) Thread[id=5806, name=zkCallback-803-thread-2, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithKerberos]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=5802, name=TEST-TestSolrCloudWithKerberos.testKerberizedSolr-seed#[7E178E8EB7F6CF8F]-SendThread(127.0.0.1:54508), state=TIMED_WAITING, group=TGRP-TestSolrCloudWithKerberos]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)    3) Thread[id=5805, name=zkCallback-803-thread-1, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithKerberos]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)    4) Thread[id=5841, name=zkCallback-803-thread-3, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithKerberos]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)    5) Thread[id=5803, name=TEST-TestSolrCloudWithKerberos.testKerberizedSolr-seed#[7E178E8EB7F6CF8F]-EventThread, state=WAITING, group=TGRP-TestSolrCloudWithKerberos]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 5 threads leaked from SUITE scope at org.apache.solr.cloud.TestSolrCloudWithKerberos: 
   1) Thread[id=5806, name=zkCallback-803-thread-2, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithKerberos]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=5802, name=TEST-TestSolrCloudWithKerberos.testKerberizedSolr-seed#[7E178E8EB7F6CF8F]-SendThread(127.0.0.1:54508), state=TIMED_WAITING, group=TGRP-TestSolrCloudWithKerberos]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
   3) Thread[id=5805, name=zkCallback-803-thread-1, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithKerberos]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
   4) Thread[id=5841, name=zkCallback-803-thread-3, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithKerberos]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
   5) Thread[id=5803, name=TEST-TestSolrCloudWithKerberos.testKerberizedSolr-seed#[7E178E8EB7F6CF8F]-EventThread, state=WAITING, group=TGRP-TestSolrCloudWithKerberos]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
	at __randomizedtesting.SeedInfo.seed([7E178E8EB7F6CF8F]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestSolrCloudWithKerberos

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=5806, name=zkCallback-803-thread-2, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithKerberos]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=5802, name=TEST-TestSolrCloudWithKerberos.testKerberizedSolr-seed#[7E178E8EB7F6CF8F]-SendThread(127.0.0.1:54508), state=TIMED_WAITING, group=TGRP-TestSolrCloudWithKerberos]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)    3) Thread[id=5805, name=zkCallback-803-thread-1, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithKerberos]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)    4) Thread[id=5841, name=zkCallback-803-thread-3, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithKerberos]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=5806, name=zkCallback-803-thread-2, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithKerberos]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=5802, name=TEST-TestSolrCloudWithKerberos.testKerberizedSolr-seed#[7E178E8EB7F6CF8F]-SendThread(127.0.0.1:54508), state=TIMED_WAITING, group=TGRP-TestSolrCloudWithKerberos]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
   3) Thread[id=5805, name=zkCallback-803-thread-1, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithKerberos]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
   4) Thread[id=5841, name=zkCallback-803-thread-3, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithKerberos]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([7E178E8EB7F6CF8F]:0)




Build Log:
[...truncated 9651 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestSolrCloudWithKerberos
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos 7E178E8EB7F6CF8F-001/init-core-data-001
   [junit4]   2> 441308 T5642 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /sa_xas/
   [junit4]   2> 446691 T5642 oadsc.DefaultDirectoryService.showSecurityWarnings WARN You didn't change the admin password of directory service instance 'DefaultKrbServer'.  Please update the admin password as soon as possible to prevent a possible security breach.
   [junit4]   2> 447310 T5642 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 447311 T5651 oasc.ZkTestServer$2$1.setClientPort client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 447311 T5651 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 447411 T5642 oasc.ZkTestServer.run start zk server on port:54508
   [junit4]   2> 447411 T5642 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 447412 T5642 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 447413 T5656 oaz.ClientCnxn$SendThread.startConnect WARN SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos 7E178E8EB7F6CF8F-001/tempDir-001/minikdc/jaas-client.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
   [junit4]   2> 447414 T5658 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1f1214ba name:ZooKeeperConnection Watcher:127.0.0.1:54508 got event WatchedEvent state:AuthFailed type:None path:null path:null type:None
   [junit4]   2> 447415 T5658 oascc.ConnectionManager.process WARN zkClient received AuthFailed
   [junit4]   2> 447416 T5658 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1f1214ba name:ZooKeeperConnection Watcher:127.0.0.1:54508 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 447416 T5642 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 447416 T5642 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 447417 T5642 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 447420 T5642 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 447421 T5642 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 447421 T5659 oaz.ClientCnxn$SendThread.startConnect WARN SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos 7E178E8EB7F6CF8F-001/tempDir-001/minikdc/jaas-client.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
   [junit4]   2> 447422 T5661 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ca6fbdd name:ZooKeeperConnection Watcher:127.0.0.1:54508/solr got event WatchedEvent state:AuthFailed type:None path:null path:null type:None
   [junit4]   2> 447422 T5661 oascc.ConnectionManager.process WARN zkClient received AuthFailed
   [junit4]   2> 447422 T5661 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ca6fbdd name:ZooKeeperConnection Watcher:127.0.0.1:54508/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 447423 T5642 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 447423 T5642 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 447424 T5642 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 447426 T5642 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 447428 T5642 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 447430 T5642 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 447431 T5642 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 447432 T5642 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 447434 T5642 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 447434 T5642 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 447437 T5642 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 447437 T5642 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 447439 T5642 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 447439 T5642 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 447441 T5642 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 447441 T5642 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 447443 T5642 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 447443 T5642 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 447446 T5642 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 447446 T5642 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 447448 T5642 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 447449 T5642 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 447451 T5642 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 447451 T5642 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 447453 T5642 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 447454 T5642 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 447456 T5642 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 447456 T5642 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 447459 T5642 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 447460 T5642 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 447462 T5662 oaz.ClientCnxn$SendThread.startConnect WARN SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos 7E178E8EB7F6CF8F-001/tempDir-001/minikdc/jaas-client.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
   [junit4]   2> 447463 T5664 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7a3bccb4 name:ZooKeeperConnection Watcher:127.0.0.1:54508/solr got event WatchedEvent state:AuthFailed type:None path:null path:null type:None
   [junit4]   2> 447463 T5664 oascc.ConnectionManager.process WARN zkClient received AuthFailed
   [junit4]   2> 447463 T5665 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7a3bccb4 name:ZooKeeperConnection Watcher:127.0.0.1:54508/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 447463 T5642 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 447464 T5642 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 447795 T5642 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos 7E178E8EB7F6CF8F-001/control-001/cores/collection1
   [junit4]   2> 447799 T5642 oejs.Server.doStart jetty-9.2.10.v20150310
   [junit4]   2> 447804 T5642 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@ff755e9{/sa_xas,null,AVAILABLE}
   [junit4]   2> 447805 T5642 oejs.AbstractConnector.doStart Started ServerConnector@3502207{HTTP/1.1}{127.0.0.1:57696}
   [junit4]   2> 447805 T5642 oejs.Server.doStart Started @449680ms
   [junit4]   2> 447805 T5642 oascse.JettySolrRunner$1.lifeCycleStarted Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos 7E178E8EB7F6CF8F-001/tempDir-002/control/data, hostContext=/sa_xas, hostPort=57696, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos 7E178E8EB7F6CF8F-001/control-001/cores}
   [junit4]   2> 447806 T5642 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@c387f44
   [junit4]   2> 447806 T5642 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos 7E178E8EB7F6CF8F-001/control-001/'
   [junit4]   2> 447830 T5642 oasc.SolrXmlConfig.fromFile Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos 7E178E8EB7F6CF8F-001/control-001/solr.xml
   [junit4]   2> 447836 T5642 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos 7E178E8EB7F6CF8F-001/control-001/cores
   [junit4]   2> 447836 T5642 oasc.CoreContainer.<init> New CoreContainer 521419505
   [junit4]   2> 447836 T5642 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos 7E178E8EB7F6CF8F-001/control-001/]
   [junit4]   2> 447836 T5642 oasc.CoreContainer.load loading shared library: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos 7E178E8EB7F6CF8F-001/control-001/lib
   [junit4]   2> 447837 T5642 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos 7E178E8EB7F6CF8F-001/control-001/lib).
   [junit4]   2> 447844 T5642 oashc.HttpShardHandlerFactory.init created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 447846 T5642 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 447846 T5642 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 447847 T5642 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 447847 T5642 oasc.CoreContainer.load Node Name: 127.0.0.1
   [junit4]   2> 447847 T5642 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54508/solr
   [junit4]   2> 447847 T5642 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 447847 T5642 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 447848 T5642 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 447855 T5677 oaz.ClientCnxn$SendThread.startConnect WARN SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos 7E178E8EB7F6CF8F-001/tempDir-001/minikdc/jaas-client.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
   [junit4]   2> 447856 T5679 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@70fbafd name:ZooKeeperConnection Watcher:127.0.0.1:54508 got event WatchedEvent state:AuthFailed type:None path:null path:null type:None
   [junit4]   2> 447856 T5679 oascc.ConnectionManager.process WARN zkClient received AuthFailed
   [junit4]   2> 447858 T5679 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@70fbafd name:ZooKeeperConnection Watcher:127.0.0.1:54508 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 447858 T5642 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 447859 T5642 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 447864 T5642 n:127.0.0.1:57696_sa_xas oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 447873 T5680 n:127.0.0.1:57696_sa_xas oaz.ClientCnxn$SendThread.startConnect WARN SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos 7E178E8EB7F6CF8F-001/tempDir-001/minikdc/jaas-client.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
   [junit4]   2> 447888 T5682 n:127.0.0.1:57696_sa_xas oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@54a1612 name:ZooKeeperConnection Watcher:127.0.0.1:54508/solr got event WatchedEvent state:AuthFailed type:None path:null path:null type:None
   [junit4]   2> 447888 T5682 n:127.0.0.1:57696_sa_xas oascc.ConnectionManager.process WARN zkClient received AuthFailed
   [junit4]   2> 447888 T5683 n:127.0.0.1:57696_sa_xas oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@54a1612 name:ZooKeeperConnection Watcher:127.0.0.1:54508/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 447888 T5642 n:127.0.0.1:57696_sa_xas oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 447890 T5642 n:127.0.0.1:57696_sa_xas oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 447892 T5642 n:127.0.0.1:57696_sa_xas oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 447894 T5642 n:127.0.0.1:57696_sa_xas oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 447896 T5642 n:127.0.0.1:57696_sa_xas oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 447897 T5642 n:127.0.0.1:57696_sa_xas oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 447899 T5642 n:127.0.0.1:57696_sa_xas oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 447900 T5642 n:127.0.0.1:57696_sa_xas oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 447901 T5642 n:127.0.0.1:57696_sa_xas oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 447902 T5642 n:127.0.0.1:57696_sa_xas oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57696_sa_xas
   [junit4]   2> 447903 T5642 n:127.0.0.1:57696_sa_xas oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:57696_sa_xas
   [junit4]   2> 447904 T5642 n:127.0.0.1:57696_sa_xas oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 447905 T5642 n:127.0.0.1:57696_sa_xas oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 447906 T5642 n:127.0.0.1:57696_sa_xas oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 447907 T5642 n:127.0.0.1:57696_sa_xas oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:57696_sa_xas
   [junit4]   2> 447908 T5642 n:127.0.0.1:57696_sa_xas oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 447909 T5642 n:127.0.0.1:57696_sa_xas oasc.Overseer.start Overseer (id=93874899301171204-127.0.0.1:57696_sa_xas-n_0000000000) starting
   [junit4]   2> 447910 T5642 n:127.0.0.1:57696_sa_xas oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 447916 T5642 n:127.0.0.1:57696_sa_xas oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 447953 T5684 n:127.0.0.1:57696_sa_xas oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 447961 T5685 n:127.0.0.1:57696_sa_xas oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 447974 T5642 n:127.0.0.1:57696_sa_xas oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 447977 T5642 n:127.0.0.1:57696_sa_xas oasc.CoreContainer.initializeAuthenticationPlugin Authentication plugin class obtained from ZK: org.apache.solr.security.KerberosPlugin
   [junit4]   2> 447984 T5642 n:127.0.0.1:57696_sa_xas oass.KerberosPlugin.init Params: {token.valid=30, kerberos.principal=HTTP/127.0.0.1, cookie.domain=127.0.0.1, kerberos.keytab=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos 7E178E8EB7F6CF8F-001/tempDir-001/minikdc/keytabs, cookie.path=/, type=kerberos, kerberos.name.rules=DEFAULT}
   [junit4]   2> 448033 T5642 n:127.0.0.1:57696_sa_xas oasc.CoreContainer.initializeAuthenticationPlugin Reconfiguring the shard handler factory and update shard handler.
   [junit4]   2> 448034 T5642 n:127.0.0.1:57696_sa_xas oashc.HttpShardHandlerFactory.reconfigureHttpClient Reconfiguring the default client with: org.apache.solr.client.solrj.impl.Krb5HttpClientConfigurer@1a634ce2
   [junit4]   2> 448034 T5642 n:127.0.0.1:57696_sa_xas oascsi.Krb5HttpClientConfigurer.configure Setting up SPNego auth with config: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos 7E178E8EB7F6CF8F-001/tempDir-001/minikdc/jaas-client.conf
   [junit4]   2> 448037 T5642 n:127.0.0.1:57696_sa_xas oasu.UpdateShardHandler.reconfigureHttpClient Reconfiguring the default client with: org.apache.solr.client.solrj.impl.Krb5HttpClientConfigurer@1a634ce2
   [junit4]   2> 448038 T5642 n:127.0.0.1:57696_sa_xas oascsi.Krb5HttpClientConfigurer.configure Setting up SPNego auth with config: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos 7E178E8EB7F6CF8F-001/tempDir-001/minikdc/jaas-client.conf
   [junit4]   2> 448040 T5642 n:127.0.0.1:57696_sa_xas oasc.CorePropertiesLocator.discover Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos 7E178E8EB7F6CF8F-001/control-001/cores
   [junit4]   2> 448041 T5642 n:127.0.0.1:57696_sa_xas oasc.CoreDescriptor.<init> CORE DESCRIPTOR: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos 7E178E8EB7F6CF8F-001/control-001/cores/collection1, collection=control_collection, absoluteInstDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos 7E178E8EB7F6CF8F-001/control-001/cores/collection1/, coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 448041 T5642 n:127.0.0.1:57696_sa_xas oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos 7E178E8EB7F6CF8F-001/control-001/cores/collection1/
   [junit4]   2> 448041 T5642 n:127.0.0.1:57696_sa_xas oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 448042 T5688 n:127.0.0.1:57696_sa_xas c:control_collection x:collection1 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 448042 T5688 n:127.0.0.1:57696_sa_xas c:control_collection x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 448043 T5688 n:127.0.0.1:57696_sa_xas oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 448045 T5683 n:127.0.0.1:57696_sa_xas oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 448046 T5684 n:127.0.0.1:57696_sa_xas oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:57696/sa_xas",
   [junit4]   2> 	  "node_name":"127.0.0.1:57696_sa_xas",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"} current state version: 0
   [junit4]   2> 448047 T5684 n:127.0.0.1:57696_sa_xas oasco.ReplicaMutator.updateState Update state numShards=1 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:57696/sa_xas",
   [junit4]   2> 	  "node_name":"127.0.0.1:57696_sa_xas",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 448047 T5684 n:127.0.0.1:57696_sa_xas oasco.ClusterStateMutator.createCollection building a new cName: control_collection
   [junit4]   2> 448047 T5684 n:127.0.0.1:57696_sa_xas oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 448049 T5683 n:127.0.0.1:57696_sa_xas oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 449043 T5688 n:127.0.0.1:57696_sa_xas oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 449044 T5688 n:127.0.0.1:57696_sa_xas oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 449044 T5688 n:127.0.0.1:57696_sa_xas oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 449044 T5688 n:127.0.0.1:57696_sa_xas oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 449045 T5688 n:127.0.0.1:57696_sa_xas oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 449045 T5688 n:127.0.0.1:57696_sa_xas oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos 7E178E8EB7F6CF8F-001/control-001/cores/collection1/'
   [junit4]   2> 449067 T5688 n:127.0.0.1:57696_sa_xas oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 449073 T5688 n:127.0.0.1:57696_sa_xas oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 449101 T5688 n:127.0.0.1:57696_sa_xas oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 449109 T5688 n:127.0.0.1:57696_sa_xas oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 449110 T5688 n:127.0.0.1:57696_sa_xas oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 449115 T5688 n:127.0.0.1:57696_sa_xas oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 449235 T5688 n:127.0.0.1:57696_sa_xas oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 449240 T5688 n:127.0.0.1:57696_sa_xas oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 449241 T5688 n:127.0.0.1:57696_sa_xas oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 449249 T5688 n:127.0.0.1:57696_sa_xas oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 449252 T5688 n:127.0.0.1:57696_sa_xas oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 449253 T5688 n:127.0.0.1:57696_sa_xas oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 449254 T5688 n:127.0.0.1:57696_sa_xas oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 449254 T5688 n:127.0.0.1:57696_sa_xas oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 449254 T5688 n:127.0.0.1:57696_sa_xas oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 449255 T5688 n:127.0.0.1:57696_sa_xas oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 449255 T5688 n:127.0.0.1:57696_sa_xas oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 449255 T5688 n:127.0.0.1:57696_sa_xas oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 449256 T5688 n:127.0.0.1:57696_sa_xas x:collection1 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 449256 T5688 n:127.0.0.1:57696_sa_xas x:collection1 oasc.SolrCore.<init> [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos 7E178E8EB7F6CF8F-001/control-001/cores/collection1/], dataDir=[null]
   [junit4]   2> 449256 T5688 n:127.0.0.1:57696_sa_xas x:collection1 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2817f3b1
   [junit4]   2> 449257 T5688 n:127.0.0.1:57696_sa_xas x:collection1 oasc.CachingDirectoryFactory.get return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos 7E178E8EB7F6CF8F-001/control-001/cores/collection1/data
   [junit4]   2> 449257 T5688 n:127.0.0.1:57696_sa_xas x:collection1 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos 7E178E8EB7F6CF8F-001/control-001/cores/collection1/data/index/
   [junit4]   2> 449257 T5688 n:127.0.0.1:57696_sa_xas x:collection1 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos 7E178E8EB7F6CF8F-001/control-001/cores/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 449257 T5688 n:127.0.0.1:57696_sa_xas x:collection1 oasc.CachingDirectoryFactory.get return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos 7E178E8EB7F6CF8F-001/control-001/cores/collection1/data/index
   [junit4]   2> 449258 T5688 n:127.0.0.1:57696_sa_xas x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=30, maxMergeAtOnceExplicit=12, maxMergedSegmentMB=72.904296875, floorSegmentMB=1.474609375, forceMergeDeletesPctAllowed=11.71015669263026, segmentsPerTier=50.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.551230275170388
   [junit4]   2> 449258 T5688 n:127.0.0.1:57696_sa_xas x:collection1 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@166e2cd2 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@6f314c87),segFN=segments_1,generation=1}
   [junit4]   2> 449259 T5688 n:127.0.0.1:57696_sa_xas x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 449262 T5688 n:127.0.0.1:57696_sa_xas x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 449262 T5688 n:127.0.0.1:57696_sa_xas x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 449262 T5688 n:127.0.0.1:57696_sa_xas x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 449262 T5688 n:127.0.0.1:57696_sa_xas x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 449263 T5688 n:127.0.0.1:57696_sa_xas x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 449263 T5688 n:127.0.0.1:57696_sa_xas x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 449263 T5688 n:127.0.0.1:57696_sa_xas x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 449263 T5688 n:127.0.0.1:57696_sa_xas x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 449263 T5688 n:127.0.0.1:57696_sa_xas x:collection1 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 449265 T5688 n:127.0.0.1:57696_sa_xas x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 449266 T5688 n:127.0.0.1:57696_sa_xas x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 449267 T5688 n:127.0.0.1:57696_sa_xas x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 449268 T5688 n:127.0.0.1:57696_sa_xas x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 449271 T5688 n:127.0.0.1:57696_sa_xas x:collection1 oasc.RequestHandlers.initHandlersFromConfig Registered paths: /admin/mbeans,standard,/update/csv,/update/json/docs,/admin/luke,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/schema,/admin/plugins,/admin/logging,/update/json,/admin/threads,/admin/ping,/update,/admin/file
   [junit4]   2> 449272 T5688 n:127.0.0.1:57696_sa_xas x:collection1 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 449272 T5688 n:127.0.0.1:57696_sa_xas x:collection1 oasu.UpdateHandler.<init> Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 449272 T5688 n:127.0.0.1:57696_sa_xas x:collection1 oasu.UpdateLog.init Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 449274 T5688 n:127.0.0.1:57696_sa_xas x:collection1 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 449275 T5688 n:127.0.0.1:57696_sa_xas x:collection1 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 449275 T5688 n:127.0.0.1:57696_sa_xas x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=37, maxMergeAtOnceExplicit=13, maxMergedSegmentMB=92.451171875, floorSegmentMB=1.6708984375, forceMergeDeletesPctAllowed=23.40399930118291, segmentsPerTier=17.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6173965239428705
   [junit4]   2> 449276 T5688 n:127.0.0.1:57696_sa_xas x:collection1 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@166e2cd2 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@6f314c87),segFN=segments_1,generation=1}
   [junit4]   2> 449276 T5688 n:127.0.0.1:57696_sa_xas x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 449281 T5688 n:127.0.0.1:57696_sa_xas x:collection1 oass.SolrIndexSearcher.<init> Opening Searcher@2431881e[collection1] main
   [junit4]   2> 449282 T5688 n:127.0.0.1:57696_sa_xas x:collection1 oasu.UpdateLog.onFirstSearcher On first searcher opened, looking up max value of version field
   [junit4]   2> 449282 T5688 n:127.0.0.1:57696_sa_xas x:collection1 oasu.VersionInfo.getMaxVersionFromIndex Refreshing highest value of _version_ for 65536 version buckets from index
   [junit4]   2> 449282 T5688 n:127.0.0.1:57696_sa_xas x:collection1 oasu.VersionInfo.getMaxVersionFromIndex WARN No terms found for _version_, cannot seed version bucket highest value from index
   [junit4]   2> 449282 T5688 n:127.0.0.1:57696_sa_xas x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion WARN Could not find max version in index or recent updates, using new clock 1501998390884433920
   [junit4]   2> 449286 T5688 n:127.0.0.1:57696_sa_xas x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion Took 4 ms to seed version buckets with highest version 1501998390884433920
   [junit4]   2> 449286 T5688 n:127.0.0.1:57696_sa_xas x:collection1 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 449287 T5688 n:127.0.0.1:57696_sa_xas x:collection1 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 449287 T5688 n:127.0.0.1:57696_sa_xas x:collection1 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 449288 T5688 n:127.0.0.1:57696_sa_xas x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 449288 T5688 n:127.0.0.1:57696_sa_xas x:collection1 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 449288 T5688 n:127.0.0.1:57696_sa_xas x:collection1 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 449289 T5688 n:127.0.0.1:57696_sa_xas x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 449289 T5688 n:127.0.0.1:57696_sa_xas x:collection1 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 449289 T5688 n:127.0.0.1:57696_sa_xas x:collection1 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 449289 T5688 n:127.0.0.1:57696_sa_xas x:collection1 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 449290 T5688 n:127.0.0.1:57696_sa_xas x:collection1 oasc.ZkController.getConfDirListeners watch zkdir /configs/conf1
   [junit4]   2> 449290 T5689 n:127.0.0.1:57696_sa_xas x:collection1 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2431881e[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 449290 T5688 n:127.0.0.1:57696_sa_xas x:collection1 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 449292 T5692 n:127.0.0.1:57696_sa_xas c:control_collection s:shard1 x:collection1 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:57696/sa_xas collection:control_collection shard:shard1
   [junit4]   2> 449292 T5642 n:127.0.0.1:57696_sa_xas oascsi.Krb5HttpClientConfigurer.configure Setting up SPNego auth with config: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos 7E178E8EB7F6CF8F-001/tempDir-001/minikdc/jaas-client.conf
   [junit4]   2> 449292 T5642 n:127.0.0.1:57696_sa_xas oass.SolrDispatchFilter.init user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1
   [junit4]   2> 449292 T5642 n:127.0.0.1:57696_sa_xas oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 449292 T5692 n:127.0.0.1:57696_sa_xas c:control_collection s:shard1 x:collection1 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 449294 T5642 oascsi.Krb5HttpClientConfigurer.configure Setting up SPNego auth with config: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos 7E178E8EB7F6CF8F-001/tempDir-001/minikdc/jaas-client.conf
   [junit4]   2> 449294 T5642 oascsi.Krb5HttpClientConfigurer.configure Setting up SPNego auth with config: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos 7E178E8EB7F6CF8F-001/tempDir-001/minikdc/jaas-client.conf
   [junit4]   2> 449295 T5642 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 449296 T5642 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 449302 T5692 n:127.0.0.1:57696_sa_xas c:control_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 449302 T5693 oaz.ClientCnxn$SendThread.startConnect WARN SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos 7E178E8EB7F6CF8F-001/tempDir-001/minikdc/jaas-client.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
   [junit4]   2> 449303 T5683 n:127.0.0.1:57696_sa_xas oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 449304 T5684 n:127.0.0.1:57696_sa_xas oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection"} current state version: 1
   [junit4]   2> 449309 T5692 n:127.0.0.1:57696_sa_xas c:control_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 449310 T5692 n:127.0.0.1:57696_sa_xas c:control_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 449310 T5695 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@51d3fd52 name:ZooKeeperConnection Watcher:127.0.0.1:54508/solr got event WatchedEvent state:AuthFailed type:None path:null path:null type:None
   [junit4]   2> 449310 T5695 oascc.ConnectionManager.process WARN zkClient received AuthFailed
   [junit4]   2> ASYNC  NEW_CORE C565 name=collection1 org.apache.solr.core.SolrCore@2581dd64 url=http://127.0.0.1:57696/sa_xas/collection1 node=127.0.0.1:57696_sa_xas C565_STATE=coll:control_collection core:collection1 props:{core=collection1, base_url=http://127.0.0.1:57696/sa_xas, node_name=127.0.0.1:57696_sa_xas, state=down}
   [junit4]   2> 449310 T5692 n:127.0.0.1:57696_sa_xas c:control_collection s:shard1 x:collection1 C565 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:57696/sa_xas/collection1/
   [junit4]   2> 449310 T5692 n:127.0.0.1:57696_sa_xas c:control_collection s:shard1 x:collection1 C565 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 449310 T5692 n:127.0.0.1:57696_sa_xas c:control_collection s:shard1 x:collection1 C565 oasc.SyncStrategy.syncToMe http://127.0.0.1:57696/sa_xas/collection1/ has no replicas
   [junit4]   2> 449310 T5692 n:127.0.0.1:57696_sa_xas c:control_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:57696/sa_xas/collection1/ shard1
   [junit4]   2> 449311 T5692 n:127.0.0.1:57696_sa_xas c:control_collection s:shard1 x:collection1 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 449312 T5696 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@51d3fd52 name:ZooKeeperConnection Watcher:127.0.0.1:54508/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 449313 T5642 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 449314 T5642 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 449314 T5642 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 449315 T5683 n:127.0.0.1:57696_sa_xas oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 449316 T5684 n:127.0.0.1:57696_sa_xas oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:57696/sa_xas",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "state":"active"} current state version: 1
   [junit4]   2> 449318 T5642 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 449318 T5642 oasc.AbstractFullDistribZkTestBase.createJettys Creating collection1 with stateFormat=2
   [junit4]   2> 449318 T5642 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 449319 T5642 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 449322 T5697 oaz.ClientCnxn$SendThread.startConnect WARN SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos 7E178E8EB7F6CF8F-001/tempDir-001/minikdc/jaas-client.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
   [junit4]   2> 449323 T5699 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@21e9b341 name:ZooKeeperConnection Watcher:127.0.0.1:54508/solr got event WatchedEvent state:AuthFailed type:None path:null path:null type:None
   [junit4]   2> 449323 T5699 oascc.ConnectionManager.process WARN zkClient received AuthFailed
   [junit4]   2> 449323 T5699 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@21e9b341 name:ZooKeeperConnection Watcher:127.0.0.1:54508/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 449323 T5642 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 449324 T5642 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 449325 T5683 n:127.0.0.1:57696_sa_xas oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 449330 T5684 n:127.0.0.1:57696_sa_xas oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"create",
   [junit4]   2> 	  "name":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "stateFormat":"2"} current state version: 1
   [junit4]   2> 449331 T5684 n:127.0.0.1:57696_sa_xas oasco.ClusterStateMutator.createCollection building a new cName: collection1
   [junit4]   2> 449331 T5683 n:127.0.0.1:57696_sa_xas oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 449332 T5696 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 449367 T5692 n:127.0.0.1:57696_sa_xas c:control_collection s:shard1 x:collection1 oasc.ZkController.register We are http://127.0.0.1:57696/sa_xas/collection1/ and leader is http://127.0.0.1:57696/sa_xas/collection1/
   [junit4]   2> 449367 T5692 n:127.0.0.1:57696_sa_xas c:control_collection s:shard1 x:collection1 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:57696/sa_xas
   [junit4]   2> 449367 T5692 n:127.0.0.1:57696_sa_xas c:control_collection s:shard1 x:collection1 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 449368 T5692 n:127.0.0.1:57696_sa_xas c:control_collection s:shard1 x:collection1 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 449368 T5692 n:127.0.0.1:57696_sa_xas c:control_collection s:shard1 x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 449369 T5683 n:127.0.0.1:57696_sa_xas oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 449369 T5684 n:127.0.0.1:57696_sa_xas oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:57696/sa_xas",
   [junit4]   2> 	  "node_name":"127.0.0.1:57696_sa_xas",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"} current state version: 2
   [junit4]   2> 449370 T5684 n:127.0.0.1:57696_sa_xas oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:57696/sa_xas",
   [junit4]   2> 	  "node_name":"127.0.0.1:57696_sa_xas",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 449371 T5684 n:127.0.0.1:57696_sa_xas oasco.ZkStateWriter.writePendingUpdates going to create_collection /collections/collection1/state.json
   [junit4]   2> 449371 T5696 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 449372 T5683 n:127.0.0.1:57696_sa_xas oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 449482 T5696 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 449482 T5683 n:127.0.0.1:57696_sa_xas oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 449643 T5642 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos 7E178E8EB7F6CF8F-001/shard-1-001/cores/collection1
   [junit4]   2> 449644 T5642 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos 7E178E8EB7F6CF8F-001/shard-1-001
   [junit4]   2> 449645 T5642 oejs.Server.doStart jetty-9.2.10.v20150310
   [junit4]   2> 449647 T5642 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@791de837{/sa_xas,null,AVAILABLE}
   [junit4]   2> 449647 T5642 oejs.AbstractConnector.doStart Started ServerConnector@170bd73{HTTP/1.1}{127.0.0.1:57923}
   [junit4]   2> 449647 T5642 oejs.Server.doStart Started @451523ms
   [junit4]   2> 449648 T5642 oascse.JettySolrRunner$1.lifeCycleStarted Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos 7E178E8EB7F6CF8F-001/tempDir-002/jetty1, solrconfig=solrconfig.xml, hostContext=/sa_xas, hostPort=57923, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos 7E178E8EB7F6CF8F-001/shard-1-001/cores}
   [junit4]   2> 449648 T5642 oascsi.Krb5HttpClientConfigurer.configure Setting up SPNego auth with config: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos 7E178E8EB7F6CF8F-001/tempDir-001/minikdc/jaas-client.conf
   [junit4]   2> 449649 T5642 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@c387f44
   [junit4]   2> 449649 T5642 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos 7E178E8EB7F6CF8F-001/shard-1-001/'
   [junit4]   2> 449662 T5642 oasc.SolrXmlConfig.fromFile Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos 7E178E8EB7F6CF8F-001/shard-1-001/solr.xml
   [junit4]   2> 449677 T5642 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos 7E178E8EB7F6CF8F-001/shard-1-001/cores
   [junit4]   2> 449677 T5642 oasc.CoreContainer.<init> New CoreContainer 345418655
   [junit4]   2> 449678 T5642 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos 7E178E8EB7F6CF8F-001/shard-1-001/]
   [junit4]   2> 449679 T5642 oasc.CoreContainer.load loading shared library: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos 7E178E8EB7F6CF8F-001/shard-1-001/lib
   [junit4]   2> 449679 T5642 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos 7E178E8EB7F6CF8F-001/shard-1-001/lib).
   [junit4]   2> 449690 T5642 oashc.HttpShardHandlerFactory.init created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 449693 T5642 oascsi.Krb5HttpClientConfigurer.configure Setting up SPNego auth with config: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos 7E178E8EB7F6CF8F-001/tempDir-001/minikdc/jaas-client.conf
   [junit4]   2> 449693 T5642 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 449694 T5642 oascsi.Krb5HttpClientConfigurer.configure Setting up SPNego auth with config: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos 7E178E8EB7F6CF8F-001/tempDir-001/minikdc/jaas-client.conf
   [junit4]   2> 449695 T5642 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 449695 T5642 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 449695 T5642 oasc.CoreContainer.load Node Name: 127.0.0.1
   [junit4]   2> 449695 T5642 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54508/solr
   [junit4]   2> 449696 T5642 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 449696 T5642 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 449697 T5642 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 449698 T5711 oaz.ClientCnxn$SendThread.startConnect WARN SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos 7E178E8EB7F6CF8F-001/tempDir-001/minikdc/jaas-client.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
   [junit4]   2> 449699 T5713 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1cc072f6 name:ZooKeeperConnection Watcher:127.0.0.1:54508 got event WatchedEvent state:AuthFailed type:None path:null path:null type:None
   [junit4]   2> 449700 T5713 oascc.ConnectionManager.process WARN zkClient received AuthFailed
   [junit4]   2> 449701 T5713 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1cc072f6 name:ZooKeeperConnection Watcher:127.0.0.1:54508 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 449701 T5642 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 449702 T5642 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 449706 T5642 n:127.0.0.1:57923_sa_xas oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 449710 T5714 n:127.0.0.1:57923_sa_xas oaz.ClientCnxn$SendThread.startConnect WARN SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos 7E178E8EB7F6CF8F-001/tempDir-001/minikdc/jaas-client.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
   [junit4]   2> 449712 T5716 n:127.0.0.1:57923_sa_xas oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2ad38e5e name:ZooKeeperConnection Watcher:127.0.0.1:54508/solr got event WatchedEvent state:AuthFailed type:None path:null path:null type:None
   [junit4]   2> 449712 T5716 n:127.0.0.1:57923_sa_xas oascc.ConnectionManager.process WARN zkClient received AuthFailed
   [junit4]   2> 449718 T5717 n:127.0.0.1:57923_sa_xas oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2ad38e5e name:ZooKeeperConnection Watcher:127.0.0.1:54508/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 449718 T5642 n:127.0.0.1:57923_sa_xas oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 449728 T5642 n:127.0.0.1:57923_sa_xas oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 450733 T5642 n:127.0.0.1:57923_sa_xas oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57923_sa_xas
   [junit4]   2> 450734 T5642 n:127.0.0.1:57923_sa_xas oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:57923_sa_xas
   [junit4]   2> 450737 T5642 n:127.0.0.1:57923_sa_xas oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 450738 T5642 n:127.0.0.1:57923_sa_xas oasc.CoreContainer.initializeAuthenticationPlugin Authentication plugin class obtained from ZK: org.apache.solr.security.KerberosPlugin
   [junit4]   2> 450739 T5642 n:127.0.0.1:57923_sa_xas oass.KerberosPlugin.init Params: {token.valid=30, kerberos.principal=HTTP/127.0.0.1, cookie.domain=127.0.0.1, kerberos.keytab=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos 7E178E8EB7F6CF8F-001/tempDir-001/minikdc/keytabs, cookie.path=/, type=kerberos, kerberos.name.rules=DEFAULT}
   [junit4]   2> 450744 T5642 n:127.0.0.1:57923_sa_xas oasc.CoreContainer.initializeAuthenticationPlugin Reconfiguring the shard handler factory and update shard handler.
   [junit4]   2> 450744 T5642 n:127.0.0.1:57923_sa_xas oashc.HttpShardHandlerFactory.reconfigureHttpClient Reconfiguring the default client with: org.apache.solr.client.solrj.impl.Krb5HttpClientConfigurer@3ff25e5d
   [junit4]   2> 450744 T5642 n:127.0.0.1:57923_sa_xas oascsi.Krb5HttpClientConfigurer.configure Setting up SPNego auth with config: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos 7E178E8EB7F6CF8F-001/tempDir-001/minikdc/jaas-client.conf
   [junit4]   2> 450745 T5642 n:127.0.0.1:57923_sa_xas oasu.UpdateShardHandler.reconfigureHttpClient Reconfiguring the default client with: org.apache.solr.client.solrj.impl.Krb5HttpClientConfigurer@3ff25e5d
   [junit4]   2> 450745 T5642 n:127.0.0.1:57923_sa_xas oascsi.Krb5HttpClientConfigurer.configure Setting up SPNego auth with config: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos 7E178E8EB7F6CF8F-001/tempDir-001/minikdc/jaas-client.conf
   [junit4]   2> 450746 T5642 n:127.0.0.1:57923_sa_xas oasc.CorePropertiesLocator.discover Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos 7E178E8EB7F6CF8F-001/shard-1-001/cores
   [junit4]   2> 450747 T5642 n:127.0.0.1:57923_sa_xas oasc.CoreDescriptor.<init> CORE DESCRIPTOR: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos 7E178E8EB7F6CF8F-001/shard-1-001/cores/collection1, collection=collection1, absoluteInstDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos 7E178E8EB7F6CF8F-001/shard-1-001/cores/collection1/, coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 450748 T5642 n:127.0.0.1:57923_sa_xas oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos 7E178E8EB7F6CF8F-001/shard-1-001/cores/collection1/
   [junit4]   2> 450748 T5642 n:127.0.0.1:57923_sa_xas oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 450750 T5719 n:127.0.0.1:57923_sa_xas c:collection1 x:collection1 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 450750 T5719 n:127.0.0.1:57923_sa_xas c:collection1 x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 450752 T5683 n:127.0.0.1:57696_sa_xas oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 450752 T5684 n:127.0.0.1:57696_sa_xas oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:57923/sa_xas",
   [junit4]   2> 	  "node_name":"127.0.0.1:57923_sa_xas",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"} current state version: 4
   [junit4]   2> 450753 T5684 n:127.0.0.1:57696_sa_xas oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:57923/sa_xas",
   [junit4]   2> 	  "node_name":"127.0.0.1:57923_sa_xas",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   

[...truncated too long message...]

breville -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J1 | TestSolrCloudWithKerberos (suite) <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: Some resources were not closed, shutdown, or released.
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([7E178E8EB7F6CF8F]:0)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:234)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: 5 threads leaked from SUITE scope at org.apache.solr.cloud.TestSolrCloudWithKerberos: 
   [junit4]    >    1) Thread[id=5806, name=zkCallback-803-thread-2, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithKerberos]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >    2) Thread[id=5802, name=TEST-TestSolrCloudWithKerberos.testKerberizedSolr-seed#[7E178E8EB7F6CF8F]-SendThread(127.0.0.1:54508), state=TIMED_WAITING, group=TGRP-TestSolrCloudWithKerberos]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
   [junit4]    >    3) Thread[id=5805, name=zkCallback-803-thread-1, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithKerberos]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >    4) Thread[id=5841, name=zkCallback-803-thread-3, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithKerberos]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >    5) Thread[id=5803, name=TEST-TestSolrCloudWithKerberos.testKerberizedSolr-seed#[7E178E8EB7F6CF8F]-EventThread, state=WAITING, group=TGRP-TestSolrCloudWithKerberos]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([7E178E8EB7F6CF8F]:0)Throwable #3: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=5806, name=zkCallback-803-thread-2, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithKerberos]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >    2) Thread[id=5802, name=TEST-TestSolrCloudWithKerberos.testKerberizedSolr-seed#[7E178E8EB7F6CF8F]-SendThread(127.0.0.1:54508), state=TIMED_WAITING, group=TGRP-TestSolrCloudWithKerberos]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
   [junit4]    >    3) Thread[id=5805, name=zkCallback-803-thread-1, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithKerberos]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >    4) Thread[id=5841, name=zkCallback-803-thread-3, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithKerberos]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([7E178E8EB7F6CF8F]:0)
   [junit4] Completed [143/502] on J1 in 52.93s, 1 test, 1 failure, 3 errors <<< FAILURES!

[...truncated 1092 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/build.xml:526: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/build.xml:474: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/build.xml:61: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/extra-targets.xml:39: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build.xml:229: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/common-build.xml:512: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/lucene/common-build.xml:1415: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/lucene/common-build.xml:973: There were test failures: 502 suites, 1983 tests, 3 suite-level errors, 1 error, 1212 ignored (21 assumptions)

Total time: 37 minutes 53 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



RE: [JENKINS] Lucene-Solr-Tests-trunk-Java8 - Build # 7 - Still Failing

Posted by Ishan Chattopadhyaya <ic...@gmail.com>.
I'll have a look. So far I've not hit a single failure for this in about 15 full suite runs.

Sent from my Windows Phone

-----Original Message-----
From: "Apache Jenkins Server" <je...@builds.apache.org>
Sent: ‎5/‎23/‎2015 7:12 PM
To: "dev@lucene.apache.org" <de...@lucene.apache.org>
Subject: [JENKINS] Lucene-Solr-Tests-trunk-Java8 - Build # 7 - Still Failing

Build: https://builds.apache.org/job/Lucene-Solr-Tests-trunk-Java8/7/

4 tests failed.
FAILED:  org.apache.solr.cloud.TestSolrCloudWithKerberos.testKerberizedSolr

Error Message:
Error from server at http://127.0.0.1:41477/jv: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 401 </title> </head> <body> <h2>HTTP ERROR: 401</h2> <p>Problem accessing /jv/admin/collections. Reason: <pre>    Authentication required</pre></p> <hr /><i><small>Powered by Jetty://</small></i> </body> </html> 

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:41477/jv: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 401 </title>
</head>
<body>
<h2>HTTP ERROR: 401</h2>
<p>Problem accessing /jv/admin/collections. Reason:
<pre>    Authentication required</pre></p>
<hr /><i><small>Powered by Jetty://</small></i>
</body>
</html>

	at __randomizedtesting.SeedInfo.seed([CEC58863C9722182:65DC5EA48313E1B5]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:529)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:235)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:227)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:376)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:328)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1086)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:856)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:799)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:135)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:152)
	at org.apache.solr.cloud.TestSolrCloudWithKerberos.testKerberizedSolr(TestSolrCloudWithKerberos.java:157)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:960)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:935)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestSolrCloudWithKerberos

Error Message:
Some resources were not closed, shutdown, or released.

Stack Trace:
java.lang.AssertionError: Some resources were not closed, shutdown, or released.
	at __randomizedtesting.SeedInfo.seed([CEC58863C9722182]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:234)
	at sun.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:799)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestSolrCloudWithKerberos

Error Message:
5 threads leaked from SUITE scope at org.apache.solr.cloud.TestSolrCloudWithKerberos:     1) Thread[id=11044, name=zkCallback-1760-thread-1, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithKerberos]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=11080, name=zkCallback-1760-thread-3, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithKerberos]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)    3) Thread[id=11043, name=TEST-TestSolrCloudWithKerberos.testKerberizedSolr-seed#[CEC58863C9722182]-EventThread, state=WAITING, group=TGRP-TestSolrCloudWithKerberos]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)    4) Thread[id=11042, name=TEST-TestSolrCloudWithKerberos.testKerberizedSolr-seed#[CEC58863C9722182]-SendThread(127.0.0.1:57610), state=TIMED_WAITING, group=TGRP-TestSolrCloudWithKerberos]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)    5) Thread[id=11045, name=zkCallback-1760-thread-2, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithKerberos]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 5 threads leaked from SUITE scope at org.apache.solr.cloud.TestSolrCloudWithKerberos: 
   1) Thread[id=11044, name=zkCallback-1760-thread-1, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithKerberos]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=11080, name=zkCallback-1760-thread-3, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithKerberos]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
   3) Thread[id=11043, name=TEST-TestSolrCloudWithKerberos.testKerberizedSolr-seed#[CEC58863C9722182]-EventThread, state=WAITING, group=TGRP-TestSolrCloudWithKerberos]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
   4) Thread[id=11042, name=TEST-TestSolrCloudWithKerberos.testKerberizedSolr-seed#[CEC58863C9722182]-SendThread(127.0.0.1:57610), state=TIMED_WAITING, group=TGRP-TestSolrCloudWithKerberos]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
   5) Thread[id=11045, name=zkCallback-1760-thread-2, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithKerberos]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([CEC58863C9722182]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestSolrCloudWithKerberos

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=11044, name=zkCallback-1760-thread-1, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithKerberos]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=11080, name=zkCallback-1760-thread-3, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithKerberos]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)    3) Thread[id=11042, name=TEST-TestSolrCloudWithKerberos.testKerberizedSolr-seed#[CEC58863C9722182]-SendThread(127.0.0.1:57610), state=TIMED_WAITING, group=TGRP-TestSolrCloudWithKerberos]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)    4) Thread[id=11045, name=zkCallback-1760-thread-2, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithKerberos]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.

[The entire original message is not included.]

[JENKINS] Lucene-Solr-Tests-trunk-Java8 - Build # 7 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-trunk-Java8/7/

4 tests failed.
FAILED:  org.apache.solr.cloud.TestSolrCloudWithKerberos.testKerberizedSolr

Error Message:
Error from server at http://127.0.0.1:41477/jv: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 401 </title> </head> <body> <h2>HTTP ERROR: 401</h2> <p>Problem accessing /jv/admin/collections. Reason: <pre>    Authentication required</pre></p> <hr /><i><small>Powered by Jetty://</small></i> </body> </html> 

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:41477/jv: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 401 </title>
</head>
<body>
<h2>HTTP ERROR: 401</h2>
<p>Problem accessing /jv/admin/collections. Reason:
<pre>    Authentication required</pre></p>
<hr /><i><small>Powered by Jetty://</small></i>
</body>
</html>

	at __randomizedtesting.SeedInfo.seed([CEC58863C9722182:65DC5EA48313E1B5]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:529)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:235)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:227)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:376)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:328)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1086)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:856)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:799)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:135)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:152)
	at org.apache.solr.cloud.TestSolrCloudWithKerberos.testKerberizedSolr(TestSolrCloudWithKerberos.java:157)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:960)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:935)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestSolrCloudWithKerberos

Error Message:
Some resources were not closed, shutdown, or released.

Stack Trace:
java.lang.AssertionError: Some resources were not closed, shutdown, or released.
	at __randomizedtesting.SeedInfo.seed([CEC58863C9722182]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:234)
	at sun.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:799)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestSolrCloudWithKerberos

Error Message:
5 threads leaked from SUITE scope at org.apache.solr.cloud.TestSolrCloudWithKerberos:     1) Thread[id=11044, name=zkCallback-1760-thread-1, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithKerberos]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=11080, name=zkCallback-1760-thread-3, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithKerberos]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)    3) Thread[id=11043, name=TEST-TestSolrCloudWithKerberos.testKerberizedSolr-seed#[CEC58863C9722182]-EventThread, state=WAITING, group=TGRP-TestSolrCloudWithKerberos]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)    4) Thread[id=11042, name=TEST-TestSolrCloudWithKerberos.testKerberizedSolr-seed#[CEC58863C9722182]-SendThread(127.0.0.1:57610), state=TIMED_WAITING, group=TGRP-TestSolrCloudWithKerberos]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)    5) Thread[id=11045, name=zkCallback-1760-thread-2, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithKerberos]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 5 threads leaked from SUITE scope at org.apache.solr.cloud.TestSolrCloudWithKerberos: 
   1) Thread[id=11044, name=zkCallback-1760-thread-1, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithKerberos]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=11080, name=zkCallback-1760-thread-3, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithKerberos]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
   3) Thread[id=11043, name=TEST-TestSolrCloudWithKerberos.testKerberizedSolr-seed#[CEC58863C9722182]-EventThread, state=WAITING, group=TGRP-TestSolrCloudWithKerberos]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
   4) Thread[id=11042, name=TEST-TestSolrCloudWithKerberos.testKerberizedSolr-seed#[CEC58863C9722182]-SendThread(127.0.0.1:57610), state=TIMED_WAITING, group=TGRP-TestSolrCloudWithKerberos]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
   5) Thread[id=11045, name=zkCallback-1760-thread-2, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithKerberos]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([CEC58863C9722182]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestSolrCloudWithKerberos

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=11044, name=zkCallback-1760-thread-1, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithKerberos]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=11080, name=zkCallback-1760-thread-3, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithKerberos]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)    3) Thread[id=11042, name=TEST-TestSolrCloudWithKerberos.testKerberizedSolr-seed#[CEC58863C9722182]-SendThread(127.0.0.1:57610), state=TIMED_WAITING, group=TGRP-TestSolrCloudWithKerberos]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)    4) Thread[id=11045, name=zkCallback-1760-thread-2, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithKerberos]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=11044, name=zkCallback-1760-thread-1, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithKerberos]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=11080, name=zkCallback-1760-thread-3, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithKerberos]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
   3) Thread[id=11042, name=TEST-TestSolrCloudWithKerberos.testKerberizedSolr-seed#[CEC58863C9722182]-SendThread(127.0.0.1:57610), state=TIMED_WAITING, group=TGRP-TestSolrCloudWithKerberos]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
   4) Thread[id=11045, name=zkCallback-1760-thread-2, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithKerberos]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([CEC58863C9722182]:0)




Build Log:
[...truncated 20811 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestSolrCloudWithKerberos
   [junit4]   2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos CEC58863C9722182-001/init-core-data-001
   [junit4]   2> 1635349 T10881 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /jv/
   [junit4]   2> 1640266 T10881 oadsc.DefaultDirectoryService.showSecurityWarnings WARN You didn't change the admin password of directory service instance 'DefaultKrbServer'.  Please update the admin password as soon as possible to prevent a possible security breach.
   [junit4]   2> 1640958 T10881 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 1640967 T10890 oasc.ZkTestServer$2$1.setClientPort client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1640968 T10890 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1641068 T10881 oasc.ZkTestServer.run start zk server on port:57610
   [junit4]   2> 1641077 T10897 oascc.ConnectionManager.process WARN zkClient received AuthFailed
   [junit4]   2> 1641088 T10901 oascc.ConnectionManager.process WARN zkClient received AuthFailed
   [junit4]   2> 1641093 T10881 oasc.AbstractZkTestCase.putConfig put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1641095 T10881 oasc.AbstractZkTestCase.putConfig put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 1641097 T10881 oasc.AbstractZkTestCase.putConfig put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1641099 T10881 oasc.AbstractZkTestCase.putConfig put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1641101 T10881 oasc.AbstractZkTestCase.putConfig put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1641104 T10881 oasc.AbstractZkTestCase.putConfig put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1641106 T10881 oasc.AbstractZkTestCase.putConfig put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1641107 T10881 oasc.AbstractZkTestCase.putConfig put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1641109 T10881 oasc.AbstractZkTestCase.putConfig put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1641111 T10881 oasc.AbstractZkTestCase.putConfig put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1641113 T10881 oasc.AbstractZkTestCase.putConfig put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1641124 T10905 oascc.ConnectionManager.process WARN zkClient received AuthFailed
   [junit4]   2> 1641432 T10881 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos CEC58863C9722182-001/control-001/cores/collection1
   [junit4]   2> 1641434 T10881 oejs.Server.doStart jetty-9.2.10.v20150310
   [junit4]   2> 1641435 T10881 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@36c2a8b{/jv,null,AVAILABLE}
   [junit4]   2> 1641436 T10881 oejs.AbstractConnector.doStart Started ServerConnector@edb95cf{HTTP/1.1}{127.0.0.1:41477}
   [junit4]   2> 1641436 T10881 oejs.Server.doStart Started @1643600ms
   [junit4]   2> 1641436 T10881 oascse.JettySolrRunner$1.lifeCycleStarted Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos CEC58863C9722182-001/tempDir-002/control/data, hostContext=/jv, hostPort=41477, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos CEC58863C9722182-001/control-001/cores}
   [junit4]   2> 1641437 T10881 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@c387f44
   [junit4]   2> 1641437 T10881 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos CEC58863C9722182-001/control-001/'
   [junit4]   2> 1641451 T10881 oasc.SolrXmlConfig.fromFile Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos CEC58863C9722182-001/control-001/solr.xml
   [junit4]   2> 1641455 T10881 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos CEC58863C9722182-001/control-001/cores
   [junit4]   2> 1641456 T10881 oasc.CoreContainer.<init> New CoreContainer 1591338152
   [junit4]   2> 1641456 T10881 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos CEC58863C9722182-001/control-001/]
   [junit4]   2> 1641456 T10881 oasc.CoreContainer.load loading shared library: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos CEC58863C9722182-001/control-001/lib
   [junit4]   2> 1641457 T10881 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos CEC58863C9722182-001/control-001/lib).
   [junit4]   2> 1641464 T10881 oashc.HttpShardHandlerFactory.init created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 1641465 T10881 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1641466 T10881 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1641466 T10881 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1641466 T10881 oasc.CoreContainer.load Node Name: 127.0.0.1
   [junit4]   2> 1641466 T10881 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:57610/solr
   [junit4]   2> 1641467 T10881 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1641468 T10920 oascc.ConnectionManager.process WARN zkClient received AuthFailed
   [junit4]   2> 1641470 T10924 n:127.0.0.1:41477_jv oascc.ConnectionManager.process WARN zkClient received AuthFailed
   [junit4]   2> 1641483 T10881 n:127.0.0.1:41477_jv oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41477_jv
   [junit4]   2> 1641486 T10881 n:127.0.0.1:41477_jv oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1641488 T10881 n:127.0.0.1:41477_jv oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:41477_jv
   [junit4]   2> 1641489 T10881 n:127.0.0.1:41477_jv oasc.Overseer.start Overseer (id=93872981088665604-127.0.0.1:41477_jv-n_0000000000) starting
   [junit4]   2> 1641495 T10881 n:127.0.0.1:41477_jv oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 1641496 T10927 n:127.0.0.1:41477_jv oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1641496 T10926 n:127.0.0.1:41477_jv oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1641500 T10881 n:127.0.0.1:41477_jv oasc.CoreContainer.initializeAuthenticationPlugin Authentication plugin class obtained from ZK: org.apache.solr.security.KerberosPlugin
   [junit4]   2> 1641511 T10881 n:127.0.0.1:41477_jv oass.KerberosPlugin.init Params: {token.valid=30, kerberos.principal=HTTP/127.0.0.1, cookie.domain=127.0.0.1, kerberos.keytab=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos CEC58863C9722182-001/tempDir-001/minikdc/keytabs, cookie.path=/, type=kerberos, kerberos.name.rules=DEFAULT}
   [junit4]   2> 1641533 T10881 n:127.0.0.1:41477_jv oasc.CoreContainer.initializeAuthenticationPlugin Reconfiguring the shard handler factory and update shard handler.
   [junit4]   2> 1641534 T10881 n:127.0.0.1:41477_jv oashc.HttpShardHandlerFactory.reconfigureHttpClient Reconfiguring the default client with: org.apache.solr.client.solrj.impl.Krb5HttpClientConfigurer@35cded9b
   [junit4]   2> 1641534 T10881 n:127.0.0.1:41477_jv oascsi.Krb5HttpClientConfigurer.configure Setting up SPNego auth with config: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos CEC58863C9722182-001/tempDir-001/minikdc/jaas-client.conf
   [junit4]   2> 1641564 T10881 n:127.0.0.1:41477_jv oasu.UpdateShardHandler.reconfigureHttpClient Reconfiguring the default client with: org.apache.solr.client.solrj.impl.Krb5HttpClientConfigurer@35cded9b
   [junit4]   2> 1641565 T10881 n:127.0.0.1:41477_jv oascsi.Krb5HttpClientConfigurer.configure Setting up SPNego auth with config: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos CEC58863C9722182-001/tempDir-001/minikdc/jaas-client.conf
   [junit4]   2> 1641566 T10881 n:127.0.0.1:41477_jv oasc.CorePropertiesLocator.discover Looking for core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos CEC58863C9722182-001/control-001/cores
   [junit4]   2> 1641567 T10881 n:127.0.0.1:41477_jv oasc.CoreDescriptor.<init> CORE DESCRIPTOR: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, instanceDir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos CEC58863C9722182-001/control-001/cores/collection1, collection=control_collection, absoluteInstDir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos CEC58863C9722182-001/control-001/cores/collection1/, coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 1641567 T10881 n:127.0.0.1:41477_jv oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos CEC58863C9722182-001/control-001/cores/collection1/
   [junit4]   2> 1641568 T10881 n:127.0.0.1:41477_jv oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1641572 T10930 n:127.0.0.1:41477_jv c:control_collection x:collection1 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 1641572 T10930 n:127.0.0.1:41477_jv c:control_collection x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1641573 T10924 n:127.0.0.1:41477_jv oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1641573 T10930 n:127.0.0.1:41477_jv oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1641573 T10926 n:127.0.0.1:41477_jv oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:41477/jv",
   [junit4]   2> 	  "node_name":"127.0.0.1:41477_jv",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"} current state version: 0
   [junit4]   2> 1641574 T10926 n:127.0.0.1:41477_jv oasco.ReplicaMutator.updateState Update state numShards=1 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:41477/jv",
   [junit4]   2> 	  "node_name":"127.0.0.1:41477_jv",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 1641574 T10926 n:127.0.0.1:41477_jv oasco.ClusterStateMutator.createCollection building a new cName: control_collection
   [junit4]   2> 1641574 T10926 n:127.0.0.1:41477_jv oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1642573 T10930 n:127.0.0.1:41477_jv oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1642574 T10930 n:127.0.0.1:41477_jv oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 1642574 T10930 n:127.0.0.1:41477_jv oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1642575 T10930 n:127.0.0.1:41477_jv oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos CEC58863C9722182-001/control-001/cores/collection1/'
   [junit4]   2> 1642591 T10930 n:127.0.0.1:41477_jv oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 1642595 T10930 n:127.0.0.1:41477_jv oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 1642599 T10930 n:127.0.0.1:41477_jv oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 1642607 T10930 n:127.0.0.1:41477_jv oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1642608 T10930 n:127.0.0.1:41477_jv oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 1642612 T10930 n:127.0.0.1:41477_jv oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1642712 T10930 n:127.0.0.1:41477_jv oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1642718 T10930 n:127.0.0.1:41477_jv oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1642719 T10930 n:127.0.0.1:41477_jv oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1642727 T10930 n:127.0.0.1:41477_jv oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1642729 T10930 n:127.0.0.1:41477_jv oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1642731 T10930 n:127.0.0.1:41477_jv oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1642731 T10930 n:127.0.0.1:41477_jv oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1642731 T10930 n:127.0.0.1:41477_jv oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1642732 T10930 n:127.0.0.1:41477_jv oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1642732 T10930 n:127.0.0.1:41477_jv oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1642732 T10930 n:127.0.0.1:41477_jv oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1642733 T10930 n:127.0.0.1:41477_jv oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 1642733 T10930 n:127.0.0.1:41477_jv x:collection1 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1642733 T10930 n:127.0.0.1:41477_jv x:collection1 oasc.SolrCore.<init> [[collection1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos CEC58863C9722182-001/control-001/cores/collection1/], dataDir=[null]
   [junit4]   2> 1642733 T10930 n:127.0.0.1:41477_jv x:collection1 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4fcd2d1d
   [junit4]   2> 1642734 T10930 n:127.0.0.1:41477_jv x:collection1 oasc.CachingDirectoryFactory.get return new directory for /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos CEC58863C9722182-001/control-001/cores/collection1/data
   [junit4]   2> 1642734 T10930 n:127.0.0.1:41477_jv x:collection1 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos CEC58863C9722182-001/control-001/cores/collection1/data/index/
   [junit4]   2> 1642734 T10930 n:127.0.0.1:41477_jv x:collection1 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos CEC58863C9722182-001/control-001/cores/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1642735 T10930 n:127.0.0.1:41477_jv x:collection1 oasc.CachingDirectoryFactory.get return new directory for /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos CEC58863C9722182-001/control-001/cores/collection1/data/index
   [junit4]   2> 1642736 T10930 n:127.0.0.1:41477_jv x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=46, maxMergeAtOnceExplicit=16, maxMergedSegmentMB=41.1806640625, floorSegmentMB=1.806640625, forceMergeDeletesPctAllowed=12.035944439818008, segmentsPerTier=10.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1642744 T10930 n:127.0.0.1:41477_jv x:collection1 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(SimpleFSDirectory@/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos CEC58863C9722182-001/index-SimpleFSDirectory-001 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6273c1af),segFN=segments_1,generation=1}
   [junit4]   2> 1642744 T10930 n:127.0.0.1:41477_jv x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1642751 T10930 n:127.0.0.1:41477_jv x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1642752 T10930 n:127.0.0.1:41477_jv x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1642752 T10930 n:127.0.0.1:41477_jv x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1642752 T10930 n:127.0.0.1:41477_jv x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1642752 T10930 n:127.0.0.1:41477_jv x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1642753 T10930 n:127.0.0.1:41477_jv x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1642753 T10930 n:127.0.0.1:41477_jv x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1642753 T10930 n:127.0.0.1:41477_jv x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1642753 T10930 n:127.0.0.1:41477_jv x:collection1 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1642756 T10930 n:127.0.0.1:41477_jv x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1642757 T10930 n:127.0.0.1:41477_jv x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1642758 T10930 n:127.0.0.1:41477_jv x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1642759 T10930 n:127.0.0.1:41477_jv x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1642761 T10930 n:127.0.0.1:41477_jv x:collection1 oasc.RequestHandlers.initHandlersFromConfig Registered paths: /admin/mbeans,standard,/update/csv,/update/json/docs,/admin/luke,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/schema,/admin/plugins,/admin/logging,/update/json,/admin/threads,/admin/ping,/update,/admin/file
   [junit4]   2> 1642762 T10930 n:127.0.0.1:41477_jv x:collection1 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 1642763 T10930 n:127.0.0.1:41477_jv x:collection1 oasu.UpdateHandler.<init> Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1642763 T10930 n:127.0.0.1:41477_jv x:collection1 oasu.UpdateLog.init Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1642764 T10930 n:127.0.0.1:41477_jv x:collection1 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1642764 T10930 n:127.0.0.1:41477_jv x:collection1 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1642765 T10930 n:127.0.0.1:41477_jv x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=38, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=1.1103515625, noCFSRatio=0.2491958114430644]
   [junit4]   2> 1642766 T10930 n:127.0.0.1:41477_jv x:collection1 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(SimpleFSDirectory@/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos CEC58863C9722182-001/index-SimpleFSDirectory-001 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6273c1af),segFN=segments_1,generation=1}
   [junit4]   2> 1642766 T10930 n:127.0.0.1:41477_jv x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1642767 T10930 n:127.0.0.1:41477_jv x:collection1 oass.SolrIndexSearcher.<init> Opening Searcher@5d35578e[collection1] main
   [junit4]   2> 1642767 T10930 n:127.0.0.1:41477_jv x:collection1 oasu.UpdateLog.onFirstSearcher On first searcher opened, looking up max value of version field
   [junit4]   2> 1642767 T10930 n:127.0.0.1:41477_jv x:collection1 oasu.VersionInfo.getMaxVersionFromIndex Refreshing highest value of _version_ for 65536 version buckets from index
   [junit4]   2> 1642767 T10930 n:127.0.0.1:41477_jv x:collection1 oasu.VersionInfo.getMaxVersionFromIndex WARN No terms found for _version_, cannot seed version bucket highest value from index
   [junit4]   2> 1642767 T10930 n:127.0.0.1:41477_jv x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion WARN Could not find max version in index or recent updates, using new clock 1501967699298746368
   [junit4]   2> 1642770 T10930 n:127.0.0.1:41477_jv x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion Took 3 ms to seed version buckets with highest version 1501967699298746368
   [junit4]   2> 1642771 T10930 n:127.0.0.1:41477_jv x:collection1 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1642771 T10930 n:127.0.0.1:41477_jv x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1642771 T10930 n:127.0.0.1:41477_jv x:collection1 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 1642771 T10930 n:127.0.0.1:41477_jv x:collection1 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1642772 T10930 n:127.0.0.1:41477_jv x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1642772 T10930 n:127.0.0.1:41477_jv x:collection1 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1642772 T10930 n:127.0.0.1:41477_jv x:collection1 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 1642772 T10930 n:127.0.0.1:41477_jv x:collection1 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 1642773 T10930 n:127.0.0.1:41477_jv x:collection1 oasc.ZkController.getConfDirListeners watch zkdir /configs/conf1
   [junit4]   2> 1642773 T10931 n:127.0.0.1:41477_jv x:collection1 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5d35578e[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1642773 T10930 n:127.0.0.1:41477_jv x:collection1 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1642774 T10934 n:127.0.0.1:41477_jv c:control_collection s:shard1 x:collection1 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:41477/jv collection:control_collection shard:shard1
   [junit4]   2> 1642774 T10881 n:127.0.0.1:41477_jv oascsi.Krb5HttpClientConfigurer.configure Setting up SPNego auth with config: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos CEC58863C9722182-001/tempDir-001/minikdc/jaas-client.conf
   [junit4]   2> 1642775 T10881 n:127.0.0.1:41477_jv oass.SolrDispatchFilter.init user.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1
   [junit4]   2> 1642775 T10881 n:127.0.0.1:41477_jv oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1642776 T10881 oascsi.Krb5HttpClientConfigurer.configure Setting up SPNego auth with config: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos CEC58863C9722182-001/tempDir-001/minikdc/jaas-client.conf
   [junit4]   2> 1642777 T10881 oascsi.Krb5HttpClientConfigurer.configure Setting up SPNego auth with config: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos CEC58863C9722182-001/tempDir-001/minikdc/jaas-client.conf
   [junit4]   2> 1642782 T10937 oascc.ConnectionManager.process WARN zkClient received AuthFailed
   [junit4]   2> 1642783 T10934 n:127.0.0.1:41477_jv c:control_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 1642787 T10924 n:127.0.0.1:41477_jv oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1642788 T10934 n:127.0.0.1:41477_jv c:control_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1642788 T10934 n:127.0.0.1:41477_jv c:control_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C1463 name=collection1 org.apache.solr.core.SolrCore@2b1c6732 url=http://127.0.0.1:41477/jv/collection1 node=127.0.0.1:41477_jv C1463_STATE=coll:control_collection core:collection1 props:{core=collection1, base_url=http://127.0.0.1:41477/jv, node_name=127.0.0.1:41477_jv, state=down}
   [junit4]   2> 1642788 T10934 n:127.0.0.1:41477_jv c:control_collection s:shard1 x:collection1 C1463 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:41477/jv/collection1/
   [junit4]   2> 1642788 T10934 n:127.0.0.1:41477_jv c:control_collection s:shard1 x:collection1 C1463 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1642788 T10934 n:127.0.0.1:41477_jv c:control_collection s:shard1 x:collection1 C1463 oasc.SyncStrategy.syncToMe http://127.0.0.1:41477/jv/collection1/ has no replicas
   [junit4]   2> 1642788 T10934 n:127.0.0.1:41477_jv c:control_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:41477/jv/collection1/ shard1
   [junit4]   2> 1642789 T10926 n:127.0.0.1:41477_jv oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection"} current state version: 1
   [junit4]   2> 1642791 T10881 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1642795 T10924 n:127.0.0.1:41477_jv oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1642796 T10926 n:127.0.0.1:41477_jv oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:41477/jv",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "state":"active"} current state version: 1
   [junit4]   2> 1642943 T10934 n:127.0.0.1:41477_jv c:control_collection s:shard1 x:collection1 oasc.ZkController.register We are http://127.0.0.1:41477/jv/collection1/ and leader is http://127.0.0.1:41477/jv/collection1/
   [junit4]   2> 1642943 T10934 n:127.0.0.1:41477_jv c:control_collection s:shard1 x:collection1 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:41477/jv
   [junit4]   2> 1642943 T10934 n:127.0.0.1:41477_jv c:control_collection s:shard1 x:collection1 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1642944 T10934 n:127.0.0.1:41477_jv c:control_collection s:shard1 x:collection1 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 1642944 T10934 n:127.0.0.1:41477_jv c:control_collection s:shard1 x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1642945 T10924 n:127.0.0.1:41477_jv oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1642946 T10926 n:127.0.0.1:41477_jv oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:41477/jv",
   [junit4]   2> 	  "node_name":"127.0.0.1:41477_jv",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"} current state version: 2
   [junit4]   2> 1642946 T10926 n:127.0.0.1:41477_jv oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:41477/jv",
   [junit4]   2> 	  "node_name":"127.0.0.1:41477_jv",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 1643085 T10881 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos CEC58863C9722182-001/shard-1-001/cores/collection1
   [junit4]   2> 1643086 T10881 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos CEC58863C9722182-001/shard-1-001
   [junit4]   2> 1643087 T10881 oejs.Server.doStart jetty-9.2.10.v20150310
   [junit4]   2> 1643089 T10881 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@4e1e8ebc{/jv,null,AVAILABLE}
   [junit4]   2> 1643090 T10881 oejs.AbstractConnector.doStart Started ServerConnector@3d324caf{HTTP/1.1}{127.0.0.1:34557}
   [junit4]   2> 1643090 T10881 oejs.Server.doStart Started @1645254ms
   [junit4]   2> 1643090 T10881 oascse.JettySolrRunner$1.lifeCycleStarted Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos CEC58863C9722182-001/tempDir-002/jetty1, solrconfig=solrconfig.xml, hostContext=/jv, hostPort=34557, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos CEC58863C9722182-001/shard-1-001/cores}
   [junit4]   2> 1643091 T10881 oascsi.Krb5HttpClientConfigurer.configure Setting up SPNego auth with config: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos CEC58863C9722182-001/tempDir-001/minikdc/jaas-client.conf
   [junit4]   2> 1643091 T10881 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@c387f44
   [junit4]   2> 1643091 T10881 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos CEC58863C9722182-001/shard-1-001/'
   [junit4]   2> 1643105 T10881 oasc.SolrXmlConfig.fromFile Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos CEC58863C9722182-001/shard-1-001/solr.xml
   [junit4]   2> 1643110 T10881 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos CEC58863C9722182-001/shard-1-001/cores
   [junit4]   2> 1643110 T10881 oasc.CoreContainer.<init> New CoreContainer 1091447886
   [junit4]   2> 1643111 T10881 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos CEC58863C9722182-001/shard-1-001/]
   [junit4]   2> 1643111 T10881 oasc.CoreContainer.load loading shared library: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos CEC58863C9722182-001/shard-1-001/lib
   [junit4]   2> 1643111 T10881 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos CEC58863C9722182-001/shard-1-001/lib).
   [junit4]   2> 1643118 T10881 oashc.HttpShardHandlerFactory.init created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 1643119 T10881 oascsi.Krb5HttpClientConfigurer.configure Setting up SPNego auth with config: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos CEC58863C9722182-001/tempDir-001/minikdc/jaas-client.conf
   [junit4]   2> 1643120 T10881 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1643120 T10881 oascsi.Krb5HttpClientConfigurer.configure Setting up SPNego auth with config: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos CEC58863C9722182-001/tempDir-001/minikdc/jaas-client.conf
   [junit4]   2> 1643121 T10881 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1643121 T10881 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1643121 T10881 oasc.CoreContainer.load Node Name: 127.0.0.1
   [junit4]   2> 1643121 T10881 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:57610/solr
   [junit4]   2> 1643122 T10881 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1643123 T10952 oascc.ConnectionManager.process WARN zkClient received AuthFailed
   [junit4]   2> 1643126 T10955 n:127.0.0.1:34557_jv oascc.ConnectionManager.process WARN zkClient received AuthFailed
   [junit4]   2> 1644134 T10881 n:127.0.0.1:34557_jv oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34557_jv
   [junit4]   2> 1644138 T10881 n:127.0.0.1:34557_jv oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1644140 T10881 n:127.0.0.1:34557_jv oasc.CoreContainer.initializeAuthenticationPlugin Authentication plugin class obtained from ZK: org.apache.solr.security.KerberosPlugin
   [junit4]   2> 1644140 T10881 n:127.0.0.1:34557_jv oass.KerberosPlugin.init Params: {token.valid=30, kerberos.principal=HTTP/127.0.0.1, cookie.domain=127.0.0.1, kerberos.keytab=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos CEC58863C9722182-001/tempDir-001/minikdc/keytabs, cookie.path=/, type=kerberos, kerberos.name.rules=DEFAULT}
   [junit4]   2> 1644142 T10881 n:127.0.0.1:34557_jv oasc.CoreContainer.initializeAuthenticationPlugin Reconfiguring the shard handler factory and update shard handler.
   [junit4]   2> 1644143 T10881 n:127.0.0.1:34557_jv oashc.HttpShardHandlerFactory.reconfigureHttpClient Reconfiguring the default client with: org.apache.solr.client.solrj.impl.Krb5HttpClientConfigurer@5966fd90
   [junit4]   2> 1644143 T10881 n:127.0.0.1:34557_jv oascsi.Krb5HttpClientConfigurer.configure Setting up SPNego auth with config: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos CEC58863C9722182-001/tempDir-001/minikdc/jaas-client.conf
   [junit4]   2> 1644143 T10881 n:127.0.0.1:34557_jv oasu.UpdateShardHandler.reconfigureHttpClient Reconfiguring the default client with: org.apache.solr.client.solrj.impl.Krb5HttpClientConfigurer@5966fd90
   [junit4]   2> 1644144 T10881 n:127.0.0.1:34557_jv oascsi.Krb5HttpClientConfigurer.configure Setting up SPNego auth with config: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos CEC58863C9722182-001/tempDir-001/minikdc/jaas-client.conf
   [junit4]   2> 1644145 T10881 n:127.0.0.1:34557_jv oasc.CorePropertiesLocator.discover Looking for core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos CEC58863C9722182-001/shard-1-001/cores
   [junit4]   2> 1644146 T10881 n:127.0.0.1:34557_jv oasc.CoreDescriptor.<init> CORE DESCRIPTOR: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, instanceDir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos CEC58863C9722182-001/shard-1-001/cores/collection1, collection=collection1, absoluteInstDir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos CEC58863C9722182-001/shard-1-001/cores/collection1/, coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 1644146 T10881 n:127.0.0.1:34557_jv oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos CEC58863C9722182-001/shard-1-001/cores/collection1/
   [junit4]   2> 1644147 T10881 n:127.0.0.1:34557_jv oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1644147 T10957 n:127.0.0.1:34557_jv c:collection1 x:collection1 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 1644148 T10957 n:127.0.0.1:34557_jv c:collection1 x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1644148 T10924 n:127.0.0.1:41477_jv oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1644148 T10957 n:127.0.0.1:34557_jv oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1644149 T10926 n:127.0.0.1:41477_jv oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:34557/jv",
   [junit4]   2> 	  "node_name":"127.0.0.1:34557_jv",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"} current state version: 3
   [junit4]   2> 1644150 T10926 n:127.0.0.1:41477_jv oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:34557/jv",
   [junit4]   2> 	  "node_name":"127.0.0.1:34557_jv",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 1644150 T10926 n:127.0.0.1:41477_jv oasco.ClusterStateMutator.createCollection building a new cName: collection1
   [junit4]   2> 1644150 T10926 n:127.0.0.1:41477_jv oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 1645149 T10957 n:127.0.0.1:34557_jv oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1645149 T10957 n:127.0.0.1:34557_jv oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 1645150 T10957 n:127.0.0.1:34557_jv oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1645151 T10957 n:127.0.0.1:34557_jv oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos CEC58863C9722182-001/shard-1-001/cores/collection1/'
   [junit4]   2> 1645166 T10957 n:127.0.0.1:34557_jv oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 1645171 T10957 n:127.0.0.1:34557_jv oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 1645175 T10957 n:127.0.0.1:34557_jv oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 1645184 T10957 n:127.0.0.1:34557_jv oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1645185 T10957 n:127.0.0.1:34557_jv oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 1645190 T10957 n:127.0.0.1:34557_jv oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1645305 T10957 n:127.0.0.1:34557_jv oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1645313 T10957 n:127.0.0.1:34557_jv oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1645314 T10957 n:127.0.0.1:34557_jv oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1645335 T10957 n:127.0.0.1:34557_jv oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1645338 T10957 n:127.0.0.1:34557_jv oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1645339 T10957 n:127.0.0.1:34557_jv oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1645343 T10957 n:127.0.0.1:34557_jv oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1645344 T10957 n:127.0.0.1:34557_jv oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1645344 T10957 n:127.0.0.1:34557_jv oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1645346 T10957 n:127.0.0.1:34557_jv oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1645346 T10957 n:127.0.0.1:34557_jv oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1645346 T10957 n:127.0.0.1:34557_jv oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 1645346 T10957 n:127.0.0.1:34557_jv x:collection1 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1645347 T10957 n:127.0.0.1:34557_jv x:collection1 oasc.SolrCore.<init> [[collection1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos CEC58863C9722182-001/shard-1-001/cores/collection1/], dataDir=[null]
   [junit4]   2> 1645347 T10957 n:127.0.0.1:34557_jv x:collection1 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4fcd2d1d
   [junit4]   2> 1645348 T10957 n:127.0.0.1:34557_jv x:collection1 oasc.CachingDirectoryFactory.get return new directory for /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos CEC58863C9722182-001/shard-1-001/cores/collection1/data
   [junit4]   2> 1645348 T10957 n:127.0.0.1:34557_jv x:collection1 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos CEC58863C9722182-001/shard-1-001/cores/collection1/data/index/
   [junit4]   2> 1645348 T10957 n:127.0.0.1:34557_jv x:collection1 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos CEC58863C9722182-001/shard-1-001/cores/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1645349 T10957 n:127.0.0.1:34557_jv x:collection1 oasc.CachingDirectoryFactory.get return new directory for /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos CEC58863C9722182-001/shard-1-001/cores/collection1/data/index
   [junit4]   2> 1645349 T10957 n:127.0.0.1:34557_jv x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=46, maxMergeAtOnceExplicit=16, maxMergedSegmentMB=41.1806640625, floorSegmentMB=1.806640625, forceMergeDeletesPctAllowed=12.035944439818008, segmentsPerTier=10.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1645369 T10957 n:127.0.0.1:34557_jv x:collection1 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(SimpleFSDirectory@/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos CEC58863C9722182-001/index-SimpleFSDirectory-002 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6273c1af),segFN=segments_1,generation=1}
   [junit4]   2> 1645370 T10957 n:127.0.0.1:34557_jv x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1645374 T10957 n:127.0.0.1:34557_jv x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1645375 T10957 n:127.0.0.1:34557_jv x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1645375 T10957 n:127.0.0.1:34557_jv x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1645375 T10957 n:127.0.0.1:34557_jv x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1645375 T10957 n:127.0.0.1:34557_jv x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1645375 T10957 n:127.0.0.1:34557_jv x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1645376 T10957 n:127.0.0.1:34557_jv x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1645376 T10957 n:127.0.0.1:34557_jv x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1645376 T10957 n:127.0.0.1:34557_jv x:collection1 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1645378 T10957 n:127.0.0.1:34557_jv x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1645380 T10957 n:127.0.0.1:34557_jv x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1645381 T10957 n:127.0.0.1:34557_jv x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1645382 T10957 n:127.0.0.1:34557_jv x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1645389 T10957 n:127.0.0.1:34557_jv x:collection1 oasc.RequestHandlers.initHandlersFromConfig Registered paths: /admin/mbeans,standard,/update/csv,/update/json/docs,/admin/luke,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/schema,/admin/plugins,/admin/logging,/update/json,/admin/threads,/admin/ping,/update,/admin/file
   [junit4]   2> 1645390 T10957 n:127.0.0.1:34557_jv x:collection1 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 1645392 T10957 n:127.0.0.1:34557_jv x:collection1 oasu.UpdateHandler.<init> Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1645392 T10957 n:127.0.0.1:34557_jv x:collection1 oasu.UpdateLog.init Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1645393 T10957 n:127.0.0.1:34557_jv x:collection1 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1645393 T10957 n:127.0.0.1:34557_jv x:collection1 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1645400 T10957 n:127.0.0.1:34557_jv x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=38, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=1.1103515625, noCFSRatio=0.2491958114430644]
   [junit4]   2> 1645402 T10957 n:127.0.0.1:34557_jv x:collection1 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(SimpleFSDirectory@/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos CEC58863C9722182-001/index-SimpleFSDirectory-002 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6273c1af),segFN=segments_1,generation=1}
   [junit4]   2> 1645402 T10957 n:127.0.0.1:34557_jv x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1645402 T10957 n:127.0.0.1:34557_jv x:collection1 oass.SolrIndexSearcher.<init> Opening Searcher@3928bacb[collection1] main
   [junit4]   2> 1645403 T10957 n:127.0.0.1:34557_jv x:collection1 oasu.UpdateLog.onFirstSearcher On first searcher opened, looking up max value of version field
   [junit4]   2> 1645403 T10957 n:127.0.0.1:34557_jv x:collection1 oasu.VersionInfo.getMaxVersionFromIndex Refreshing highest value of _version_ for 65536 version buckets from index
   [junit4]   2> 1645403 T10957 n:127.0.0.1:34557_jv x:collection1 oasu.VersionInfo.getMaxVersionFromIndex WARN No terms found for _version_, cannot seed version bucket highest value from index
   [junit4]   2> 1645403 T10957 n:127.0.0.1:34557_jv x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion WARN Could not find max version in index or recent updates, using new clock 1501967702062792704
   [junit4]   2> 1645406 T10957 n:127.0.0.1:34557_jv x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion Took 3 ms to seed version buckets with highest version 1501967702062792704
   [junit4]   2> 1645407 T10957 n:127.0.0.1:34557_jv x:collection1 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1645408 T10957 n:127.0.0.1:34557_jv x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1645408 T10957 n:127.0.0.1:34557_jv x:collection1 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 1645409 T10957 n:127.0.0.1:34557_jv x:collection1 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1645409 T10957 n:127.0.0.1:34557_jv x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1645410 T10957 n:127.0.0.1:34557_jv x:collection1 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1645410 T10957 n:127.0.0.1:34557_jv x:collection1 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 1645410 T10957 n:127.0.0.1:34557_jv x:collection1 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 1645411 T10958 n:127.0.0.1:34557_jv x:collection1 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3928bacb[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1645411 T10957 n:127.0.0.1:34557_jv x:collection1 oasc.ZkController.getConfDirListeners watch zkdir /configs/conf1
   [junit4]   2> 1645412 T10957 n:127.0.0.1:34557_jv x:collection1 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1645413 T10961 n:127.0.0.1:34557_jv c:collection1 s:shard2 x:collection1 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:34557/jv collection:collection1 shard:shard2
   [junit4]   2> 1645413 T10881 n:127.0.0.1:34557_jv oascsi.Krb5HttpClientConfigurer.configure Setting up SPNego auth with config: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos CEC58863C9722182-001/tempDir-001/minikdc/jaas-client.conf
   [junit4]   2> 1645414 T10881 n:127.0.0.1:34557_jv oass.SolrDispatchFilter.init user.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1
   [junit4]   2> 1645414 T10881 n:127.0.0.1:34557_jv oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1645415 T10881 oascsi.Krb5HttpClientConfigurer.configure Setting up SPNego auth with config: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos CEC58863C9722182-001/tempDir-001/minikdc/jaas-client.conf
   [junit4]   2> 1645420 T10961 n:127.0.0.1:34557_jv c:collection1 s:shard2 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 1645421 T10924 n:127.0.0.1:41477_jv oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1645422 T10961 n:127.0.0.1:34557_jv c:collection1 s:shard2 x:collection1 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1645422 T10961 n:127.0.0.1:34557_jv c:collection1 s:shard2 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 1645422 T10926 n:127.0.0.1:41477_jv oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1"} current state version: 4
   [junit4]   2> ASYNC  NEW_CORE C1464 name=collection1 org.apache.solr.core.SolrCore@75f86748 url=http://127.0.0.1:34557/jv/collection1 node=127.0.0.1:34557_jv C1464_STATE=coll:collection1 core:collection1 props:{core=collection1, base_url=http://127.0.0.1:34557/jv, node_name=127.0.0.1:34557_jv, state=down}
   [junit4]   2> 1645422 T10961 n:127.0.0.1:34557_jv c:collection1 s:shard2 x:collection1 C1464 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:34557/jv/collection1/
   [junit4]   2> 1645423 T10961 n:127.0.0.1:34557_jv c:collection1 s:shard2 x:collection1 C1464 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1645423 T10961 n:127.0.0.1:34557_jv c:collection1 s:shard2 x:collection1 C1464 oasc.SyncStrategy.syncToMe http://127.0.0.1:34557/jv/collection1/ has no replicas
   [junit4]   2> 1645423 T10961 n:127.0.0.1:34557_jv c:collection1 s:shard2 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:34557/jv/collection1/ shard2
   [junit4]   2> 1645427 T10924 n:127.0.0.1:41477_jv oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1645427 T10926 n:127.0.0.1:41477_jv oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:34557/jv",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "state":"active"} current state version: 4
   [junit4]   2> 1645578 T10961 n:127.0.0.1:34557_jv c:collection1 s:shard2 x:collection1 oasc.ZkController.register We are http://127.0.0.1:34557/jv/collection1/ and leader is http://127.0.0.1:34557/jv/collection1/
   [junit4]   2> 1645579 T10961 n:127.0.0.1:34557_jv c:collection1 s:shard2 x:collect

[...truncated too long message...]

unit4]    > 	at __randomizedtesting.SeedInfo.seed([CEC58863C9722182]:0)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:234)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: 5 threads leaked from SUITE scope at org.apache.solr.cloud.TestSolrCloudWithKerberos: 
   [junit4]    >    1) Thread[id=11044, name=zkCallback-1760-thread-1, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithKerberos]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >    2) Thread[id=11080, name=zkCallback-1760-thread-3, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithKerberos]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >    3) Thread[id=11043, name=TEST-TestSolrCloudWithKerberos.testKerberizedSolr-seed#[CEC58863C9722182]-EventThread, state=WAITING, group=TGRP-TestSolrCloudWithKerberos]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
   [junit4]    >    4) Thread[id=11042, name=TEST-TestSolrCloudWithKerberos.testKerberizedSolr-seed#[CEC58863C9722182]-SendThread(127.0.0.1:57610), state=TIMED_WAITING, group=TGRP-TestSolrCloudWithKerberos]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
   [junit4]    >    5) Thread[id=11045, name=zkCallback-1760-thread-2, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithKerberos]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([CEC58863C9722182]:0)Throwable #3: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=11044, name=zkCallback-1760-thread-1, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithKerberos]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >    2) Thread[id=11080, name=zkCallback-1760-thread-3, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithKerberos]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >    3) Thread[id=11042, name=TEST-TestSolrCloudWithKerberos.testKerberizedSolr-seed#[CEC58863C9722182]-SendThread(127.0.0.1:57610), state=TIMED_WAITING, group=TGRP-TestSolrCloudWithKerberos]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
   [junit4]    >    4) Thread[id=11045, name=zkCallback-1760-thread-2, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithKerberos]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([CEC58863C9722182]:0)
   [junit4] Completed [424/502] on J1 in 52.86s, 1 test, 1 failure, 3 errors <<< FAILURES!

[...truncated 240 lines...]
BUILD FAILED
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/build.xml:526: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/build.xml:474: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/build.xml:61: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/extra-targets.xml:39: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build.xml:229: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/common-build.xml:512: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/lucene/common-build.xml:1415: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/lucene/common-build.xml:973: There were test failures: 502 suites, 1986 tests, 3 suite-level errors, 1 error, 358 ignored (25 assumptions)

Total time: 51 minutes 53 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-Tests-trunk-Java8 - Build # 6 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-trunk-Java8/6/

No tests ran.

Build Log:
[...truncated 54 lines...]
A         soERROR: Failed to check out http://svn.apache.org/repos/asf/lucene/dev/trunk
org.tmatesoft.svn.core.SVNException: svn: E204899: Cannot write entries file '/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/.svn/entries': No space left on device
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:85)
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:69)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNAdminArea14.saveEntries(SVNAdminArea14.java:630)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNLogRunner.logFailed(SVNLogRunner.java:527)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNAdminArea.runLogs(SVNAdminArea.java:594)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNAdminArea.runLogs(SVNAdminArea.java:569)
	at org.tmatesoft.svn.core.internal.wc.SVNUpdateEditor$SVNDirectoryInfo.cleanup(SVNUpdateEditor.java:2341)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNWCAccess.doClose(SVNWCAccess.java:482)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNWCAccess.doClose(SVNWCAccess.java:463)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNWCAccess.close(SVNWCAccess.java:346)
	at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.update(SVNUpdateClient16.java:524)
	at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.doCheckout(SVNUpdateClient16.java:915)
	at org.tmatesoft.svn.core.internal.wc2.old.SvnOldCheckout.run(SvnOldCheckout.java:19)
	at org.tmatesoft.svn.core.internal.wc2.old.SvnOldCheckout.run(SvnOldCheckout.java:8)
	at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:20)
	at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1238)
	at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)
	at org.tmatesoft.svn.core.wc.SVNUpdateClient.doCheckout(SVNUpdateClient.java:777)
	at hudson.scm.subversion.CheckoutUpdater$1.perform(CheckoutUpdater.java:99)
	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:161)
	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:169)
	at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:133)
	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:161)
	at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:1030)
	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:1011)
	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:987)
	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2474)
	at hudson.remoting.UserRequest.perform(UserRequest.java:118)
	at hudson.remoting.UserRequest.perform(UserRequest.java:48)
	at hudson.remoting.Request$2.run(Request.java:328)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: No space left on device
	at java.io.FileOutputStream.writeBytes(Native Method)
	at java.io.FileOutputStream.write(FileOutputStream.java:345)
	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
	at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
	at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:297)
	at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141)
	at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNAdminArea14.writeEntry(SVNAdminArea14.java:1426)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNAdminArea14.writeEntries(SVNAdminArea14.java:1130)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNAdminArea14.saveEntries(SVNAdminArea14.java:625)
	... 32 more
lr/contrib/uima/src/java/org/apache
A         solr/contrib/uima/src/java/org/apache/solr
A         solr/contrib/uima/src/java/org/apache/solr/uima
A         solr/contrib/uima/src/java/org/apache/solr/uima/processor
AU        solr/contrib/uima/src/java/org/apache/solr/uima/processor/SolrUIMAConfiguration.java
AU        solr/contrib/uima/src/java/org/apache/solr/uima/processor/UIMAToSolrMapper.java
AU        solr/contrib/uima/src/java/org/apache/solr/uima/processor/package-info.java
AU        solr/contrib/uima/src/java/org/apache/solr/uima/processor/SolrUIMAConfigurationReader.java
AU        solr/contrib/uima/src/java/org/apache/solr/uima/processor/UIMAUpdateRequestProcessor.java
AU        solr/contrib/uima/src/java/org/apache/solr/uima/processor/FieldMappingException.java
AU        solr/contrib/uima/src/java/org/apache/solr/uima/processor/UIMAUpdateRequestProcessorFactory.java
AU        solr/contrib/uima/src/java/overview.html
A         solr/contrib/uima/src/resources
java.io.IOException: remote file operation failed: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8 at hudson.remoting.Channel@412a89f6:lucene
	at hudson.FilePath.act(FilePath.java:916)
	at hudson.FilePath.act(FilePath.java:893)
	at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:936)
	at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:871)
	at hudson.model.AbstractProject.checkout(AbstractProject.java:1252)
	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:615)
	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:524)
	at hudson.model.Run.execute(Run.java:1706)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:88)
	at hudson.model.Executor.run(Executor.java:232)
Caused by: java.io.IOException: Failed to check out http://svn.apache.org/repos/asf/lucene/dev/trunk
	at hudson.scm.subversion.CheckoutUpdater$1.perform(CheckoutUpdater.java:110)
	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:161)
	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:169)
	at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:133)
	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:161)
	at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:1030)
	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:1011)
	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:987)
	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2474)
	at hudson.remoting.UserRequest.perform(UserRequest.java:118)
	at hudson.remoting.UserRequest.perform(UserRequest.java:48)
	at hudson.remoting.Request$2.run(Request.java:328)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.tmatesoft.svn.core.SVNException: svn: E204899: Cannot write entries file '/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/.svn/entries': No space left on device
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:85)
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:69)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNAdminArea14.saveEntries(SVNAdminArea14.java:630)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNLogRunner.logFailed(SVNLogRunner.java:527)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNAdminArea.runLogs(SVNAdminArea.java:594)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNAdminArea.runLogs(SVNAdminArea.java:569)
	at org.tmatesoft.svn.core.internal.wc.SVNUpdateEditor$SVNDirectoryInfo.cleanup(SVNUpdateEditor.java:2341)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNWCAccess.doClose(SVNWCAccess.java:482)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNWCAccess.doClose(SVNWCAccess.java:463)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNWCAccess.close(SVNWCAccess.java:346)
	at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.update(SVNUpdateClient16.java:524)
	at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.doCheckout(SVNUpdateClient16.java:915)
	at org.tmatesoft.svn.core.internal.wc2.old.SvnOldCheckout.run(SvnOldCheckout.java:19)
	at org.tmatesoft.svn.core.internal.wc2.old.SvnOldCheckout.run(SvnOldCheckout.java:8)
	at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:20)
	at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1238)
	at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)
	at org.tmatesoft.svn.core.wc.SVNUpdateClient.doCheckout(SVNUpdateClient.java:777)
	at hudson.scm.subversion.CheckoutUpdater$1.perform(CheckoutUpdater.java:99)
	... 16 more
Caused by: java.io.IOException: No space left on device
	at java.io.FileOutputStream.writeBytes(Native Method)
	at java.io.FileOutputStream.write(FileOutputStream.java:345)
	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
	at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
	at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:297)
	at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141)
	at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNAdminArea14.writeEntry(SVNAdminArea14.java:1426)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNAdminArea14.writeEntries(SVNAdminArea14.java:1130)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNAdminArea14.saveEntries(SVNAdminArea14.java:625)
	... 32 more
Retrying after 10 seconds
Cleaning up /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/.
Deleting /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build.xml
Updating http://svn.apache.org/repos/asf/lucene/dev/trunk at revision '2015-05-23T05:29:10.099 -0400'
Workspace appear to be locked, so getting a fresh workspace
Cleaning local Directory .
Checking out http://svn.apache.org/repos/asf/lucene/dev/trunk at revision '2015-05-23T05:29:10.099 -0400'
A         solr
A         solr/cloud-dev
AU        solr/cloud-dev/control.sh
AU        solr/cloud-dev/solrcloud-start.sh
AU        solr/cloud-dev/clean.sh
AU        solr/cloud-dev/cli-test-solrcloud-start.sh
AU        solr/cloud-dev/stop.sh
AU        solr/cloud-dev/functions.sh
AU        solr/cloud-dev/solrcloud-start-existing.sh
AU        solr/cloud-dev/example1.sh
AU        solr/cloud-dev/example2.sh
AU        solr/cloud-dev/example3.sh
AU        solr/common-build.xml
AU        solr/CHANGES.txt
A         solr/scripts
AU        solr/scripts/snappuller-enable
AU        solr/scripts/README.txt
AU        solr/scripts/snapinstaller
AU        solr/scripts/abc
AU        solr/scripts/rsyncd-stop
AU        solr/scripts/snappuller
AU        solr/scripts/rsyncd-enable
AU        solr/scripts/snappuller-disable
AU        solr/scripts/commit
AU        solr/scripts/snapcleaner
AU        solr/scripts/rsyncd-start
AU        solr/scripts/backupcleaner
AU        solr/scripts/optimize
AU        solr/scripts/abo
AU        solr/scripts/scripts-util
AU        solr/scripts/rsyncd-disable
AU        solr/scripts/snapshooter
AU        solr/scripts/backup
A         solr/core
AU        solr/core/ivy.xml
A         solr/core/lib
A         solr/core/src
A         solr/core/src/test
A         solr/core/src/test/org
A         solr/core/src/test/org/apache
A         solr/core/src/test/org/apache/solr
A         solr/core/src/test/org/apache/solr/request
AU        solr/core/src/test/org/apache/solr/request/SimpleFacetsTest.java
AU        solr/core/src/test/org/apache/solr/request/TestBinaryResponseWriter.java
AU        solr/core/src/test/org/apache/solr/request/TestFaceting.java
A         solr/core/src/test/org/apache/solr/request/macro
AU        solr/core/src/test/org/apache/solr/request/macro/TestMacros.java
AU        solr/core/src/test/org/apache/solr/request/TestIntervalFaceting.java
AU        solr/core/src/test/org/apache/solr/request/TestRemoteStreaming.java
AU        solr/core/src/test/org/apache/solr/request/JSONWriterTest.java
AU        soERROR: Failed to check out http://svn.apache.org/repos/asf/lucene/dev/trunk
org.tmatesoft.svn.core.SVNException: svn: E204899: Cannot write entries file '/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/.svn/entries': No space left on device
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:85)
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:69)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNAdminArea14.saveEntries(SVNAdminArea14.java:630)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNLogRunner.logFailed(SVNLogRunner.java:527)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNAdminArea.runLogs(SVNAdminArea.java:594)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNAdminArea.runLogs(SVNAdminArea.java:569)
	at org.tmatesoft.svn.core.internal.wc.SVNUpdateEditor$SVNDirectoryInfo.cleanup(SVNUpdateEditor.java:2341)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNWCAccess.doClose(SVNWCAccess.java:482)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNWCAccess.doClose(SVNWCAccess.java:463)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNWCAccess.close(SVNWCAccess.java:346)
	at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.update(SVNUpdateClient16.java:524)
	at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.doCheckout(SVNUpdateClient16.java:915)
	at org.tmatesoft.svn.core.internal.wc2.old.SvnOldCheckout.run(SvnOldCheckout.java:19)
	at org.tmatesoft.svn.core.internal.wc2.old.SvnOldCheckout.run(SvnOldCheckout.java:8)
	at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:20)
	at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1238)
	at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)
	at org.tmatesoft.svn.core.wc.SVNUpdateClient.doCheckout(SVNUpdateClient.java:777)
	at hudson.scm.subversion.CheckoutUpdater$1.perform(CheckoutUpdater.java:99)
	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:161)
	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:169)
	at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:182)
	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:161)
	at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:1030)
	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:1011)
	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:987)
	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2474)
	at hudson.remoting.UserRequest.perform(UserRequest.java:118)
	at hudson.remoting.UserRequest.perform(UserRequest.java:48)
	at hudson.remoting.Request$2.run(Request.java:328)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: No space left on device
	at java.io.FileOutputStream.writeBytes(Native Method)
	at java.io.FileOutputStream.write(FileOutputStream.java:345)
	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
	at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
	at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:297)
	at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141)
	at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNAdminArea14.writeEntry(SVNAdminArea14.java:1426)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNAdminArea14.writeEntries(SVNAdminArea14.java:1130)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNAdminArea14.saveEntries(SVNAdminArea14.java:625)
	... 32 more
lr/core/src/test/org/apache/solr/request/TestWriterPerf.java
java.io.IOException: remote file operation failed: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8 at hudson.remoting.Channel@412a89f6:lucene
	at hudson.FilePath.act(FilePath.java:916)
	at hudson.FilePath.act(FilePath.java:893)
	at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:936)
	at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:871)
	at hudson.model.AbstractProject.checkout(AbstractProject.java:1252)
	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:615)
	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:524)
	at hudson.model.Run.execute(Run.java:1706)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:88)
	at hudson.model.Executor.run(Executor.java:232)
Caused by: java.io.IOException: Failed to check out http://svn.apache.org/repos/asf/lucene/dev/trunk
	at hudson.scm.subversion.CheckoutUpdater$1.perform(CheckoutUpdater.java:110)
	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:161)
	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:169)
	at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:182)
	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:161)
	at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:1030)
	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:1011)
	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:987)
	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2474)
	at hudson.remoting.UserRequest.perform(UserRequest.java:118)
	at hudson.remoting.UserRequest.perform(UserRequest.java:48)
	at hudson.remoting.Request$2.run(Request.java:328)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.tmatesoft.svn.core.SVNException: svn: E204899: Cannot write entries file '/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/.svn/entries': No space left on device
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:85)
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:69)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNAdminArea14.saveEntries(SVNAdminArea14.java:630)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNLogRunner.logFailed(SVNLogRunner.java:527)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNAdminArea.runLogs(SVNAdminArea.java:594)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNAdminArea.runLogs(SVNAdminArea.java:569)
	at org.tmatesoft.svn.core.internal.wc.SVNUpdateEditor$SVNDirectoryInfo.cleanup(SVNUpdateEditor.java:2341)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNWCAccess.doClose(SVNWCAccess.java:482)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNWCAccess.doClose(SVNWCAccess.java:463)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNWCAccess.close(SVNWCAccess.java:346)
	at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.update(SVNUpdateClient16.java:524)
	at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.doCheckout(SVNUpdateClient16.java:915)
	at org.tmatesoft.svn.core.internal.wc2.old.SvnOldCheckout.run(SvnOldCheckout.java:19)
	at org.tmatesoft.svn.core.internal.wc2.old.SvnOldCheckout.run(SvnOldCheckout.java:8)
	at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:20)
	at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1238)
	at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)
	at org.tmatesoft.svn.core.wc.SVNUpdateClient.doCheckout(SVNUpdateClient.java:777)
	at hudson.scm.subversion.CheckoutUpdater$1.perform(CheckoutUpdater.java:99)
	... 16 more
Caused by: java.io.IOException: No space left on device
	at java.io.FileOutputStream.writeBytes(Native Method)
	at java.io.FileOutputStream.write(FileOutputStream.java:345)
	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
	at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
	at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:297)
	at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141)
	at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNAdminArea14.writeEntry(SVNAdminArea14.java:1426)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNAdminArea14.writeEntries(SVNAdminArea14.java:1130)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNAdminArea14.saveEntries(SVNAdminArea14.java:625)
	... 32 more
Retrying after 10 seconds
Cleaning up /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/.
Deleting /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/common-build.xml
Deleting /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test/org/apache/solr/request/SimpleFacetsTest.java
Deleting /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test/org/apache/solr/request/TestBinaryResponseWriter.java
Deleting /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test/org/apache/solr/request/TestFaceting.java
Deleting /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test/org/apache/solr/request/TestIntervalFaceting.java
Updating http://svn.apache.org/repos/asf/lucene/dev/trunk at revision '2015-05-23T05:29:10.099 -0400'
Workspace appear to be locked, so getting a fresh workspace
Cleaning local Directory .
Checking out http://svn.apache.org/repos/asf/lucene/dev/trunk at revision '2015-05-23T05:29:10.099 -0400'
AU        .gitignore
AU        extra-targets.xml
AU        README.txt
A         lucene
A         lucene/benchmark
AU        lucene/benchmark/build.xml
AU        lucene/benchmark/README.enwiki
A         lucene/benchmark/conf
AU        lucene/benchmark/conf/micro-standard.alg
AU        lucene/benchmark/conf/indexing-flush-by-RAM.alg
AU        lucene/benchmark/conf/createLineFile.alg
AU        lucene/benchmark/conf/standard-highlights-tv.alg
AU        lucene/benchmark/conf/indexLineFile.alg
AU        lucene/benchmark/conf/wikipediaOneRound.alg
AU        lucene/benchmark/conf/sample.alg
AU        lucene/benchmark/conf/collation.alg
AU        lucene/benchmark/conf/readContentSource.alg
AU        lucene/benchmark/conf/tokenize.alg
AU        lucene/benchmark/conf/spatial.alg
AU        lucene/benchmark/conf/compound-penalty.alg
AU        lucene/benchmark/conf/indexing-multithreaded.alg
AU        lucene/benchmark/conf/standard.alg
AU        lucene/benchmark/conf/extractWikipedia.alg
AU        lucene/benchmark/conf/indexing-flush-by-RAM-multithreaded.alg
AU        lucene/benchmark/conf/highlight-vs-vector-highlight.alg
AU        lucene/benchmark/conf/facets.alg
AU        lucene/benchmark/conf/vector-highlight-profile.alg
AU        lucene/benchmark/conf/sloppy-phrase.alg
AU        lucene/benchmark/conf/addIndexes.alg
AU        lucene/benchmark/conf/wikipedia-flush-by-RAM.alg
AU        lucene/benchmark/conf/english-porter-comparison.alg
AU        lucene/benchmark/conf/micro-standard-flush-by-ram.alg
AU        lucene/benchmark/conf/standard-highlights-notv.alg
AU        lucene/benchmark/conf/analyzer.alg
AU        lucene/benchmark/conf/sort-standard.alg
AU        lucene/benchmark/conf/collector.alg
AU        lucene/benchmark/conf/shingle.alg
AU        lucene/benchmark/conf/collector-small.alg
AU        lucene/benchmark/conf/standard-flush-by-RAM.alg
AU        lucene/benchmark/conf/indexing.alg
AU        lucene/benchmark/conf/basicNRT.alg
AU        lucene/benchmark/conf/highlight-profile.alg
AU        lucene/benchmark/conf/wikipedia.alg
AU        lucene/benchmark/ivy.xml
A         lucene/benchmark/lib
A         lucene/benchmark/scripts
AU        lucene/benchmark/scripts/shingle.bm2jira.pl
AU        lucene/benchmark/scripts/compare.shingle.benchmark.tables.pl
AU        lucene/benchmark/scripts/collation.bm2jira.pl
AU        lucene/benchmark/scripts/compare.collation.benchmark.tables.pl
A         lucene/benchmark/src
A         lucene/benchmark/src/test
A         lucene/benchmark/src/test/conf
AU        lucene/benchmark/src/test/conf/ConfLoader.java
A         lucene/benchmark/src/test/org
A         lucene/benchmark/src/test/org/apache
A         lucene/benchmark/src/test/org/apache/lucene
A         lucene/benchmark/src/test/org/apache/lucene/benchmark
A         lucene/benchmark/src/test/org/apache/lucene/benchmark/byTask
A         lucene/benchmark/src/test/org/apache/lucene/benchmark/byTask/feeds
AU        lucene/benchmark/src/test/org/apache/lucene/benchmark/byTask/feeds/trecdocs.zip
AU        lucene/benchmark/src/test/org/apache/lucene/benchmark/byTask/feeds/TrecContentSourceTest.java
AU        lucene/benchmark/src/test/org/apache/lucene/benchmark/byTask/feeds/LineDocSourceTest.java
AU        lucene/benchmark/src/test/org/apache/lucene/benchmark/byTask/feeds/TestHtmlParser.java
AU        lucene/benchmark/src/test/org/apache/lucene/benchmark/byTask/feeds/EnwikiContentSourceTest.java
AU        lucene/benchmark/src/test/org/apache/lucene/benchmark/byTask/feeds/DocMakerTest.java
AU        lucene/benchmark/src/test/org/apache/lucene/benchmark/byTask/reuters.first20.lines.txt
AU        lucene/benchmark/src/test/org/apache/lucene/benchmark/byTask/TestPerfTasksParse.java
AU        lucene/benchmark/src/test/org/apache/lucene/benchmark/byTask/test-mapping-ISOLatin1Accent-partial.txt
AU        lucene/benchmark/src/test/org/apache/lucene/benchmark/byTask/TestPerfTasksLogic.java
A         lucene/benchmark/src/test/org/apache/lucene/benchmark/byTask/utils
AU        lucene/benchmark/src/test/org/apache/lucene/benchmark/byTask/utils/TestConfig.java
AU        lucene/benchmark/src/test/org/apache/lucene/benchmark/byTask/utils/StreamUtilsTest.java
A         lucene/benchmark/src/test/org/apache/lucene/benchmark/byTask/tasks
AU        lucene/benchmark/src/test/org/apache/lucene/benchmark/byTask/tasks/CreateIndexTaskTest.java
AU        lucene/benchmark/src/test/org/apache/lucene/benchmark/byTask/tasks/CountingSearchTestTask.java
AU        lucene/benchmark/src/test/org/apache/lucene/benchmark/byTask/tasks/PerfTaskTest.java
A         lucene/benchmark/src/test/org/apache/lucene/benchmark/byTask/tasks/alt
AU        lucene/benchmark/src/test/org/apache/lucene/benchmark/byTask/tasks/alt/AltPackageTaskTest.java
AU        lucene/benchmark/src/test/org/apache/lucene/benchmark/byTask/tasks/alt/AltTestTask.java
AU        lucene/benchmark/src/test/org/apache/lucene/benchmark/byTask/tasks/WriteEnwikiLineDocTaskTest.java
AU        lucene/benchmark/src/test/org/apache/lucene/benchmark/byTask/tasks/CommitIndexTaskTest.java
AU        lucene/benchmark/src/test/org/apache/lucene/bERROR: Failed to check out http://svn.apache.org/repos/asf/lucene/dev/trunk
org.tmatesoft.svn.core.SVNException: svn: E155009: Error processing command 'cp-and-translate' in '/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8'
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:85)
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:69)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNLogRunner.runCommand(SVNLogRunner.java:477)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNLog.run(SVNLog.java:303)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNAdminArea.runLogs(SVNAdminArea.java:586)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNAdminArea.runLogs(SVNAdminArea.java:569)
	at org.tmatesoft.svn.core.internal.wc.SVNUpdateEditor$SVNDirectoryInfo.cleanup(SVNUpdateEditor.java:2341)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNWCAccess.doClose(SVNWCAccess.java:482)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNWCAccess.doClose(SVNWCAccess.java:463)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNWCAccess.close(SVNWCAccess.java:346)
	at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.update(SVNUpdateClient16.java:524)
	at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.doCheckout(SVNUpdateClient16.java:915)
	at org.tmatesoft.svn.core.internal.wc2.old.SvnOldCheckout.run(SvnOldCheckout.java:19)
	at org.tmatesoft.svn.core.internal.wc2.old.SvnOldCheckout.run(SvnOldCheckout.java:8)
	at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:20)
	at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1238)
	at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)
	at org.tmatesoft.svn.core.wc.SVNUpdateClient.doCheckout(SVNUpdateClient.java:777)
	at hudson.scm.subversion.CheckoutUpdater$1.perform(CheckoutUpdater.java:99)
	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:161)
	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:169)
	at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:182)
	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:161)
	at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:1030)
	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:1011)
	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:987)
	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2474)
	at hudson.remoting.UserRequest.perform(UserRequest.java:118)
	at hudson.remoting.UserRequest.perform(UserRequest.java:48)
	at hudson.remoting.Request$2.run(Request.java:328)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.tmatesoft.svn.core.SVNException: svn: E204899: Failed to translate '/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/extra-targets.xml'
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:85)
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:69)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNTranslator.translationError(SVNTranslator.java:733)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNTranslator.translate(SVNTranslator.java:187)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNTranslator.translate(SVNTranslator.java:150)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNTranslator.translate(SVNTranslator.java:104)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNLogRunner.runCommand(SVNLogRunner.java:339)
	... 32 more
Caused by: java.io.IOException: No space left on device
	at java.io.FileOutputStream.writeBytes(Native Method)
	at java.io.FileOutputStream.write(FileOutputStream.java:345)
	at java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNTranslatorOutputStream.write(SVNTranslatorOutputStream.java:55)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNTranslator.copy(SVNTranslator.java:493)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNTranslator.translate(SVNTranslator.java:185)
	... 35 more
enchmark/byTask/tasks/CountingHighlighterTestTask.java
AU        lucene/benchmark/src/test/org/apache/lucene/benchmark/byTask/tasks/AddIndexesTaskTest.java
AU        lucene/benchmark/src/test/org/apache/lucene/benchmark/byTask/tasks/WriteLineDocTaskTest.java
AU        lucene/benchmark/src/test/org/apache/lucene/benchmark/byTask/tasks/SearchWithSortTaskTest.java
java.io.IOException: remote file operation failed: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8 at hudson.remoting.Channel@412a89f6:lucene
	at hudson.FilePath.act(FilePath.java:916)
	at hudson.FilePath.act(FilePath.java:893)
	at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:936)
	at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:871)
	at hudson.model.AbstractProject.checkout(AbstractProject.java:1252)
	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:615)
	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:524)
	at hudson.model.Run.execute(Run.java:1706)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:88)
	at hudson.model.Executor.run(Executor.java:232)
Caused by: java.io.IOException: Failed to check out http://svn.apache.org/repos/asf/lucene/dev/trunk
	at hudson.scm.subversion.CheckoutUpdater$1.perform(CheckoutUpdater.java:110)
	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:161)
	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:169)
	at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:182)
	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:161)
	at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:1030)
	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:1011)
	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:987)
	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2474)
	at hudson.remoting.UserRequest.perform(UserRequest.java:118)
	at hudson.remoting.UserRequest.perform(UserRequest.java:48)
	at hudson.remoting.Request$2.run(Request.java:328)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.tmatesoft.svn.core.SVNException: svn: E155009: Error processing command 'cp-and-translate' in '/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8'
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:85)
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:69)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNLogRunner.runCommand(SVNLogRunner.java:477)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNLog.run(SVNLog.java:303)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNAdminArea.runLogs(SVNAdminArea.java:586)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNAdminArea.runLogs(SVNAdminArea.java:569)
	at org.tmatesoft.svn.core.internal.wc.SVNUpdateEditor$SVNDirectoryInfo.cleanup(SVNUpdateEditor.java:2341)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNWCAccess.doClose(SVNWCAccess.java:482)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNWCAccess.doClose(SVNWCAccess.java:463)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNWCAccess.close(SVNWCAccess.java:346)
	at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.update(SVNUpdateClient16.java:524)
	at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.doCheckout(SVNUpdateClient16.java:915)
	at org.tmatesoft.svn.core.internal.wc2.old.SvnOldCheckout.run(SvnOldCheckout.java:19)
	at org.tmatesoft.svn.core.internal.wc2.old.SvnOldCheckout.run(SvnOldCheckout.java:8)
	at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:20)
	at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1238)
	at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)
	at org.tmatesoft.svn.core.wc.SVNUpdateClient.doCheckout(SVNUpdateClient.java:777)
	at hudson.scm.subversion.CheckoutUpdater$1.perform(CheckoutUpdater.java:99)
	... 16 more
Caused by: org.tmatesoft.svn.core.SVNException: svn: E204899: Failed to translate '/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/extra-targets.xml'
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:85)
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:69)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNTranslator.translationError(SVNTranslator.java:733)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNTranslator.translate(SVNTranslator.java:187)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNTranslator.translate(SVNTranslator.java:150)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNTranslator.translate(SVNTranslator.java:104)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNLogRunner.runCommand(SVNLogRunner.java:339)
	... 32 more
Caused by: java.io.IOException: No space left on device
	at java.io.FileOutputStream.writeBytes(Native Method)
	at java.io.FileOutputStream.write(FileOutputStream.java:345)
	at java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNTranslatorOutputStream.write(SVNTranslatorOutputStream.java:55)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNTranslator.copy(SVNTranslator.java:493)
	at org.tmatesoft.svn.core.internal.wc.admin.SVNTranslator.translate(SVNTranslator.java:185)
	... 35 more
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Re: [JENKINS] Lucene-Solr-Tests-trunk-Java8 - Build # 5 - Still Failing

Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
+1 to moving to Ubuntu VM.

On Fri, Nov 21, 2014 at 8:13 PM, Uwe Schindler <uw...@thetaphi.de> wrote:

> Hi,
>
> you might have noticed, Lucene trunk is now on Java 8. Policeman Jenkins
> was already changed to handle that, builds are succeeding perfectly.
>
> Unfortunately, FreeBSD Jenkins has a problem, because OpenJDK8, namely the
> FreeBSD Port, crushes always while running Solr tests. Reason seems to be a
> problem in the networking layer, which is heavy modified from default
> OpenJDK. So there seems to be a bug in the patch that FreBSD applys to
> OpenJDK.
>
> I already updated to OpenJDK 8u25, but this did not help. I will open an
> issue at the FreeBSD website, but the problem is: The crush always looks
> different (sometimes sigbus, sometimes sigsegv, sometimes crushing inside
> free()) - I have also seen a failed malloc() call [caused by corrumpted
> heap].
>
> For now I disabled the test builds (nightly-smoke, nightly-tests, and
> hourly-tests). The artifact builds are still enabled, so people can
> download nightly trunk artifacts. The maven build that publishes to ASF
> snapshot repo was modified to have "-DskipTests" passed to ANT/Maven. So it
> runs and deploys the artifacts.
>
> I am not sure if we can solve the FreeBSD issues soon, so I would suggest
> that we ask ASF INFRA, if they could provide us with an Ubuntu VM machine
> so we can migrate our builds there. I have to check with them to get enough
> CPU power and free disk space, so we can run the heavy nightly tests. The
> other good thing would be: no more blackhole :-)
>
> Comments?
>
> Uwe
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
> > -----Original Message-----
> > From: Apache Jenkins Server [mailto:jenkins@builds.apache.org]
> > Sent: Friday, November 21, 2014 8:02 PM
> > To: dev@lucene.apache.org
> > Subject: [JENKINS] Lucene-Solr-Tests-trunk-Java8 - Build # 5 - Still
> Failing
> >
> > Build: https://builds.apache.org/job/Lucene-Solr-Tests-trunk-Java8/5/
> >
> > 2 tests failed.
> > REGRESSION:  org.apache.solr.TestDistributedGrouping.testDistribSearch
> >
> > Error Message:
> > org.apache.solr.client.solrj.SolrServerException: No live SolrServers
> available
> > to handle this request:[https://[ff01::114]:33332, https://
> [ff01::083]:33332,
> > https://127.0.0.1:37975, https://[ff01::213]:33332]
> >
> > Stack Trace:
> > org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
> > org.apache.solr.client.solrj.SolrServerException: No live SolrServers
> available
> > to handle this request:[https://[ff01::114]:33332, https://
> [ff01::083]:33332,
> > https://127.0.0.1:37975, https://[ff01::213]:33332]
> >       at
> > __randomizedtesting.SeedInfo.seed([103A566D0461F871:91DCD875733E984
> > D]:0)
> >       at
> >
> org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrSer
> > ver.java:569)
> >       at
> >
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:
> > 215)
> >       at
> >
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:
> > 211)
> >       at
> > org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.ja
> > va:91)
> >       at
> org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
> >       at
> > org.apache.solr.BaseDistributedSearchTestCase.queryServer(BaseDistribute
> > dSearchTestCase.java:512)
> >       at
> > org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearc
> > hTestCase.java:560)
> >       at
> > org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearc
> > hTestCase.java:542)
> >       at
> > org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearc
> > hTestCase.java:521)
> >       at
> > org.apache.solr.TestDistributedGrouping.doTest(TestDistributedGrouping.ja
> > va:206)
> >       at
> >
> org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistri
> > butedSearchTestCase.java:875)
> >       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >       at
> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j
> > ava:62)
> >       at
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
> > sorImpl.java:43)
> >       at java.lang.reflect.Method.invoke(Method.java:483)
> >       at
> > com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(Randomize
> > dRunner.java:1618)
> >       at
> > com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(Rando
> > mizedRunner.java:827)
> >       at
> > com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(Rando
> > mizedRunner.java:863)
> >       at
> > com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(Rando
> > mizedRunner.java:877)
> >       at
> > com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.
> > evaluate(SystemPropertiesRestoreRule.java:53)
> >       at
> > org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRule
> > SetupTeardownChained.java:50)
> >       at
> > org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeA
> > fterRule.java:46)
> >       at
> > com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1
> > .evaluate(SystemPropertiesInvariantRule.java:55)
> >       at
> > org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleTh
> > readAndTestName.java:49)
> >       at
> > org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRule
> > IgnoreAfterMaxFailures.java:65)
> >       at
> > org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure
> > .java:48)
> >       at
> > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stat
> > ementAdapter.java:36)
> >       at
> > com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.
> > run(ThreadLeakControl.java:365)
> >       at
> > com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask
> > (ThreadLeakControl.java:798)
> >       at
> > com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadL
> > eakControl.java:458)
> >       at
> > com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(Ran
> > domizedRunner.java:836)
> >       at
> > com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(Rando
> > mizedRunner.java:738)
> >       at
> > com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(Rando
> > mizedRunner.java:772)
> >       at
> > com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(Rando
> > mizedRunner.java:783)
> >       at
> > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stat
> > ementAdapter.java:36)
> >       at
> > com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.
> > evaluate(SystemPropertiesRestoreRule.java:53)
> >       at
> > org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeA
> > fterRule.java:46)
> >       at
> > org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreCl
> > assName.java:42)
> >       at
> > com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1
> > .evaluate(SystemPropertiesInvariantRule.java:55)
> >       at
> > com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMet
> > hodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> >       at
> > com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMet
> > hodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> >       at
> > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stat
> > ementAdapter.java:36)
> >       at
> > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stat
> > ementAdapter.java:36)
> >       at
> > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stat
> > ementAdapter.java:36)
> >       at
> > org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAss
> > ertionsRequired.java:54)
> >       at
> > org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure
> > .java:48)
> >       at
> > org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRule
> > IgnoreAfterMaxFailures.java:65)
> >       at
> > org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnore
> > TestSuites.java:55)
> >       at
> > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stat
> > ementAdapter.java:36)
> >       at
> > com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.
> > run(ThreadLeakControl.java:365)
> >       at java.lang.Thread.run(Thread.java:745)
> >
> >
> > FAILED:
> > org.apache.solr.handler.TestSolrConfigHandlerConcurrent.testDistribSearch
> >
> > Error Message:
> > [[size property not set, expected = 11, actual null, initialSize
> property not set,
> > expected = 12, actual null, autowarmCount property not set, expected =
> 13,
> > actual null], [], [], []]
> >
> > Stack Trace:
> > java.lang.AssertionError: [[size property not set, expected = 11, actual
> null,
> > initialSize property not set, expected = 12, actual null, autowarmCount
> > property not set, expected = 13, actual null], [], [], []]
> >       at
> > __randomizedtesting.SeedInfo.seed([103A566D0461F871:91DCD875733E984
> > D]:0)
> >       at org.junit.Assert.fail(Assert.java:93)
> >       at org.junit.Assert.assertTrue(Assert.java:43)
> >       at
> > org.apache.solr.handler.TestSolrConfigHandlerConcurrent.doTest(TestSolrCo
> > nfigHandlerConcurrent.java:112)
> >       at
> >
> org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistri
> > butedSearchTestCase.java:869)
> >       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >       at
> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j
> > ava:62)
> >       at
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
> > sorImpl.java:43)
> >       at java.lang.reflect.Method.invoke(Method.java:483)
> >       at
> > com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(Randomize
> > dRunner.java:1618)
> >       at
> > com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(Rando
> > mizedRunner.java:827)
> >       at
> > com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(Rando
> > mizedRunner.java:863)
> >       at
> > com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(Rando
> > mizedRunner.java:877)
> >       at
> > com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.
> > evaluate(SystemPropertiesRestoreRule.java:53)
> >       at
> > org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRule
> > SetupTeardownChained.java:50)
> >       at
> > org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeA
> > fterRule.java:46)
> >       at
> > com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1
> > .evaluate(SystemPropertiesInvariantRule.java:55)
> >       at
> > org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleTh
> > readAndTestName.java:49)
> >       at
> > org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRule
> > IgnoreAfterMaxFailures.java:65)
> >       at
> > org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure
> > .java:48)
> >       at
> > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stat
> > ementAdapter.java:36)
> >       at
> > com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.
> > run(ThreadLeakControl.java:365)
> >       at
> > com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask
> > (ThreadLeakControl.java:798)
> >       at
> > com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadL
> > eakControl.java:458)
> >       at
> > com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(Ran
> > domizedRunner.java:836)
> >       at
> > com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(Rando
> > mizedRunner.java:738)
> >       at
> > com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(Rando
> > mizedRunner.java:772)
> >       at
> > com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(Rando
> > mizedRunner.java:783)
> >       at
> > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stat
> > ementAdapter.java:36)
> >       at
> > com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.
> > evaluate(SystemPropertiesRestoreRule.java:53)
> >       at
> > org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeA
> > fterRule.java:46)
> >       at
> > org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreCl
> > assName.java:42)
> >       at
> > com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1
> > .evaluate(SystemPropertiesInvariantRule.java:55)
> >       at
> > com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMet
> > hodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> >       at
> > com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMet
> > hodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> >       at
> > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stat
> > ementAdapter.java:36)
> >       at
> > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stat
> > ementAdapter.java:36)
> >       at
> > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stat
> > ementAdapter.java:36)
> >       at
> > org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAss
> > ertionsRequired.java:54)
> >       at
> > org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure
> > .java:48)
> >       at
> > org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRule
> > IgnoreAfterMaxFailures.java:65)
> >       at
> > org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnore
> > TestSuites.java:55)
> >       at
> > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stat
> > ementAdapter.java:36)
> >       at
> > com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.
> > run(ThreadLeakControl.java:365)
> >       at java.lang.Thread.run(Thread.java:745)
> >
> >
> >
> >
> > Build Log:
> > [...truncated 13391 lines...]
> >    [junit4] Suite:
> org.apache.solr.handler.TestSolrConfigHandlerConcurrent
> >    [junit4]   2> Creating dataDir: /usr/home/jenkins/jenkins-
> > slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-
> > core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> > 103A566D0461F871-001/init-core-data-001
> >    [junit4]   2> 409504 T2325 oas.SolrTestCaseJ4.buildSSLConfig
> Randomized ssl
> > (false) and clientAuth (false)
> >    [junit4]   2> 409505 T2325
> > oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext
> > system property: /orln/
> >    [junit4]   2> 409508 T2325 oas.SolrTestCaseJ4.setUp ###Starting
> > testDistribSearch
> >    [junit4]   2> 409509 T2325 oasc.ZkTestServer.run STARTING ZK TEST
> SERVER
> >    [junit4]   1> client port:0.0.0.0/0.0.0.0:0
> >    [junit4]   2> 409510 T2326
> oasc.ZkTestServer$ZKServerMain.runFromConfig
> > Starting server
> >    [junit4]   2> 409610 T2325 oasc.ZkTestServer.run start zk server on
> > port:47646
> >    [junit4]   2> 409611 T2325
> > oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default
> > ZkCredentialsProvider
> >    [junit4]   2> 409612 T2325 oascc.ConnectionManager.waitForConnected
> > Waiting for client to connect to ZooKeeper
> >    [junit4]   2> 409616 T2333 oascc.ConnectionManager.process Watcher
> > org.apache.solr.common.cloud.ConnectionManager@3d627ba3
> > name:ZooKeeperConnection Watcher:127.0.0.1:47646 got event
> > WatchedEvent state:SyncConnected type:None path:null path:null
> > type:None
> >    [junit4]   2> 409617 T2325 oascc.ConnectionManager.waitForConnected
> > Client is connected to ZooKeeper
> >    [junit4]   2> 409617 T2325 oascc.SolrZkClient.createZkACLProvider
> Using
> > default ZkACLProvider
> >    [junit4]   2> 409617 T2325 oascc.SolrZkClient.makePath makePath: /solr
> >    [junit4]   2> 409620 T2325
> > oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default
> > ZkCredentialsProvider
> >    [junit4]   2> 409621 T2325 oascc.ConnectionManager.waitForConnected
> > Waiting for client to connect to ZooKeeper
> >    [junit4]   2> 409622 T2336 oascc.ConnectionManager.process Watcher
> > org.apache.solr.common.cloud.ConnectionManager@2458f3dd
> > name:ZooKeeperConnection Watcher:127.0.0.1:47646/solr got event
> > WatchedEvent state:SyncConnected type:None path:null path:null
> > type:None
> >    [junit4]   2> 409622 T2325 oascc.ConnectionManager.waitForConnected
> > Client is connected to ZooKeeper
> >    [junit4]   2> 409623 T2325 oascc.SolrZkClient.createZkACLProvider
> Using
> > default ZkACLProvider
> >    [junit4]   2> 409623 T2325 oascc.SolrZkClient.makePath makePath:
> > /collections/collection1
> >    [junit4]   2> 409625 T2325 oascc.SolrZkClient.makePath makePath:
> > /collections/collection1/shards
> >    [junit4]   2> 409626 T2325 oascc.SolrZkClient.makePath makePath:
> > /collections/control_collection
> >    [junit4]   2> 409627 T2325 oascc.SolrZkClient.makePath makePath:
> > /collections/control_collection/shards
> >    [junit4]   2> 409629 T2325 oasc.AbstractZkTestCase.putConfig put
> > /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-
> > Java8/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml
> to
> > /configs/conf1/solrconfig.xml
> >    [junit4]   2> 409630 T2325 oascc.SolrZkClient.makePath makePath:
> > /configs/conf1/solrconfig.xml
> >    [junit4]   2> 409632 T2325 oasc.AbstractZkTestCase.putConfig put
> > /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-
> > Java8/solr/core/src/test-files/solr/collection1/conf/schema.xml to
> > /configs/conf1/schema.xml
> >    [junit4]   2> 409632 T2325 oascc.SolrZkClient.makePath makePath:
> > /configs/conf1/schema.xml
> >    [junit4]   2> 409734 T2325 oasc.AbstractZkTestCase.putConfig put
> > /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-
> > Java8/solr/core/src/test-
> > files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to
> > /configs/conf1/solrconfig.snippet.randomindexconfig.xml
> >    [junit4]   2> 409735 T2325 oascc.SolrZkClient.makePath makePath:
> > /configs/conf1/solrconfig.snippet.randomindexconfig.xml
> >    [junit4]   2> 409736 T2325 oasc.AbstractZkTestCase.putConfig put
> > /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-
> > Java8/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to
> > /configs/conf1/stopwords.txt
> >    [junit4]   2> 409737 T2325 oascc.SolrZkClient.makePath makePath:
> > /configs/conf1/stopwords.txt
> >    [junit4]   2> 409738 T2325 oasc.AbstractZkTestCase.putConfig put
> > /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-
> > Java8/solr/core/src/test-files/solr/collection1/conf/protwords.txt to
> > /configs/conf1/protwords.txt
> >    [junit4]   2> 409739 T2325 oascc.SolrZkClient.makePath makePath:
> > /configs/conf1/protwords.txt
> >    [junit4]   2> 409740 T2325 oasc.AbstractZkTestCase.putConfig put
> > /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-
> > Java8/solr/core/src/test-files/solr/collection1/conf/currency.xml to
> > /configs/conf1/currency.xml
> >    [junit4]   2> 409741 T2325 oascc.SolrZkClient.makePath makePath:
> > /configs/conf1/currency.xml
> >    [junit4]   2> 409742 T2325 oasc.AbstractZkTestCase.putConfig put
> > /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-
> > Java8/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to
> > /configs/conf1/enumsConfig.xml
> >    [junit4]   2> 409743 T2325 oascc.SolrZkClient.makePath makePath:
> > /configs/conf1/enumsConfig.xml
> >    [junit4]   2> 409745 T2325 oasc.AbstractZkTestCase.putConfig put
> > /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-
> > Java8/solr/core/src/test-files/solr/collection1/conf/open-exchange-
> > rates.json to /configs/conf1/open-exchange-rates.json
> >    [junit4]   2> 409745 T2325 oascc.SolrZkClient.makePath makePath:
> > /configs/conf1/open-exchange-rates.json
> >    [junit4]   2> 409747 T2325 oasc.AbstractZkTestCase.putConfig put
> > /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-
> > Java8/solr/core/src/test-files/solr/collection1/conf/mapping-
> > ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
> >    [junit4]   2> 409747 T2325 oascc.SolrZkClient.makePath makePath:
> > /configs/conf1/mapping-ISOLatin1Accent.txt
> >    [junit4]   2> 409749 T2325 oasc.AbstractZkTestCase.putConfig put
> > /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-
> > Java8/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to
> > /configs/conf1/old_synonyms.txt
> >    [junit4]   2> 409750 T2325 oascc.SolrZkClient.makePath makePath:
> > /configs/conf1/old_synonyms.txt
> >    [junit4]   2> 409751 T2325 oasc.AbstractZkTestCase.putConfig put
> > /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-
> > Java8/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to
> > /configs/conf1/synonyms.txt
> >    [junit4]   2> 409751 T2325 oascc.SolrZkClient.makePath makePath:
> > /configs/conf1/synonyms.txt
> >    [junit4]   2> 410973 T2325 oejs.Server.doStart jetty-8.1.10.v20130312
> >    [junit4]   2> 410994 T2325 oejs.AbstractConnector.doStart Started
> > SocketConnector@127.0.0.1:37737
> >    [junit4]   2> 410994 T2325 oass.SolrDispatchFilter.init
> SolrDispatchFilter.init()
> >    [junit4]   2> 410994 T2325 oasc.SolrResourceLoader.locateSolrHome JNDI
> > not configured for solr (NoInitialContextEx)
> >    [junit4]   2> 410995 T2325 oasc.SolrResourceLoader.locateSolrHome
> using
> > system property solr.solr.home:
> > ../../../../../../../../../../../usr/home/jenkins/jenkins-
> > slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-
> > core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> > 103A566D0461F871-001/tempDir-002
> >    [junit4]   2> 410995 T2325 oasc.SolrResourceLoader.<init> new
> > SolrResourceLoader for directory:
> > '../../../../../../../../../../../usr/home/jenkins/jenkins-
> > slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-
> > core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> > 103A566D0461F871-001/tempDir-002/'
> >    [junit4]   2> 411025 T2325 oasc.ConfigSolr.fromFile Loading container
> > configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-
> > Solr-Tests-trunk-Java8/solr/build/solr-
> > core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-
> > slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-
> > core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> > 103A566D0461F871-001/tempDir-002/solr.xml
> >    [junit4]   2> 411038 T2325 oasc.CoreContainer.<init> New CoreContainer
> > 1421426377
> >    [junit4]   2> 411039 T2325 oasc.CoreContainer.load Loading cores into
> > CoreContainer
> > [instanceDir=../../../../../../../../../../../usr/home/jenkins/jenkins-
> > slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-
> > core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> > 103A566D0461F871-001/tempDir-002/]
> >    [junit4]   2> 411039 T2325 oashc.HttpShardHandlerFactory.getParameter
> > Setting socketTimeout to: 90000
> >    [junit4]   2> 411040 T2325 oashc.HttpShardHandlerFactory.getParameter
> > Setting urlScheme to:
> >    [junit4]   2> 411040 T2325 oashc.HttpShardHandlerFactory.getParameter
> > Setting connTimeout to: 15000
> >    [junit4]   2> 411040 T2325 oashc.HttpShardHandlerFactory.getParameter
> > Setting maxConnectionsPerHost to: 20
> >    [junit4]   2> 411040 T2325 oashc.HttpShardHandlerFactory.getParameter
> > Setting maxConnections to: 10000
> >    [junit4]   2> 411040 T2325 oashc.HttpShardHandlerFactory.getParameter
> > Setting corePoolSize to: 0
> >    [junit4]   2> 411041 T2325 oashc.HttpShardHandlerFactory.getParameter
> > Setting maximumPoolSize to: 2147483647
> >    [junit4]   2> 411041 T2325 oashc.HttpShardHandlerFactory.getParameter
> > Setting maxThreadIdleTime to: 5
> >    [junit4]   2> 411041 T2325 oashc.HttpShardHandlerFactory.getParameter
> > Setting sizeOfQueue to: -1
> >    [junit4]   2> 411041 T2325 oashc.HttpShardHandlerFactory.getParameter
> > Setting fairnessPolicy to: false
> >    [junit4]   2> 411042 T2325 oasu.UpdateShardHandler.<init> Creating
> > UpdateShardHandler HTTP client with params:
> > socketTimeout=340000&connTimeout=45000&retry=false
> >    [junit4]   2> 411042 T2325 oasl.LogWatcher.createWatcher SLF4J impl is
> > org.slf4j.impl.Log4jLoggerFactory
> >    [junit4]   2> 411042 T2325 oasl.LogWatcher.newRegisteredLogWatcher
> > Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
> >    [junit4]   2> 411043 T2325 oasc.CoreContainer.load Host Name:
> 127.0.0.1
> >    [junit4]   2> 411043 T2325 oasc.ZkContainer.initZooKeeper Zookeeper
> > client=127.0.0.1:47646/solr
> >    [junit4]   2> 411043 T2325 oasc.ZkController.checkChrootPath zkHost
> > includes chroot
> >    [junit4]   2> 411044 T2325
> > oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default
> > ZkCredentialsProvider
> >    [junit4]   2> 411044 T2325 oascc.ConnectionManager.waitForConnected
> > Waiting for client to connect to ZooKeeper
> >    [junit4]   2> 411046 T2348 oascc.ConnectionManager.process Watcher
> > org.apache.solr.common.cloud.ConnectionManager@5640bd9
> > name:ZooKeeperConnection Watcher:127.0.0.1:47646 got event
> > WatchedEvent state:SyncConnected type:None path:null path:null
> > type:None
> >    [junit4]   2> 411047 T2325 oascc.ConnectionManager.waitForConnected
> > Client is connected to ZooKeeper
> >    [junit4]   2> 411047 T2325 oascc.SolrZkClient.createZkACLProvider
> Using
> > default ZkACLProvider
> >    [junit4]   2> 411049 T2325 oascc.ConnectionManager.waitForConnected
> > Waiting for client to connect to ZooKeeper
> >    [junit4]   2> 411050 T2351 oascc.ConnectionManager.process Watcher
> > org.apache.solr.common.cloud.ConnectionManager@286c5bed
> > name:ZooKeeperConnection Watcher:127.0.0.1:47646/solr got event
> > WatchedEvent state:SyncConnected type:None path:null path:null
> > type:None
> >    [junit4]   2> 411051 T2325 oascc.ConnectionManager.waitForConnected
> > Client is connected to ZooKeeper
> >    [junit4]   2> 411052 T2325 oascc.SolrZkClient.makePath makePath:
> > /overseer/queue
> >    [junit4]   2> 411053 T2325 oascc.SolrZkClient.makePath makePath:
> > /overseer/collection-queue-work
> >    [junit4]   2> 411055 T2325 oascc.SolrZkClient.makePath makePath:
> > /overseer/collection-map-running
> >    [junit4]   2> 411056 T2325 oascc.SolrZkClient.makePath makePath:
> > /overseer/collection-map-completed
> >    [junit4]   2> 411058 T2325 oascc.SolrZkClient.makePath makePath:
> > /overseer/collection-map-failure
> >    [junit4]   2> 411060 T2325 oascc.SolrZkClient.makePath makePath:
> > /live_nodes
> >    [junit4]   2> 411060 T2325 oasc.ZkController.createEphemeralLiveNode
> > Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37737_orln
> >    [junit4]   2> 411061 T2325 oascc.SolrZkClient.makePath makePath:
> > /live_nodes/127.0.0.1:37737_orln
> >    [junit4]   2> 411062 T2325 oascc.SolrZkClient.makePath makePath:
> > /overseer_elect
> >    [junit4]   2> 411063 T2325 oascc.SolrZkClient.makePath makePath:
> > /overseer_elect/election
> >    [junit4]   2> 411064 T2325 oasc.Overseer.close Overseer (id=null)
> closing
> >    [junit4]   2> 411065 T2325
> oasc.OverseerElectionContext.runLeaderProcess I
> > am going to be the leader 127.0.0.1:37737_orln
> >    [junit4]   2> 411066 T2325 oascc.SolrZkClient.makePath makePath:
> > /overseer_elect/leader
> >    [junit4]   2> 411067 T2325 oasc.Overseer.start Overseer
> > (id=92837767038042115-127.0.0.1:37737_orln-n_0000000000) starting
> >    [junit4]   2> 411068 T2325 oascc.SolrZkClient.makePath makePath:
> > /overseer/queue-work
> >    [junit4]   2> 411072 T2325
> oasc.OverseerAutoReplicaFailoverThread.<init>
> > Starting OverseerAutoReplicaFailoverThread
> > autoReplicaFailoverWorkLoopDelay=10000
> > autoReplicaFailoverWaitAfterExpiration=30000
> > autoReplicaFailoverBadNodeExpiration=60000
> >    [junit4]   2> 411073 T2353 oasc.OverseerCollectionProcessor.run
> Process
> > current queue of collection creations
> >    [junit4]   2> 411073 T2325 oascc.SolrZkClient.makePath makePath:
> > /clusterstate.json
> >    [junit4]   2> 411075 T2325 oascc.SolrZkClient.makePath makePath:
> > /aliases.json
> >    [junit4]   2> 411075 T2325
> > oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating
> > cluster state from ZooKeeper...
> >    [junit4]   2> 411077 T2352 oasc.Overseer$ClusterStateUpdater.run
> Starting
> > to work on the main queue
> >    [junit4]   2> 411079 T2355 oasc.ZkController.publish publishing
> > core=collection1 state=down collection=control_collection
> >    [junit4]   2> 411080 T2355 oasc.ZkController.publish numShards not
> found
> > on descriptor - reading it from system property
> >    [junit4]   2> 411080 T2351 oasc.DistributedQueue$LatchWatcher.process
> > NodeChildrenChanged fired on path /overseer/queue state SyncConnected
> >    [junit4]   2> 411080 T2355 oasc.ZkController.waitForCoreNodeName look
> for
> > our core node name
> >    [junit4]   2> 411082 T2352
> oasc.Overseer$ClusterStateUpdater.updateState
> > Update state numShards=1 message={
> >    [junit4]   2>        "core":"collection1",
> >    [junit4]   2>        "roles":null,
> >    [junit4]   2>        "base_url":"http://127.0.0.1:37737/orln",
> >    [junit4]   2>        "node_name":"127.0.0.1:37737_orln",
> >    [junit4]   2>        "numShards":"1",
> >    [junit4]   2>        "state":"down",
> >    [junit4]   2>        "shard":null,
> >    [junit4]   2>        "collection":"control_collection",
> >    [junit4]   2>        "operation":"state"}
> >    [junit4]   2> 411082 T2352
> > oasc.Overseer$ClusterStateUpdater.createCollection Create collection
> > control_collection with shards [shard1]
> >    [junit4]   2> 411082 T2352
> > oasc.Overseer$ClusterStateUpdater.createCollection state version
> > control_collection 1
> >    [junit4]   2> 411082 T2352
> oasc.Overseer$ClusterStateUpdater.updateState
> > Assigning new node to shard shard=shard1
> >    [junit4]   2> 411084 T2351 oascc.ZkStateReader$2.process A cluster
> state
> > change: WatchedEvent state:SyncConnected type:NodeDataChanged
> > path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> >    [junit4]   2> 412081 T2355 oasc.ZkController.waitForShardId waiting
> to find
> > shard id in clusterstate for collection1
> >    [junit4]   2> 412082 T2355 oasc.ZkController.createCollectionZkNode
> Check
> > for collection zkNode:control_collection
> >    [junit4]   2> 412082 T2355 oasc.ZkController.createCollectionZkNode
> > Collection zkNode exists
> >    [junit4]   2> 412083 T2355 oascc.ZkStateReader.readConfigName Load
> > collection config from:/collections/control_collection
> >    [junit4]   2> 412083 T2355 oascc.ZkStateReader.readConfigName
> > path=/collections/control_collection configName=conf1 specified config
> > exists in ZooKeeper
> >    [junit4]   2> 412083 T2355 oasc.SolrResourceLoader.<init> new
> > SolrResourceLoader for directory:
> > '../../../../../../../../../../../usr/home/jenkins/jenkins-
> > slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-
> > core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> > 103A566D0461F871-001/tempDir-002/collection1/'
> >    [junit4]   2> 412085 T2355 oasc.SolrResourceLoader.replaceClassLoader
> > Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-
> > trunk-Java8/solr/build/solr-
> > core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> > 103A566D0461F871-001/tempDir-002/collection1/lib/.svn/' to classloader
> >    [junit4]   2> 412086 T2355 oasc.SolrResourceLoader.replaceClassLoader
> > Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-
> > trunk-Java8/solr/build/solr-
> > core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> > 103A566D0461F871-001/tempDir-002/collection1/lib/README' to classloader
> >    [junit4]   2> 412087 T2355 oasc.SolrResourceLoader.replaceClassLoader
> > Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-
> > trunk-Java8/solr/build/solr-
> > core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> > 103A566D0461F871-001/tempDir-002/collection1/lib/classes/' to classloader
> >    [junit4]   2> 412119 T2355 oasc.ZkController.watchZKConfDir watch
> zkdir
> > /configs/conf1
> >    [junit4]   2> 412120 T2355 oasc.Config.<init> loaded config
> solrconfig.xml
> > with version 0
> >    [junit4]   2> 412134 T2355 oasc.SolrConfig.<init> Using Lucene
> > MatchVersion: 6.0.0
> >    [junit4]   2> 412145 T2355 oasc.SolrConfig.<init> Loaded SolrConfig:
> > solrconfig.xml
> >    [junit4]   2> 412246 T2355 oass.IndexSchema.readSchema Reading Solr
> > Schema from /configs/conf1/schema.xml
> >    [junit4]   2> 412252 T2355 oass.IndexSchema.readSchema [collection1]
> > Schema name=test
> >    [junit4]   2> 412366 T2355 oass.OpenExchangeRatesOrgProvider.init
> > Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
> >    [junit4]   2> 412376 T2355 oass.IndexSchema.readSchema default search
> > field in schema is text
> >    [junit4]   2> 412378 T2355 oass.IndexSchema.readSchema unique key
> field:
> > id
> >    [junit4]   2> 412385 T2355 oass.FileExchangeRateProvider.reload
> Reloading
> > exchange rates from file currency.xml
> >    [junit4]   2> 412388 T2355 oass.FileExchangeRateProvider.reload
> Reloading
> > exchange rates from file currency.xml
> >    [junit4]   2> 412390 T2355 oass.OpenExchangeRatesOrgProvider.reload
> > Reloading exchange rates from open-exchange-rates.json
> >    [junit4]   2> 412391 T2355
> > oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN
> > Unknown key IMPORTANT NOTE
> >    [junit4]   2> 412392 T2355
> > oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN
> > Expected key, got STRING
> >    [junit4]   2> 412392 T2355 oass.OpenExchangeRatesOrgProvider.reload
> > Reloading exchange rates from open-exchange-rates.json
> >    [junit4]   2> 412392 T2355
> > oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN
> > Unknown key IMPORTANT NOTE
> >    [junit4]   2> 412393 T2355
> > oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN
> > Expected key, got STRING
> >    [junit4]   2> 412393 T2355 oasc.CoreContainer.create Creating SolrCore
> > 'collection1' using configuration from collection control_collection
> >    [junit4]   2> 412393 T2355 oasc.SolrCore.initDirectoryFactory
> > org.apache.solr.core.MockDirectoryFactory
> >    [junit4]   2> 412393 T2355 oasc.SolrCore.<init> [collection1] Opening
> new
> > SolrCore at ../../../../../../../../../../../usr/home/jenkins/jenkins-
> > slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-
> > core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> > 103A566D0461F871-001/tempDir-002/collection1/,
> > dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-
> > trunk-Java8/solr/build/solr-
> > core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> > 103A566D0461F871-001/tempDir-001/control/data/
> >    [junit4]   2> 412394 T2355 oasc.JmxMonitoredMap.<init> JMX monitoring
> is
> > enabled. Adding Solr mbeans to JMX Server:
> > com.sun.jmx.mbeanserver.JmxMBeanServer@330fda4c
> >    [junit4]   2> 412394 T2355 oasc.CachingDirectoryFactory.get return new
> > directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-
> > Tests-trunk-Java8/solr/build/solr-
> > core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> > 103A566D0461F871-001/tempDir-001/control/data
> >    [junit4]   2> 412395 T2355 oasc.SolrCore.getNewIndexDir New index
> > directory detected: old=null new=/usr/home/jenkins/jenkins-
> > slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-
> > core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> > 103A566D0461F871-001/tempDir-001/control/data/index/
> >    [junit4]   2> 412395 T2355 oasc.SolrCore.initIndex WARN [collection1]
> Solr
> > index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-
> > Tests-trunk-Java8/solr/build/solr-
> > core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> > 103A566D0461F871-001/tempDir-001/control/data/index' doesn't exist.
> > Creating new index...
> >    [junit4]   2> 412395 T2355 oasc.CachingDirectoryFactory.get return new
> > directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-
> > Tests-trunk-Java8/solr/build/solr-
> > core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> > 103A566D0461F871-001/tempDir-001/control/data/index
> >    [junit4]   2> 412396 T2355 oasu.RandomMergePolicy.<init>
> > RandomMergePolicy wrapping class
> > org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy:
> > maxMergeAtOnce=30, maxMergeAtOnceExplicit=41,
> > maxMergedSegmentMB=64.537109375, floorSegmentMB=1.693359375,
> > forceMergeDeletesPctAllowed=3.111209946074762, segmentsPerTier=41.0,
> > maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
> >    [junit4]   2> 412397 T2355 oasc.SolrDeletionPolicy.onCommit
> > SolrDeletionPolicy.onCommit: commits: num=1
> >    [junit4]   2>
> >       commit{dir=MockDirectoryWrapper(RAMDirectory@17bfecc0
> > lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@3d315b92),
> > segFN=segments_1,generation=1}
> >    [junit4]   2> 412397 T2355 oasc.SolrDeletionPolicy.updateCommits
> newest
> > commit generation = 1
> >    [junit4]   2> 412400 T2355 oasup.UpdateRequestProcessorChain.init
> > creating updateRequestProcessorChain "nodistrib"
> >    [junit4]   2> 412400 T2355 oasup.UpdateRequestProcessorChain.init
> > creating updateRequestProcessorChain "dedupe"
> >    [junit4]   2> 412401 T2355 oasup.UpdateRequestProcessorChain.init
> > inserting DistributedUpdateProcessorFactory into
> > updateRequestProcessorChain "dedupe"
> >    [junit4]   2> 412401 T2355 oasup.UpdateRequestProcessorChain.init
> > creating updateRequestProcessorChain "stored_sig"
> >    [junit4]   2> 412401 T2355 oasup.UpdateRequestProcessorChain.init
> > inserting DistributedUpdateProcessorFactory into
> > updateRequestProcessorChain "stored_sig"
> >    [junit4]   2> 412401 T2355 oasup.UpdateRequestProcessorChain.init
> > creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
> >    [junit4]   2> 412402 T2355 oasup.UpdateRequestProcessorChain.init
> > creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
> >    [junit4]   2> 412402 T2355 oasup.UpdateRequestProcessorChain.init
> > inserting DistributedUpdateProcessorFactory into
> > updateRequestProcessorChain "distrib-dup-test-chain-implicit"
> >    [junit4]   2> 412402 T2355 oasc.SolrCore.loadUpdateProcessorChains no
> > updateRequestProcessorChain defined as default, creating implicit default
> >    [junit4]   2> 412403 T2355 oasc.RequestHandlers.initHandlersFromConfig
> > created /config: org.apache.solr.handler.SolrConfigHandler
> >    [junit4]   2> 412404 T2355 oasc.RequestHandlers.initHandlersFromConfig
> > created /update/csv: org.apache.solr.handler.UpdateRequestHandler
> >    [junit4]   2> 412404 T2355 oasc.RequestHandlers.initHandlersFromConfig
> > created /update/json: org.apache.solr.handler.UpdateRequestHandler
> >    [junit4]   2> 412404 T2355 oasc.RequestHandlers.initHandlersFromConfig
> > created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
> >    [junit4]   2> 412405 T2355 oasc.RequestHandlers.initHandlersFromConfig
> > created standard: solr.StandardRequestHandler
> >    [junit4]   2> 412405 T2355 oasc.RequestHandlers.initHandlersFromConfig
> > created /get: solr.RealTimeGetHandler
> >    [junit4]   2> 412405 T2355 oasc.RequestHandlers.initHandlersFromConfig
> > adding lazy requestHandler: solr.ReplicationHandler
> >    [junit4]   2> 412406 T2355 oasc.RequestHandlers.initHandlersFromConfig
> > created /replication: solr.ReplicationHandler
> >    [junit4]   2> 412406 T2355 oasc.RequestHandlers.initHandlersFromConfig
> > created /update: solr.UpdateRequestHandler
> >    [junit4]   2> 412406 T2355 oasc.RequestHandlers.initHandlersFromConfig
> > created /admin/: org.apache.solr.handler.admin.AdminHandlers
> >    [junit4]   2> 412407 T2355 oasc.RequestHandlers.initHandlersFromConfig
> > created /admin/file: solr.admin.ShowFileRequestHandler
> >    [junit4]   2> 412407 T2355 oasc.RequestHandlers.initHandlersFromConfig
> > created /admin/fileedit: solr.admin.EditFileRequestHandler
> >    [junit4]   2> 412417 T2355 oashl.XMLLoader.init
> > xsltCacheLifetimeSeconds=60
> >    [junit4]   2> 412418 T2355 oashl.XMLLoader.init
> > xsltCacheLifetimeSeconds=60
> >    [junit4]   2> 412420 T2355 oashl.XMLLoader.init
> > xsltCacheLifetimeSeconds=60
> >    [junit4]   2> 412421 T2355 oashl.XMLLoader.init
> > xsltCacheLifetimeSeconds=60
> >    [junit4]   2> 412425 T2355 oasu.CommitTracker.<init> Hard AutoCommit:
> > disabled
> >    [junit4]   2> 412425 T2355 oasu.CommitTracker.<init> Soft AutoCommit:
> > disabled
> >    [junit4]   2> 412425 T2355 oasu.RandomMergePolicy.<init>
> > RandomMergePolicy wrapping class
> > org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy:
> > maxMergeAtOnce=16, maxMergeAtOnceExplicit=15,
> > maxMergedSegmentMB=60.369140625, floorSegmentMB=1.87109375,
> > forceMergeDeletesPctAllowed=16.64506491494333, segmentsPerTier=38.0,
> > maxCFSSegmentSizeMB=8.796093022207999E12,
> > noCFSRatio=0.3115152765900482
> >    [junit4]   2> 412426 T2355 oasc.SolrDeletionPolicy.onInit
> > SolrDeletionPolicy.onInit: commits: num=1
> >    [junit4]   2>
> >       commit{dir=MockDirectoryWrapper(RAMDirectory@17bfecc0
> > lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@3d315b92),
> > segFN=segments_1,generation=1}
> >    [junit4]   2> 412426 T2355 oasc.SolrDeletionPolicy.updateCommits
> newest
> > commit generation = 1
> >    [junit4]   2> 412427 T2355 oass.SolrIndexSearcher.<init> Opening
> > Searcher@48d7329e[collection1] main
> >    [junit4]   2> 412427 T2355 oascc.ZkStateReader.readConfigName Load
> > collection config from:/collections/control_collection
> >    [junit4]   2> 412428 T2355 oascc.ZkStateReader.readConfigName
> > path=/collections/control_collection configName=conf1 specified config
> > exists in ZooKeeper
> >    [junit4]   2> 412428 T2355 oasr.ManagedResourceStorage.newStorageIO
> > Setting up ZooKeeper-based storage for the RestManager with znodeBase:
> > /configs/conf1
> >    [junit4]   2> 412428 T2355
> > oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured
> > ZooKeeperStorageIO with znodeBase: /configs/conf1
> >    [junit4]   2> 412428 T2355 oasr.RestManager.init Initializing
> RestManager
> > with initArgs: {}
> >    [junit4]   2> 412429 T2355 oasr.ManagedResourceStorage.load Reading
> > _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
> >    [junit4]   2> 412429 T2355
> > oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No
> > data found for znode /configs/conf1/_rest_managed.json
> >    [junit4]   2> 412429 T2355 oasr.ManagedResourceStorage.load Loaded
> null
> > at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
> >    [junit4]   2> 412430 T2355 oascc.SolrZkClient.makePath makePath:
> > /configs/conf1/_rest_managed.json
> >    [junit4]   2> 412431 T2355
> > oasr.ManagedResourceStorage$ZooKeeperStorageIO$1.close Wrote 38
> > bytes to new znode /configs/conf1/_rest_managed.json
> >    [junit4]   2> 412432 T2355
> > oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to
> > path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
> >    [junit4]   2> 412432 T2355 oasr.RestManager.init Initializing 0
> registered
> > ManagedResources
> >    [junit4]   2> 412434 T2356 oasc.SolrCore.registerSearcher
> [collection1]
> > Registered new searcher Searcher@48d7329e[collection1]
> > main{ExitableDirectoryReader(UninvertingDirectoryReader())}
> >    [junit4]   2> 412435 T2355 oasc.CoreContainer.registerCore
> registering core:
> > collection1
> >    [junit4]   2> 412436 T2359 oasc.ZkController.register Register
> replica -
> > core:collection1 address:http://127.0.0.1:37737/orln
> > collection:control_collection shard:shard1
> >    [junit4]   2> 412436 T2325 oass.SolrDispatchFilter.init
> > user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-
> > trunk-Java8/solr/build/solr-core/test/J3
> >    [junit4]   2> 412436 T2325 oass.SolrDispatchFilter.init
> SolrDispatchFilter.init()
> > done
> >    [junit4]   2> 412436 T2359 oascc.SolrZkClient.makePath makePath:
> > /collections/control_collection/leader_elect/shard1/election
> >    [junit4]   2> 412437 T2325
> > oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default
> > ZkCredentialsProvider
> >    [junit4]   2> 412438 T2325 oascc.ConnectionManager.waitForConnected
> > Waiting for client to connect to ZooKeeper
> >    [junit4]   2> 412440 T2362 oascc.ConnectionManager.process Watcher
> > org.apache.solr.common.cloud.ConnectionManager@7e3464f1
> > name:ZooKeeperConnection Watcher:127.0.0.1:47646/solr got event
> > WatchedEvent state:SyncConnected type:None path:null path:null
> > type:None
> >    [junit4]   2> 412440 T2325 oascc.ConnectionManager.waitForConnected
> > Client is connected to ZooKeeper
> >    [junit4]   2> 412440 T2325 oascc.SolrZkClient.createZkACLProvider
> Using
> > default ZkACLProvider
> >    [junit4]   2> 412441 T2325
> > oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating
> > cluster state from ZooKeeper...
> >    [junit4]   2> 412441 T2359
> > oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader
> > process for shard shard1
> >    [junit4]   2> 412443 T2351 oasc.DistributedQueue$LatchWatcher.process
> > NodeChildrenChanged fired on path /overseer/queue state SyncConnected
> >    [junit4]   2> 412443 T2359
> > oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough
> > replicas found to continue.
> >    [junit4]   2> 412443 T2359
> > oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader
> > - try and sync
> >    [junit4]   2> ASYNC  NEW_CORE C136 name=collection1
> > org.apache.solr.core.SolrCore@3982b644
> > url=http://127.0.0.1:37737/orln/collection1 node=127.0.0.1:37737_orln
> > C136_STATE=coll:control_collection core:collection1
> props:{core=collection1,
> > base_url=http://127.0.0.1:37737/orln, node_name=127.0.0.1:37737_orln,
> > state=down}
> >    [junit4]   2> 412443 T2359 C136 P37737 oasc.SyncStrategy.sync Sync
> replicas
> > to http://127.0.0.1:37737/orln/collection1/
> >    [junit4]   2> 412444 T2359 C136 P37737 oasc.SyncStrategy.syncReplicas
> Sync
> > Success - now sync replicas to me
> >    [junit4]   2> 412444 T2325 oasc.ChaosMonkey.monkeyLog monkey: init -
> > expire sessions:false cause connection loss:false
> >    [junit4]   2> 412444 T2359 C136 P37737 oasc.SyncStrategy.syncToMe
> > http://127.0.0.1:37737/orln/collection1/ has no replicas
> >    [junit4]   2> 412445 T2325
> oasc.AbstractFullDistribZkTestBase.createJettys
> > Creating collection1 with stateFormat=2
> >    [junit4]   2> 412445 T2359
> > oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader:
> > http://127.0.0.1:37737/orln/collection1/ shard1
> >    [junit4]   2> 412446 T2325
> > oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default
> > ZkCredentialsProvider
> >    [junit4]   2> 412446 T2359 oascc.SolrZkClient.makePath makePath:
> > /collections/control_collection/leaders/shard1
> >    [junit4]   2> 412447 T2325 oascc.ConnectionManager.waitForConnected
> > Waiting for client to connect to ZooKeeper
> >    [junit4]   2> 412448 T2365 oascc.ConnectionManager.process Watcher
> > org.apache.solr.common.cloud.ConnectionManager@5249c5fa
> > name:ZooKeeperConnection Watcher:127.0.0.1:47646/solr got event
> > WatchedEvent state:SyncConnected type:None path:null path:null
> > type:None
> >    [junit4]   2> 412448 T2325 oascc.ConnectionManager.waitForConnected
> > Client is connected to ZooKeeper
> >    [junit4]   2> 412449 T2325 oascc.SolrZkClient.createZkACLProvider
> Using
> > default ZkACLProvider
> >    [junit4]   2> 412450 T2351 oasc.DistributedQueue$LatchWatcher.process
> > NodeChildrenChanged fired on path /overseer/queue state SyncConnected
> >    [junit4]   2> 412453 T2352
> > oasc.Overseer$ClusterStateUpdater.buildCollection building a new
> > collection: collection1
> >    [junit4]   2> 412454 T2352
> > oasc.Overseer$ClusterStateUpdater.createCollection Create collection
> > collection1 with shards [shard1, shard2]
> >    [junit4]   2> 412454 T2352
> > oasc.Overseer$ClusterStateUpdater.createCollection state version
> > collection1 2
> >    [junit4]   2> 412455 T2351 oasc.DistributedQueue$LatchWatcher.process
> > NodeChildrenChanged fired on path /overseer/queue state SyncConnected
> >    [junit4]   2> 412455 T2352
> > oasc.Overseer$ClusterStateUpdater.updateZkStates going to
> > create_collection /collections/collection1/state.json
> >    [junit4]   2> 412456 T2351 oascc.ZkStateReader$2.process A cluster
> state
> > change: WatchedEvent state:SyncConnected type:NodeDataChanged
> > path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> >    [junit4]   2> 412456 T2362 oascc.ZkStateReader$2.process A cluster
> state
> > change: WatchedEvent state:SyncConnected type:NodeDataChanged
> > path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> >    [junit4]   2> 412502 T2359 oasc.ZkController.register We are
> > http://127.0.0.1:37737/orln/collection1/ and leader is
> > http://127.0.0.1:37737/orln/collection1/
> >    [junit4]   2> 412503 T2359 oasc.ZkController.register No LogReplay
> needed
> > for core=collection1 baseURL=http://127.0.0.1:37737/orln
> >    [junit4]   2> 412503 T2359 oasc.ZkController.checkRecovery I am the
> leader,
> > no recovery necessary
> >    [junit4]   2> 412503 T2359 oasc.ZkController.publish publishing
> > core=collection1 state=active collection=control_collection
> >    [junit4]   2> 412503 T2359 oasc.ZkController.publish numShards not
> found
> > on descriptor - reading it from system property
> >    [junit4]   2> 412504 T2351 oasc.DistributedQueue$LatchWatcher.process
> > NodeChildrenChanged fired on path /overseer/queue state SyncConnected
> >    [junit4]   2> 412506 T2352
> oasc.Overseer$ClusterStateUpdater.updateState
> > Update state numShards=2 message={
> >    [junit4]   2>        "core":"collection1",
> >    [junit4]   2>        "core_node_name":"core_node1",
> >    [junit4]   2>        "roles":null,
> >    [junit4]   2>        "base_url":"http://127.0.0.1:37737/orln",
> >    [junit4]   2>        "node_name":"127.0.0.1:37737_orln",
> >    [junit4]   2>        "numShards":"2",
> >    [junit4]   2>        "state":"active",
> >    [junit4]   2>        "shard":"shard1",
> >    [junit4]   2>        "collection":"control_collection",
> >    [junit4]   2>        "operation":"state"}
> >    [junit4]   2> 412609 T2362 oascc.ZkStateReader$2.process A cluster
> state
> > change: WatchedEvent state:SyncConnected type:NodeDataChanged
> > path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> >    [junit4]   2> 412609 T2351 oascc.ZkStateReader$2.process A cluster
> state
> > change: WatchedEvent state:SyncConnected type:NodeDataChanged
> > path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> >    [junit4]   2> 413587 T2325
> oasc.AbstractFullDistribZkTestBase.createJettys
> > create jetty 1
> >    [junit4]   2> 413587 T2325 oejs.Server.doStart jetty-8.1.10.v20130312
> >    [junit4]   2> 413590 T2325 oejs.AbstractConnector.doStart Started
> > SocketConnector@127.0.0.1:35189
> >    [junit4]   2> 413590 T2325 oass.SolrDispatchFilter.init
> SolrDispatchFilter.init()
> >    [junit4]   2> 413591 T2325 oasc.SolrResourceLoader.locateSolrHome JNDI
> > not configured for solr (NoInitialContextEx)
> >    [junit4]   2> 413591 T2325 oasc.SolrResourceLoader.locateSolrHome
> using
> > system property solr.solr.home: /usr/home/jenkins/jenkins-
> > slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-
> > core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> > 103A566D0461F871-001/tempDir-003
> >    [junit4]   2> 413591 T2325 oasc.SolrResourceLoader.<init> new
> > SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-
> > slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-
> > core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> > 103A566D0461F871-001/tempDir-003/'
> >    [junit4]   2> 413620 T2325 oasc.ConfigSolr.fromFile Loading container
> > configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-
> > Solr-Tests-trunk-Java8/solr/build/solr-
> > core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> > 103A566D0461F871-001/tempDir-003/solr.xml
> >    [junit4]   2> 413633 T2325 oasc.CoreContainer.<init> New CoreContainer
> > 751864104
> >    [junit4]   2> 413634 T2325 oasc.CoreContainer.load Loading cores into
> > CoreContainer [instanceDir=/usr/home/jenkins/jenkins-
> > slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-
> > core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> > 103A566D0461F871-001/tempDir-003/]
> >    [junit4]   2> 413634 T2325 oashc.HttpShardHandlerFactory.getParameter
> > Setting socketTimeout to: 90000
> >    [junit4]   2> 413634 T2325 oashc.HttpShardHandlerFactory.getParameter
> > Setting urlScheme to:
> >    [junit4]   2> 413635 T2325 oashc.HttpShardHandlerFactory.getParameter
> > Setting connTimeout to: 15000
> >    [junit4]   2> 413635 T2325 oashc.HttpShardHandlerFactory.getParameter
> > Setting maxConnectionsPerHost to: 20
> >    [junit4]   2> 413635 T2325 oashc.HttpShardHandlerFactory.getParameter
> > Setting maxConnections to: 10000
> >    [junit4]   2> 413635 T2325 oashc.HttpShardHandlerFactory.getParameter
> > Setting corePoolSize to: 0
> >    [junit4]   2> 413635 T2325 oashc.HttpShardHandlerFactory.getParameter
> > Setting maximumPoolSize to: 2147483647
> >    [junit4]   2> 413636 T2325 oashc.HttpShardHandlerFactory.getParameter
> > Setting maxThreadIdleTime to: 5
> >    [junit4]   2> 413636 T2325 oashc.HttpShardHandlerFactory.getParameter
> > Setting sizeOfQueue to: -1
> >    [junit4]   2> 413636 T2325 oashc.HttpShardHandlerFactory.getParameter
> > Setting fairnessPolicy to: false
> >    [junit4]   2> 413637 T2325 oasu.UpdateShardHandler.<init> Creating
> > UpdateShardHandler HTTP client with params:
> > socketTimeout=340000&connTimeout=45000&retry=false
> >    [junit4]   2> 413637 T2325 oasl.LogWatcher.createWatcher SLF4J impl is
> > org.slf4j.impl.Log4jLoggerFactory
> >    [junit4]   2> 413637 T2325 oasl.LogWatcher.newRegisteredLogWatcher
> > Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
> >    [junit4]   2> 413637 T2325 oasc.CoreContainer.load Host Name:
> 127.0.0.1
> >    [junit4]   2> 413638 T2325 oasc.ZkContainer.initZooKeeper Zookeeper
> > client=127.0.0.1:47646/solr
> >    [junit4]   2> 413638 T2325 oasc.ZkController.checkChrootPath zkHost
> > includes chroot
> >    [junit4]   2> 413638 T2325
> > oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default
> > ZkCredentialsProvider
> >    [junit4]   2> 413639 T2325 oascc.ConnectionManager.waitForConnected
> > Waiting for client to connect to ZooKeeper
> >    [junit4]   2> 413641 T2377 oascc.ConnectionManager.process Watcher
> > org.apache.solr.common.cloud.ConnectionManager@3b3bc7d
> > name:ZooKeeperConnection Watcher:127.0.0.1:47646 got event
> > WatchedEvent state:SyncConnected type:None path:null path:null
> > type:None
> >    [junit4]   2> 413641 T2325 oascc.ConnectionManager.waitForConnected
> > Client is connected to ZooKeeper
> >    [junit4]   2> 413642 T2325 oascc.SolrZkClient.createZkACLProvider
> Using
> > default ZkACLProvider
> >    [junit4]   2> 413644 T2325 oascc.ConnectionManager.waitForConnected
> > Waiting for client to connect to ZooKeeper
> >    [junit4]   2> 413645 T2380 oascc.ConnectionManager.process Watcher
> > org.apache.solr.common.cloud.ConnectionManager@4ed9fdbd
> > name:ZooKeeperConnection Watcher:127.0.0.1:47646/solr got event
> > WatchedEvent state:SyncConnected type:None path:null path:null
> > type:None
> >    [junit4]   2> 413645 T2325 oascc.ConnectionManager.waitForConnected
> > Client is connected to ZooKeeper
> >    [junit4]   2> 413649 T2325
> > oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating
> > cluster state from ZooKeeper...
> >    [junit4]   2> 414653 T2325 oasc.ZkController.createEphemeralLiveNode
> > Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35189_orln
> >    [junit4]   2> 414654 T2325 oascc.SolrZkClient.makePath makePath:
> > /live_nodes/127.0.0.1:35189_orln
> >    [junit4]   2> 414656 T2325 oasc.Overseer.close Overseer (id=null)
> closing
> >    [junit4]   2> 414662 T2381 oasc.ZkController.publish publishing
> > core=collection1 state=down collection=collection1
> >    [junit4]   2> 414662 T2381 oasc.ZkController.publish numShards not
> found
> > on descriptor - reading it from system property
> >    [junit4]   2> 414663 T2351 oasc.DistributedQueue$LatchWatcher.process
> > NodeChildrenChanged fired on path /overseer/queue state SyncConnected
> >    [junit4]   2> 414663 T2381 oasc.ZkController.preRegister Registering
> watch
> > for external collection collection1
> >    [junit4]   2> 414663 T2381 oascc.ZkStateReader.addZkWatch addZkWatch
> > collection1
> >    [junit4]   2> 414664 T2381 oascc.ZkStateReader.addZkWatch Updating
> > collection state at /collections/collection1/state.json from ZooKeeper...
> >    [junit4]   2> 414665 T2381 oascc.ZkStateReader.updateWatchedCollection
> > Updating data for collection1 to ver 0
> >    [junit4]   2> 414665 T2381 oasc.ZkController.waitForCoreNodeName look
> for
> > our core node name
> >    [junit4]   2> 414668 T2352
> oasc.Overseer$ClusterStateUpdater.updateState
> > Update state numShards=2 message={
> >    [junit4]   2>        "core":"collection1",
> >    [junit4]   2>        "roles":null,
> >    [junit4]   2>        "base_url":"http://127.0.0.1:35189/orln",
> >    [junit4]   2>        "node_name":"127.0.0.1:35189_orln",
> >    [junit4]   2>        "numShards":"2",
> >    [junit4]   2>        "state":"down",
> >    [junit4]   2>        "shard":null,
> >    [junit4]   2>        "collection":"collection1",
> >    [junit4]   2>        "operation":"state"}
> >    [junit4]   2> 414670 T2352
> oasc.Overseer$ClusterStateUpdater.updateState
> > Collection already exists with numShards=2
> >    [junit4]   2> 414671 T2352
> oasc.Overseer$ClusterStateUpdater.updateState
> > Assigning new node to shard shard=shard2
> >    [junit4]   2> 414673 T2351 oasc.DistributedQueue$LatchWatcher.process
> > NodeChildrenChanged fired on path /overseer/queue state SyncConnected
> >    [junit4]   2> 414674 T2352
> > oasc.Overseer$ClusterStateUpdater.updateZkStates going to
> > update_collection /collections/collection1/state.json
> >    [junit4]   2> 414674 T2380 oascc.ZkStateReader$7.process A cluster
> state
> > change: WatchedEvent state:SyncConnected type:NodeDataChanged
> > path:/collections/collection1/state.json, has occurred - updating...
> >    [junit4]   2> 414675 T2380 oascc.ZkStateReader.updateWatchedCollection
> > Updating data for collection1 to ver 1
> >    [junit4]   2> 415666 T2381 oasc.ZkController.waitForShardId waiting
> to find
> > shard id in clusterstate for collection1
> >    [junit4]   2> 415667 T2381 oasc.ZkController.createCollectionZkNode
> Check
> > for collection zkNode:collection1
> >    [junit4]   2> 415667 T2381 oasc.ZkController.createCollectionZkNode
> > Collection zkNode exists
> >    [junit4]   2> 415667 T2381 oascc.ZkStateReader.readConfigName Load
> > collection config from:/collections/collection1
> >    [junit4]   2> 415668 T2381 oascc.ZkStateReader.readConfigName
> > path=/collections/collection1 configName=conf1 specified config exists in
> > ZooKeeper
> >    [junit4]   2> 415668 T2381 oasc.SolrResourceLoader.<init> new
> > SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-
> > slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-
> > core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> > 103A566D0461F871-001/tempDir-003/collection1/'
> >    [junit4]   2> 415670 T2381 oasc.SolrResourceLoader.replaceClassLoader
> > Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-
> > trunk-Java8/solr/build/solr-
> > core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> > 103A566D0461F871-001/tempDir-003/collection1/lib/classes/' to classloader
> >    [junit4]   2> 415670 T2381 oasc.SolrResourceLoader.replaceClassLoader
> > Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-
> > trunk-Java8/solr/build/solr-
> > core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> > 103A566D0461F871-001/tempDir-003/collection1/lib/README' to classloader
> >    [junit4]   2> 415671 T2381 oasc.SolrResourceLoader.replaceClassLoader
> > Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-
> > trunk-Java8/solr/build/solr-
> > core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> > 103A566D0461F871-001/tempDir-003/collection1/lib/.svn/' to classloader
> >    [junit4]   2> 415702 T2381 oasc.ZkController.watchZKConfDir watch
> zkdir
> > /configs/conf1
> >    [junit4]   2> 415704 T2381 oasc.Config.<init> loaded config
> solrconfig.xml
> > with version 0
> >    [junit4]   2> 415718 T2381 oasc.SolrConfig.<init> Using Lucene
> > MatchVersion: 6.0.0
> >    [junit4]   2> 415729 T2381 oasc.SolrConfig.<init> Loaded SolrConfig:
> > solrconfig.xml
> >    [junit4]   2> 415831 T2381 oass.IndexSchema.readSchema Reading Solr
> > Schema from /configs/conf1/schema.xml
> >    [junit4]   2> 415837 T2381 oass.IndexSchema.readSchema [collection1]
> > Schema name=test
> >    [junit4]   2> 415952 T2381 oass.OpenExchangeRatesOrgProvider.init
> > Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
> >    [junit4]   2> 415962 T2381 oass.IndexSchema.readSchema default search
> > field in schema is text
> >    [junit4]   2> 415964 T2381 oass.IndexSchema.readSchema unique key
> field:
> > id
> >    [junit4]   2> 415971 T2381 oass.FileExchangeRateProvider.reload
> Reloading
> > exchange rates from file currency.xml
> >    [junit4]   2> 415974 T2381 oass.FileExchangeRateProvider.reload
> Reloading
> > exchange rates from file currency.xml
> >    [junit4]   2> 415976 T2381 oass.OpenExchangeRatesOrgProvider.reload
> > Reloading exchange rates from open-exchange-rates.json
> >    [junit4]   2> 415977 T2381
> > oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN
> > Unknown key IMPORTANT NOTE
> >    [junit4]   2> 415978 T2381
> > oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN
> > Expected key, got STRING
> >    [junit4]   2> 415978 T2381 oass.OpenExchangeRatesOrgProvider.reload
> > Reloading exchange rates from open-exchange-rates.json
> >    [junit4]   2> 415978 T2381
> > oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN
> > Unknown key IMPORTANT NOTE
> >    [junit4]   2> 415979 T2381
> > oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN
> > Expected key, got STRING
> >    [junit4]   2> 415979 T2381 oasc.CoreContainer.create Creating SolrCore
> > 'collection1' using configuration from collection collection1
> >    [junit4]   2> 415979 T2381 oasc.SolrCore.initDirectoryFactory
> > org.apache.solr.core.MockDirectoryFactory
> >    [junit4]   2> 415979 T2381 oasc.SolrCore.<init> [collection1] Opening
> new
> > SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-
> > trunk-Java8/solr/build/solr-
> > core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> > 103A566D0461F871-001/tempDir-003/collection1/,
> > dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-
> > trunk-Java8/solr/build/solr-
> > core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> > 103A566D0461F871-001/tempDir-001/jetty1/
> >    [junit4]   2> 415980 T2381 oasc.JmxMonitoredMap.<init> JMX monitoring
> is
> > enabled. Adding Solr mbeans to JMX Server:
> > com.sun.jmx.mbeanserver.JmxMBeanServer@330fda4c
> >    [junit4]   2> 415981 T2381 oasc.CachingDirectoryFactory.get return new
> > directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-
> > Tests-trunk-Java8/solr/build/solr-
> > core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> > 103A566D0461F871-001/tempDir-001/jetty1
> >    [junit4]   2> 415981 T2381 oasc.SolrCore.getNewIndexDir New index
> > directory detected: old=null new=/usr/home/jenkins/jenkins-
> > slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-
> > core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> > 103A566D0461F871-001/tempDir-001/jetty1/index/
> >    [junit4]   2> 415981 T2381 oasc.SolrCore.initIndex WARN [collection1]
> Solr
> > index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-
> > Tests-trunk-Java8/solr/build/solr-
> > core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> > 103A566D0461F871-001/tempDir-001/jetty1/index' doesn't exist. Creating
> > new index...
> >    [junit4]   2> 415981 T2381 oasc.CachingDirectoryFactory.get return new
> > directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-
> > Tests-trunk-Java8/solr/build/solr-
> > core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> > 103A566D0461F871-001/tempDir-001/jetty1/index
> >    [junit4]   2> 415982 T2381 oasu.RandomMergePolicy.<init>
> > RandomMergePolicy wrapping class
> > org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy:
> > maxMergeAtOnce=30, maxMergeAtOnceExplicit=41,
> > maxMergedSegmentMB=64.537109375, floorSegmentMB=1.693359375,
> > forceMergeDeletesPctAllowed=3.111209946074762, segmentsPerTier=41.0,
> > maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
> >    [junit4]   2> 415983 T2381 oasc.SolrDeletionPolicy.onCommit
> > SolrDeletionPolicy.onCommit: commits: num=1
> >    [junit4]   2>
> >       commit{dir=MockDirectoryWrapper(RAMDirectory@1e365fa1
> > lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@66e9bbe),
> > segFN=segments_1,generation=1}
> >    [junit4]   2> 415983 T2381 oasc.SolrDeletionPolicy.updateCommits
> newest
> > commit generation = 1
> >    [junit4]   2> 415987 T2381 oasup.UpdateRequestProcessorChain.init
> > creating updateRequestProcessorChain "nodistrib"
> >    [junit4]   2> 415987 T2381 oasup.UpdateRequestProcessorChain.init
> > creating updateRequestProcessorChain "dedupe"
> >    [junit4]   2> 415987 T2381 oasup.UpdateRequestProcessorChain.init
> > inserting DistributedUpdateProcessorFactory into
> > updateRequestProcessorChain "dedupe"
> >    [junit4]   2> 415988 T2381 oasup.UpdateRequestProcessorChain.init
> > creating updateRequestProcessorChain "stored_sig"
> >    [junit4]   2> 415988 T2381 oasup.UpdateRequestProcessorChain.init
> > inserting DistributedUpdateProcessorFactory into
> > updateRequestProcessorChain "stored_sig"
> >    [junit4]   2> 415988 T2381 oasup.UpdateRequestProcessorChain.init
> > creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
> >    [junit4]   2> 415989 T2381 oasup.UpdateRequestProcessorChain.init
> > creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
> >    [junit4]   2> 415989 T2381 oasup.UpdateRequestProcessorChain.init
> > inserting DistributedUpdateProcessorFactory into
> > updateRequestProcessorChain "distrib-dup-test-chain-implicit"
> >    [junit4]   2> 415989 T2381 oasc.SolrCore.loadUpdateProcessorChains no
> > updateRequestProcessorChain defined as default, creating implicit default
> >    [junit4]   2> 415990 T2381 oasc.RequestHandlers.initHandlersFromConfig
> > created /config: org.apache.solr.handler.SolrConfigHandler
> >    [junit4]   2> 415990 T2381 oasc.RequestHandlers.initHandlersFromConfig
> > created /update/csv: org.apache.solr.handler.UpdateRequestHandler
> >    [junit4]   2> 415990 T2381 oasc.RequestHandlers.initHandlersFromConfig
> > created /update/json: org.apache.solr.handler.UpdateRequestHandler
> >    [junit4]   2> 415991 T2381 oasc.RequestHandlers.initHandlersFromConfig
> > created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
> >    [junit4]   2> 415991 T2381 oasc.RequestHandlers.initHandlersFromConfig
> > created standard: solr.StandardRequestHandler
> >    [junit4]   2> 415992 T2381 oasc.RequestHandlers.initHandlersFromConfig
> > created /get: solr.RealTimeGetHandler
> >    [junit4]   2> 415992 T2381 oasc.RequestHandlers.initHandlersFromConfig
> > adding lazy requestHandler: solr.ReplicationHandler
> >    [junit4]   2> 415992 T2381 oasc.RequestHandlers.initHandlersFromConfig
> > created /replication: solr.ReplicationHandler
> >    [junit4]   2> 415993 T2381 oasc.RequestHandlers.initHandlersFromConfig
> > created /update: solr.UpdateRequestHandler
> >    [junit4]   2> 415993 T2381 oasc.RequestHandlers.initHandlersFromConfig
> > created /admin/: org.apache.solr.handler.admin.AdminHandlers
> >    [junit4]   2> 415993 T2381 oasc.RequestHandlers.initHandlersFromConfig
> > created /admin/file: solr.admin.ShowFileRequestHandler
> >    [junit4]   2> 415994 T2381 oasc.RequestHandlers.initHandlersFromConfig
> > created /admin/fileedit: solr.admin.EditFileRequestHandler
> >    [junit4]   2> 416004 T2381 oashl.XMLLoader.init
> > xsltCacheLifetimeSeconds=60
> >    [junit4]   2> 416005 T2381 oashl.XMLLoader.init
> > xsltCacheLifetimeSeconds=60
> >    [junit4]   2> 416006 T2381 oashl.XMLLoader.init
> > xsltCacheLifetimeSeconds=60
> >    [junit4]   2> 416008 T2381 oashl.XMLLoader.init
> > xsltCacheLifetimeSeconds=60
> >    [junit4]   2> 416011 T2381 oasu.CommitTracker.<init> Hard AutoCommit:
> > disabled
> >    [junit4]   2> 416011 T2381 oasu.CommitTracker.<init> Soft AutoCommit:
> > disabled
> >    [junit4]   2> 416012 T2381 oasu.RandomMergePolicy.<init>
> > RandomMergePolicy wrapping class
> > org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy:
> > maxMergeAtOnce=16, maxMergeAtOnceExplicit=15,
> > maxMergedSegmentMB=60.369140625, floorSegmentMB=1.87109375,
> > forceMergeDeletesPctAllowed=16.64506491494333, segmentsPerTier=38.0,
> > maxCFSSegmentSizeMB=8.796093022207999E12,
> > noCFSRatio=0.3115152765900482
> >    [junit4]   2> 416012 T2381 oasc.SolrDeletionPolicy.onInit
> > SolrDeletionPolicy.onInit: commits: num=1
> >    [junit4]   2>
> >       commit{dir=MockDirectoryWrapper(RAMDirectory@1e365fa1
> > lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@66e9bbe),
> > segFN=segments_1,generation=1}
> >    [junit4]   2> 416013 T2381 oasc.SolrDeletionPolicy.updateCommits
> newest
> > commit generation = 1
> >    [junit4]   2> 416013 T2381 oass.SolrIndexSearcher.<init> Opening
> > Searcher@1e3921a8[collection1] main
> >    [junit4]   2> 416013 T2381 oascc.ZkStateReader.readConfigName Load
> > collection config from:/collections/collection1
> >    [junit4]   2> 416014 T2381 oascc.ZkStateReader.readConfigName
> > path=/collections/collection1 configName=conf1 specified config exists in
> > ZooKeeper
> >    [junit4]   2> 416014 T2381 oasr.ManagedResourceStorage.newStorageIO
> > Setting up ZooKeeper-based storage for the RestManager with znodeBase:
> > /configs/conf1
> >    [junit4]   2> 416014 T2381
> > oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured
> > ZooKeeperStorageIO with znodeBase: /configs/conf1
> >    [junit4]   2> 416015 T2381 oasr.RestManager.init Initializing
> RestManager
> > with initArgs: {}
> >    [junit4]   2> 416015 T2381 oasr.ManagedResourceStorage.load Reading
> > _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
> >    [junit4]   2> 416016 T2381
> > oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream
> > Read 38 bytes from znode /configs/conf1/_rest_managed.json
> >    [junit4]   2> 416016 T2381 oasr.ManagedResourceStorage.load Loaded
> > LinkedHashMap at path _rest_managed.json using
> > ZooKeeperStorageIO:path=/configs/conf1
> >    [junit4]   2> 416016 T2381 oasr.ManagedResource.reloadFromStorage
> > Loaded initArgs {} for /rest/managed
> >    [junit4]   2> 416016 T2381 oasr.RestManager.init Initializing 0
> registered
> > ManagedResources
> >    [junit4]   2> 416019 T2382 oasc.SolrCore.registerSearcher
> [collection1]
> > Registered new searcher Searcher@1e3921a8[collection1]
> > main{ExitableDirectoryReader(UninvertingDirectoryReader())}
> >    [junit4]   2> 416020 T2381 oasc.CoreContainer.registerCore
> registering core:
> > collection1
> >    [junit4]   2> 416020 T2385 oasc.ZkController.register Register
> replica -
> > core:collection1 address:http://127.0.0.1:35189/orln
> collection:collection1
> > shard:shard2
> >    [junit4]   2> 416021 T2325 oass.SolrDispatchFilter.init
> > user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-
> > trunk-Java8/solr/build/solr-core/test/J3
> >    [junit4]   2> 416021 T2325 oass.SolrDispatchFilter.init
> SolrDispatchFilter.init()
> > done
> >    [junit4]   2> 416021 T2385 oascc.SolrZkClient.makePath makePath:
> > /collections/collection1/leader_elect/shard2/election
> >    [junit4]   2> 416025 T2385
> > oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader
> > process for shard shard2
> >    [junit4]   2> 416026 T2351 oasc.DistributedQueue$LatchWatcher.process
> > NodeChildrenChanged fired on path /overseer/queue state SyncConnected
> >    [junit4]   2> 416026 T2385
> > oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough
> > replicas found to continue.
> >    [junit4]   2> 416026 T2385
> > oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader
> > - try and sync
> >    [junit4]   2> ASYNC  NEW_CORE C137 name=collection1
> > org.apache.solr.core.SolrCore@4e55ac0b
> > url=http://127.0.0.1:35189/orln/collection1 node=127.0.0.1:35189_orln
> > C137_STATE=coll:collection1 core:collection1 props:{core=collection1,
> > base_url=http://127.0.0.1:35189/orln, node_name=127.0.0.1:35189_orln,
> > state=down}
> >    [junit4]   2> 416026 T2385 C137 P35189 oasc.SyncStrategy.sync Sync
> replicas
> > to http://127.0.0.1:35189/orln/collection1/
> >    [junit4]   2> 416027 T2385 C137 P35189 oasc.SyncStrategy.syncReplicas
> Sync
> > Success - now sync replicas to me
> >    [junit4]   2> 416027 T2385 C137 P35189 oasc.SyncStrategy.syncToMe
> > http://127.0.0.1:35189/orln/collection1/ has no replicas
> >    [junit4]   2> 416027 T2385
> > oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader:
> > http://127.0.0.1:35189/orln/collection1/ shard2
> >    [junit4]   2> 416027 T2385 oascc.SolrZkClient.makePath makePath:
> > /collections/collection1/leaders/shard2
> >    [junit4]   2> 416030 T2352
> > oasc.Overseer$ClusterStateUpdater.updateZkStates going to
> > update_collection /collections/collection1/state.json
> >    [junit4]   2> 416030 T2380 oascc.ZkStateReader$7.process A cluster
> state
> > change: WatchedEvent state:SyncConnected type:NodeDataChanged
> > path:/collections/collection1/state.json, has occurred - updating...
> >    [junit4]   2> 416031 T2380 oascc.ZkStateReader.updateWatchedCollection
> > Updating data for collection1 to ver 2
> >    [junit4]   2> 416035 T2351 oasc.DistributedQueue$LatchWatcher.process
> > NodeChildrenChanged fired on path /overseer/queue state SyncConnected
> >    [junit4]   2> 416035 T2352
> > oasc.Overseer$ClusterStateUpdater.updateZkStates going to
> > update_collection /collections/collection1/state.json
> >    [junit4]   2> 416035 T2380 oascc.ZkStateReader$7.process A cluster
> state
> > change: WatchedEvent state:SyncConnected type:NodeDataChanged
> > path:/collections/collection1/state.json, has occurred - updating...
> >    [junit4]   2> 416036 T2380 oascc.ZkStateReader.updateWatchedCollection
> > Updating data for collection1 to ver 3
> >    [junit4]   2> 416081 T2385 oasc.ZkController.register We are
> > http://127.0.0.1:35189/orln/collection1/ and leader is
> > http://127.0.0.1:35189/orln/collection1/
> >    [junit4]   2> 416082 T2385 oasc.ZkController.register No LogReplay
> needed
> > for core=collection1 baseURL=http://127.0.0.1:35189/orln
> >    [junit4]   2> 416082 T2385 oasc.ZkController.checkRecovery I am the
> leader,
> > no recovery necessary
> >    [junit4]   2> 416082 T2385 oasc.ZkController.publish publishing
> > core=collection1 state=active collection=collection1
> >    [junit4]   2> 416082 T2385 oasc.ZkController.publish numShards not
> found
> > on descriptor - reading it from system property
> >    [junit4]   2> 416083 T2351 oasc.DistributedQueue$LatchWatcher.process
> > NodeChildrenChanged fired on path /overseer/queue state SyncConnected
> >    [junit4]   2> 416085 T2385 oascc.ZkStateReader.updateWatchedCollection
> > Updating data for collection1 to ver 3
> >    [junit4]   2> 416085 T2352
> oasc.Overseer$ClusterStateUpdater.updateState
> > Update state numShards=2 message={
> >    [junit4]   2>        "core":"collection1",
> >    [junit4]   2>        "core_node_name":"core_node1",
> >    [junit4]   2>        "roles":null,
> >    [junit4]   2>        "base_url":"http://127.0.0.1:35189/orln",
> >    [junit4]   2>        "node_name":"127.0.0.1:35189_orln",
> >    [junit4]   2>        "numShards":"2",
> >    [junit4]   2>        "state":"active",
> >    [junit4]   2>        "shard":"shard2",
> >    [junit4]   2>        "collection":"collection1",
> >    [junit4]   2>        "operation":"state"}
> >    [junit4]   2> 416088 T2352
> > oasc.Overseer$ClusterStateUpdater.updateZkStates going to
> > update_collection /collections/collection1/state.json
> >    [junit4]   2> 416089 T2380 oascc.ZkStateReader$7.process A cluster
> state
> > change: WatchedEvent state:SyncConnected type:NodeDataChanged
> > path:/collections/collection1/state.json, has occurred - updating...
> >    [junit4]   2> 416089 T2380 oascc.ZkStateReader.updateWatchedCollection
> > Updating data for collection1 to ver 4
> >    [junit4]   2> 417139 T2325
> oasc.AbstractFullDistribZkTestBase.createJettys
> > create jetty 2
> >    [junit4]   2> 417140 T2325 oejs.Server.doStart jetty-8.1.10.v20130312
> >    [junit4]   2> 417142 T2325 oejs.AbstractConnector.doStart Started
> > SocketConnector@127.0.0.1:28438
> >    [junit4]   2> 417143 T2325 oass.SolrDispatchFilter.init
> SolrDispatchFilter.init()
> >    [junit4]   2> 417143 T2325 oasc.SolrResourceLoader.locateSolrHome JNDI
> > not configured for solr (NoInitialContextEx)
> >    [junit4]   2> 417143 T2325 oasc.SolrResourceLoader.locateSolrHome
> using
> > system property solr.solr.home:
> > ../../../../../../../../../../../usr/home/jenkins/jenkins-
> > slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-
> > core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> > 103A566D0461F871-001/tempDir-004
> >    [junit4]   2> 417143 T2325 oasc.SolrResourceLoader.<init> new
> > SolrResourceLoader for directory:
> > '../../../../../../../../../../../usr/home/jenkins/jenkins-
> > slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-
> > core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> > 103A566D0461F871-001/tempDir-004/'
> >    [junit4]   2> 417173 T2325 oasc.ConfigSolr.fromFile Loading container
> > configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-
> > Solr-Tests-trunk-Java8/solr/build/solr-
> > core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-
> > slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-
> > core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> > 103A566D0461F871-001/tempDir-004/solr.xml
> >    [junit4]   2> 417186 T2325 oasc.CoreContainer.<init> New CoreContainer
> > 967093745
> >    [junit4]   2> 417187 T2325 oasc.CoreContainer.load Loading cores into
> > CoreContainer
> > [instanceDir=../../../../../../../../../../../usr/home/jenkins/jenkins-
> > slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-
> > core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> > 103A566D0461F871-001/tempDir-004/]
> >    [junit4]   2> 417187 T2325 oashc.HttpShardHandlerFactory.getParameter
> > Setting socketTimeout to: 90000
> >    [junit4]   2> 417187 T2325 oashc.HttpShardHandlerFactory.getParameter
> > Setting urlScheme to:
> >    [junit4]   2> 417188 T2325 oashc.HttpShardHandlerFactory.getParameter
> > Setting connTimeout to: 15000
> >    [junit4]   2> 417188 T2325 oashc.HttpShardHandlerFactory.getParameter
> > Setting maxConnectionsPerHost to: 20
> >    [junit4]   2> 417188 T2325 oashc.HttpShardHandlerFactory.getParameter
> > Setting maxConnections to: 10000
> >    [junit4]   2> 417188 T2325 oashc.HttpShardHandlerFactory.getParameter
> > Setting corePoolSize to: 0
> >    [junit4]   2> 417189 T2325 oashc.HttpShardHandlerFactory.getParameter
> > Setting maximumPoolSize to: 2147483647
> >    [junit4]   2> 417189 T2325 oashc.HttpShardHandlerFactory.getParameter
> > Setting maxThreadIdleTime to: 5
> >    [junit4]   2> 417189 T2325 oashc.HttpShardHandlerFactory.getParameter
> > Setting sizeOfQueue to: -1
> >    [junit4]   2> 417189 T2325 oashc.HttpShardHandlerFactory.getParameter
> > Setting fairnessPolicy to: false
> >    [junit4]   2> 417190 T2325 oasu.UpdateShardHandler.<init> Creating
> > UpdateShardHandler HTTP client with params:
> > socketTimeout=340000&connTimeout=45000&retry=false
> >    [junit4]   2> 417190 T2325 oasl.LogWatcher.createWatcher SLF4J impl is
> > org.slf4j.impl.Log4jLoggerFactory
> >    [junit4]   2> 417190 T2325 oasl.LogWatcher.newRegisteredLogWatcher
> > Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
> >    [junit4]   2> 417190 T2325 oasc.CoreContainer.load Host Name:
> 127.0.0.1
> >    [junit4]   2> 417191 T2325 oasc.ZkContainer.initZooKeeper Zookeeper
> > client=127.0.0.1:47646/solr
> >    [junit4]   2> 417191 T2325 oasc.ZkController.checkChrootPath zkHost
> > includes chroot
> >    [junit4]   2> 417191 T2325
> > oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default
> > ZkCredentialsProvider
> >    [junit4]   2> 417192 T2325 oascc.ConnectionManager.waitForConnected
> > Waiting for client to connect to ZooKeeper
> >    [junit4]   2> 417194 T2397 oascc.ConnectionManager.process Watcher
> > org.apache.solr.common.cloud.ConnectionManager@d035dfa
> > name:ZooKeeperConnection Watcher:127.0.0.1:47646 got event
> > WatchedEvent state:SyncConnected type:None path:null path:null
> > type:None
> >    [junit4]   2> 417195 T2325 oascc.ConnectionManager.waitForConnected
> > Client is connected to ZooKeeper
> >    [junit4]   2> 417195 T2325 oascc.SolrZkClient.createZkACLProvider
> Using
> > default ZkACLProvider
> >    [junit4]   2> 417197 T2325 oascc.ConnectionManager.waitForConnected
> > Waiting for client to connect to ZooKeeper
> >    [junit4]   2> 417199 T2400 oascc.ConnectionManager.process Watcher
> > org.apache.solr.common.cloud.ConnectionManager@5f3d26d0
> > name:ZooKeeperConnection Watcher:127.0.0.1:47646/solr got event
> > WatchedEvent state:SyncConnected type:None path:null path:null
> > type:None
> >    [junit4]   2> 417199 T2325 oascc.ConnectionManager.waitForConnected
> > Client is connected to ZooKeeper
> >    [junit4]   2> 417203 T2325
> > oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating
> > cluster state from ZooKeeper...
> >    [junit4]   2> 418206 T2325 oasc.ZkController.createEphemeralLiveNode
> > Register node as live in ZooKeeper:/live_nodes/127.0.0.1:28438_orln
> >    [junit4]   2> 418207 T2325 oascc.SolrZkClient.makePath makePath:
> > /live_nodes/127.0.0.1:28438_orln
> >    [junit4]   2> 418210 T2325 oasc.Overseer.close Overseer (id=null)
> closing
> >    [junit4]   2> 418215 T2401 oasc.ZkController.publish publishing
> > core=collection1 state=down collection=collection1
> >    [junit4]   2> 418215 T2401 oasc.ZkController.publish numShards not
> found
> > on descriptor - reading it from system property
> >    [junit4]   2> 418216 T2351 oasc.DistributedQueue$LatchWatcher.process
> > NodeChildrenChanged fired on path /overseer/queue state SyncConnected
> >    [junit4]   2> 418216 T2401 oasc.ZkController.preRegister Registering
> watch
> > for external collection collection1
> >    [junit4]   2> 418217 T2401 oascc.ZkStateReader.addZkWatch addZkWatch
> > collection1
> >    [junit4]   2> 418217 T2401 oascc.ZkStateReader.addZkWatch Updating
> > collection state at /collections/collection1/state.json from ZooKeeper...
> >    [junit4]   2> 418218 T2401 oascc.ZkStateReader.updateWatchedCollection
> > Updating data for collection1 to ver 4
> >    [junit4]   2> 418218 T2401 oasc.ZkController.waitForCoreNodeName look
> for
> > our core node name
> >    [junit4]   2> 418218 T2352
> oasc.Overseer$ClusterStateUpdater.updateState
> > Update state numShards=2 message={
> >    [junit4]   2>        "core":"collection1",
> >    [junit4]   2>        "roles":null,
> >    [junit4]   2>        "base_url":"http://127.0.0.1:28438/orln",
> >    [junit4]   2>        "node_name":"127.0.0.1:28438_orln",
> >    [junit4]   2>        "numShards":"2",
> >    [junit4]   2>        "state":"down",
> >    [junit4]   2>        "shard":null,
> >    [junit4]   2>        "collection":"collection1",
> >    [junit4]   2>        "operation":"state"}
> >    [junit4]   2> 418220 T2352
> oasc.Overseer$ClusterStateUpdater.updateState
> > Collection already exists with numShards=2
> >    [junit4]   2> 418220 T2352
> oasc.Overseer$ClusterStateUpdater.updateState
> > Assigning new node to shard shard=shard1
> >    [junit4]   2> 418223 T2352
> > oasc.Overseer$ClusterStateUpdater.updateZkStates going to
> > update_collection /collections/collection1/state.json
> >    [junit4]   2> 418224 T2400 oascc.ZkStateReader$7.process A cluster
> state
> > change: WatchedEvent state:SyncConnected type:NodeDataChanged
> > path:/collections/collection1/state.json, has occurred - updating...
> >    [junit4]   2> 418224 T2380 oascc.ZkStateReader$7.process A cluster
> state
> > change: WatchedEvent state:SyncConnected type:NodeDataChanged
> > path:/collections/collection1/state.json, has occurred - updating...
> >    [junit4]   2> 418224 T2400 oascc.ZkStateReader.updateWatchedCollection
> > Updating data for collection1 to ver 5
> >    [junit4]   2> 418225 T2380 oascc.ZkStateReader.updateWatchedCollection
> > Updating data for collection1 to ver 5
> >    [junit4]   2> 419218 T2401 oasc.ZkController.waitForShardId waiting
> to find
> > shard id in clusterstate for collection1
> >    [junit4]   2> 419219 T2401 oasc.ZkController.createCollectionZkNode
> Check
> > for collection zkNode:collection1
> >    [junit4]   2> 419219 T2401 oasc.ZkController.createCollectionZkNode
> > Collection zkNode exists
> >    [junit4]   2> 419220 T2401 oascc.ZkStateReader.readConfigName Load
> > collection config from:/collections/collection1
> >    [junit4]   2> 419220 T2401 oascc.ZkStateReader.readConfigName
> > path=/collections/collection1 configName=conf1 specified config exists in
> > ZooKeeper
> >    [junit4]   2> 419220 T2401 oasc.SolrResourceLoader.<init> new
> > SolrResourceLoader for directory:
> > '../../../../../../../../../../../usr/home/jenkins/jenkins-
> > slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-
> > core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> > 103A566D0461F871-001/tempDir-004/collection1/'
> >    [junit4]   2> 419222 T2401 oasc.SolrResourceLoader.replaceClassLoader
> > Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-
> > trunk-Java8/solr/build/solr-
> > core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> > 103A566D0461F871-001/tempDir-004/collection1/lib/README' to classloader
> >    [junit4]   2> 419223 T2401 oasc.SolrResourceLoader.replaceClassLoader
> > Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-
> > trunk-Java8/solr/build/solr-
> > core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> > 103A566D0461F871-001/tempDir-004/collection1/lib/.svn/' to classloader
> >    [junit4]   2> 419224 T2401 oasc.SolrResourceLoader.replaceClassLoader
> > Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-
> > trunk-Java8/solr/build/solr-
> > core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> > 103A566D0461F871-001/tempDir-004/collection1/lib/classes/' to classloader
> >    [junit4]   2> 419256 T2401 oasc.ZkController.watchZKConfDir watch
> zkdir
> > /configs/conf1
> >    [junit4]   2> 419258 T2401 oasc.Config.<init> loaded config
> solrconfig.xml
> > with version 0
> >    [junit4]   2> 419273 T2401 oasc.SolrConfig.<init> Using Lucene
> > MatchVersion: 6.0.0
> >    [junit4]   2> 419285 T2401 oasc.SolrConfig.<init> Loaded SolrConfig:
> > solrconfig.xml
> >    [junit4]   2> 419386 T2401 oass.IndexSchema.readSchema Reading Solr
> > Schema from /configs/conf1/schema.xml
> >    [junit4]   2> 419392 T2401 oass.IndexSchema.readSchema [collection1]
> > Schema name=test
> >    [junit4]   2> 419507 T2401 oass.OpenExchangeRatesOrgProvider.init
> > Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
> >    [junit4]   2> 419517 T2401 oass.IndexSchema.readSchema default search
> > field in schema is text
> >    [junit4]   2> 419518 T2401 oass.IndexSchema.readSchema unique key
> field:
> > id
> >    [junit4]   2> 419526 T2401 oass.FileExchangeRateProvider.reload
> Reloading
> > exchange rates from file currency.xml
> >    [junit4]   2> 419529 T2401 oass.FileExchangeRateProvider.reload
> Reloading
> > exchange rates from file currency.xml
> >    [junit4]   2> 419531 T2401 oass.OpenExchangeRatesOrgProvider.reload
> > Reloading exchange rates from open-exchange-rates.json
> >    [junit4]   2> 419532 T2401
> > oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN
> > Unknown key IMPORTANT NOTE
> >    [junit4]   2> 419533 T2401
> > oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN
> > Expected key, got STRING
> >    [junit4]   2> 419533 T2401 oass.OpenExchangeRatesOrgProvider.reload
> > Reloading exchange rates from open-exchange-rates.json
> >    [junit4]   2> 419533 T2401
> > oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN
> > Unknown key IMPORTANT NOTE
> >    [junit4]   2> 419534 T2401
> > oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN
> > Expected key, got STRING
> >    [junit4]   2> 419534 T2401 oasc.CoreContainer.create Creating SolrCore
> > 'collection1' using configuration from collection collection1
> >    [junit4]   2> 419534 T2401 oasc.SolrCore.initDirectoryFactory
> > org.apache.solr.core.MockDirectoryFactory
> >    [junit4]   2> 419534 T2401 oasc.SolrCore.<init> [collection1] Opening
> new
> > SolrCore at ../../../../../../../../../../../usr/home/jenkins/jenkins-
> > slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-
> > core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> > 103A566D0461F871-001/tempDir-004/collection1/,
> > dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-
> > trunk-Java8/solr/build/solr-
> > core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> > 103A566D0461F871-001/tempDir-001/jetty2/
> >    [junit4]   2> 419535 T2401 oasc.JmxMonitoredMap.<init> JMX monitoring
> is
> > enabled. Adding Solr mbeans to JMX Server:
> > com.sun.jmx.mbeanserver.JmxMBeanServer@330fda4c
> >    [junit4]   2> 419536 T2401 oasc.CachingDirectoryFactory.get return new
> > directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-
> > Tests-trunk-Java8/solr/build/solr-
> > core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> > 103A566D0461F871-001/tempDir-001/jetty2
> >    [junit4]   2> 419536 T2401 oasc.SolrCore.getNewIndexDir New index
> > directory detected: old=null new=/usr/home/jenkins/jenkins-
> > slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-
> > core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> > 103A566D0461F871-001/tempDir-001/jetty2/index/
> >    [junit4]   2> 419536 T2401 oasc.SolrCore.initIndex WARN [collection1]
> Solr
> > index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-
> > Tests-trunk-Java8/solr/build/solr-
> > core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> > 103A566D0461F871-001/tempDir-001/jetty2/index' doesn't exist. Creating
> > new index...
> >    [junit4]   2> 419537 T2401 oasc.CachingDirectoryFactory.get return new
> > directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-
> > Tests-trunk-Java8/solr/build/solr-
> > core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> > 103A566D0461F871-001/tempDir-001/jetty2/index
> >    [junit4]   2> 419537 T2401 oasu.RandomMergePolicy.<init>
> > RandomMergePolicy wrapping class
> > org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy:
> > maxMergeAtOnce=30, maxMergeAtOnceExplicit=41,
> > maxMergedSegmentMB=64.537109375, floorSegmentMB=1.693359375,
> > forceMergeDeletesPctAllowed=3.111209946074762, segmentsPerTier=41.0,
> > maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
> >    [junit4]   2> 419538 T2401 oasc.SolrDeletionPolicy.onCommit
> > SolrDeletionPolicy.onCommit: commits: num=1
> >    [junit4]   2>
> >       commit{dir=MockDirectoryWrapper(RAMDirectory@4ad441db
> > lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@505677e5),
> > segFN=segments_1,generation=1}
> >    [junit4]   2> 419538 T2401 oasc.SolrDeletionPolicy.updateCommits
> newest
> > commit generation = 1
> >    [junit4]   2> 419542 T2401 oasup.UpdateRequestProcessorChain.init
> > creating updateRequestProcessorChain "nodistrib"
> >    [junit4]   2> 419543 T2401 oasup.UpdateRequestProcessorChain.init
> > creating updateRequestProcessorChain "dedupe"
> >    [junit4]   2> 419543 T2401 oasup.UpdateRequestProcessorChain.init
> > inserting DistributedUpdateProcessorFactory into
> > updateRequestProcessorChain "dedupe"
> >    [junit4]   2> 419543 T2401 oasup.UpdateRequestProcessorChain.init
> > creating updateRequestProcessorChain "stored_sig"
> >    [junit4]   2> 419543 T2401 oasup.UpdateRequestProcessorChain.init
> > inserting DistributedUpdateProcessorFactory into
> > updateRequestProcessorChain "stored_sig"
> >    [junit4]   2> 419544 T2401 oasup.UpdateRequestProcessorChain.init
> > creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
> >    [junit4]   2> 419544 T2401 oasup.UpdateRequestProcessorChain.init
> > creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
> >    [junit4]   2> 419544 T2401 oasup.UpdateRequestProcessorChain.init
> > inserting DistributedUpdateProcessorFactory into
> > updateRequestProcessorChain "distrib-dup-test-chain-implicit"
> >    [junit4]   2> 419545 T2401 oasc.SolrCore.loadUpdateProcessorChains no
> > updateRequestProcessorChain defined as default, creating implicit default
> >    [junit4]   2> 419545 T2401 oasc.RequestHandlers.initHandlersFromConfig
> > created /config: org.apache.solr.handler.SolrConfigHandler
> >    [junit4]   2> 419546 T2401 oasc.RequestHandlers.initHandlersFromConfig
> > created /update/csv: org.apache.solr.handler.UpdateRequestHandler
> >    [junit4]   2> 419546 T2401 oasc.RequestHandlers.initHandlersFromConfig
> > created /update/json: org.apache.solr.handler.UpdateRequestHandler
> >    [junit4]   2> 419546 T2401 oasc.RequestHandlers.initHandlersFromConfig
> > created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
> >    [junit4]   2> 419547 T2401 oasc.RequestHandlers.initHandlersFromConfig
> > created standard: solr.StandardRequestHandler
> >    [junit4]   2> 419548 T2401 oasc.RequestHandlers.initHandlersFromConfig
> > created /get: solr.RealTimeGetHandler
> >    [junit4]   2> 419548 T2401 oasc.RequestHandlers.initHandlersFromConfig
> > adding lazy requestHandler: solr.ReplicationHandler
> >    [junit4]   2> 419548 T2401 oasc.RequestHandlers.initHandlersFromConfig
> > created /replication: solr.ReplicationHandler
> >    [junit4]   2> 419548 T2401 oasc.RequestHandlers.initHandlersFromConfig
> > created /update: solr.UpdateRequestHandler
> >    [junit4]   2> 419549 T2401 oasc.RequestHandlers.initHandlersFromConfig
> > created /admin/: org.apache.solr.handler.admin.AdminHandlers
> >    [junit4]   2> 419549 T2401 oasc.RequestHandlers.initHandlersFromConfig
> > created /admin/file: solr.admin.ShowFileRequestHandler
> >    [junit4]   2> 419550 T2401 oasc.RequestHandlers.initHandlersFromConfig
> > created /admin/fileedit: solr.admin.EditFileRequestHandler
> >    [junit4]   2> 419560 T2401 oashl.XMLLoader.init
> > xsltCacheLifetimeSeconds=60
> >    [junit4]   2> 419562 T2401 oashl.XMLLoader.init
> > xsltCacheLifetimeSeconds=60
> >    [junit4]   2> 419563 T2401 oashl.XMLLoader.init
> > xsltCacheLifetimeSeconds=60
> >    [junit4]   2> 419564 T2401 oashl.XMLLoader.init
> > xsltCacheLifetimeSeconds=60
> >    [junit4]   2> 419567 T2401 oasu.CommitTracker.<init> Hard AutoCommit:
> > disabled
> >    [junit4]   2> 419568 T2401 oasu.CommitTracker.<init> Soft AutoCommit:
> > disabled
> >    [junit4]   2> 419569 T2401 oasu.RandomMergePolicy.<init>
> > RandomMergePolicy wrapping class
> > org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy:
> > maxMergeAtOnce=16, maxMergeAtOnceExplicit=15,
> > maxMergedSegmentMB=60.369140625, floorSegmentMB=1.87109375,
> > forceMergeDeletesPctAllowed=16.64506491494333, segmentsPerTier=38.0,
> > maxCFSSegmentSizeMB=8.796093022207999E12,
> > noCFSRatio=0.3115152765900482
> >    [junit4]   2> 419569 T2401 oasc.SolrDeletionPolicy.onInit
> > SolrDeletionPolicy.onInit: commits: num=1
> >    [junit4]   2>
> >       commit{dir=MockDirectoryWrapper(RAMDirectory@4ad441db
> > lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@505677e5),
> > segFN=segments_1,generation=1}
> >    [junit4]   2> 419570 T2401 oasc.SolrDeletionPolicy.updateCommits
> newest
> > commit generation = 1
> >    [junit4]   2> 419570 T2401 oass.SolrIndexSearcher.<init> Opening
> > Searcher@62dc7557[collection1] main
> >    [junit4]   2> 419570 T2401 oascc.ZkStateReader.readConfigName Load
> > collection config from:/collections/collection1
> >    [junit4]   2> 419571 T2401 oascc.ZkStateReader.readConfigName
> > path=/collections/collection1 configName=conf1 specified config exists in
> > ZooKeeper
> >    [junit4]   2> 419571 T2401 oasr.ManagedResourceStorage.newStorageIO
> > Setting up ZooKeeper-based storage for the RestManager with znodeBase:
> > /configs/conf1
> >    [junit4]   2> 419572 T2401
> > oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured
> > ZooKeeperStorageIO with znodeBase: /configs/conf1
> >    [junit4]   2> 419572 T2401 oasr.RestManager.init Initializing
> RestManager
> > with initArgs: {}
> >    [junit4]   2> 419572 T2401 oasr.ManagedResourceStorage.load Reading
> > _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
> >    [junit4]   2> 419573 T2401
> > oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream
> > Read 38 bytes from znode /configs/conf1/_rest_managed.json
> >    [junit4]   2> 419573 T2401 oasr.ManagedResourceStorage.load Loaded
> > LinkedHashMap at path _rest_managed.json using
> > ZooKeeperStorageIO:path=/configs/conf1
> >    [junit4]   2> 419573 T2401 oasr.ManagedResource.reloadFromStorage
> > Loaded initArgs {} for /rest/managed
> >    [junit4]   2> 419573 T2401 oasr.RestManager.init Initializing 0
> registered
> > ManagedResources
> >    [junit4]   2> 419577 T2402 oasc.SolrCore.registerSearcher
> [collection1]
> > Registered new searcher Searcher@62dc7557[collection1]
> > main{ExitableDirectoryReader(UninvertingDirectoryReader())}
> >    [junit4]   2> 419577 T2401 oasc.CoreContainer.registerCore
> registering core:
> > collection1
> >    [junit4]   2> 419578 T2405 oasc.ZkController.register Register
> replica -
> > core:collection1 address:http://127.0.0.1:28438/orln collection:co
> >
> > [...truncated too long message...]
> >
> > -Tests-trunk-Java8/solr/core/lib/org.restlet.ext.servlet-
> > 2.1.1.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-
> > trunk-Java8/solr/core/lib/protobuf-java-
> > 2.5.0.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-
> >
> trunk-Java8/solr/core/lib/slf4j-log4j12-1.7.6.jar:/usr/home/jenkins/jenkins-
> > slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/lib/spatial4j-
> > 0.4.1.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-
> > trunk-Java8/solr/solrj/lib/commons-io-2.4.jar:/usr/home/jenkins/jenkins-
> > slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/solrj/lib/httpclient-
> > 4.3.1.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-
> > trunk-Java8/solr/solrj/lib/httpcore-4.3.jar:/usr/home/jenkins/jenkins-
> > slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/solrj/lib/httpmime-
> > 4.3.1.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-
> > trunk-Java8/solr/solrj/lib/noggit-0.6.jar:/usr/home/jenkins/jenkins-
> > slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/solrj/lib/slf4j-api-
> > 1.7.6.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-
> > trunk-Java8/solr/solrj/lib/wstx-asl-3.2.7.jar:/usr/home/jenkins/jenkins-
> > slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/solrj/lib/zookeeper-
> > 3.4.6.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-
> > trunk-Java8/solr/server/lib/jetty-continuation-
> > 8.1.10.v20130312.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-
> > Solr-Tests-trunk-Java8/solr/server/lib/jetty-deploy-
> > 8.1.10.v20130312.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-
> > Solr-Tests-trunk-Java8/solr/server/lib/jetty-http-
> > 8.1.10.v20130312.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-
> > Solr-Tests-trunk-Java8/solr/server/lib/jetty-io-
> > 8.1.10.v20130312.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-
> > Solr-Tests-trunk-Java8/solr/server/lib/jetty-jmx-
> > 8.1.10.v20130312.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-
> > Solr-Tests-trunk-Java8/solr/server/lib/jetty-security-
> > 8.1.10.v20130312.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-
> > Solr-Tests-trunk-Java8/solr/server/lib/jetty-server-
> > 8.1.10.v20130312.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-
> > Solr-Tests-trunk-Java8/solr/server/lib/jetty-servlet-
> > 8.1.10.v20130312.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-
> > Solr-Tests-trunk-Java8/solr/server/lib/jetty-util-
> > 8.1.10.v20130312.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-
> > Solr-Tests-trunk-Java8/solr/server/lib/jetty-webapp-
> > 8.1.10.v20130312.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-
> > Solr-Tests-trunk-Java8/solr/server/lib/jetty-xml-
> > 8.1.10.v20130312.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-
> > Solr-Tests-trunk-Java8/solr/server/lib/servlet-api-
> > 3.0.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-
> > trunk-Java8/solr/example/example-DIH/solr/db/lib/derby-
> > 10.9.1.0.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-
> > trunk-Java8/solr/example/example-DIH/solr/db/lib/hsqldb-
> > 1.8.0.10.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-
> > trunk-Java8/lucene/build/core/classes/java:/usr/home/jenkins/jenkins-
> > slave/workspace/Lucene-Solr-Tests-trunk-Java8/lucene/test-
> > framework/lib/junit-4.10.jar:/usr/home/jenkins/jenkins-
> > slave/workspace/Lucene-Solr-Tests-trunk-Java8/lucene/test-
> > framework/lib/randomizedtesting-runner-
> > 2.1.9.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-
> > trunk-Java8/solr/core/test-lib/cglib-nodep-
> > 2.2.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-
> > trunk-Java8/solr/core/test-lib/commons-collections-
> > 3.2.1.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-
> >
> trunk-Java8/solr/core/test-lib/easymock-3.0.jar:/usr/home/jenkins/jenkins-
> > slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/test-lib/hadoop-
> > common-2.2.0-tests.jar:/usr/home/jenkins/jenkins-
> > slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/test-lib/hadoop-
> > hdfs-2.2.0-tests.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-
> > Solr-Tests-trunk-Java8/solr/core/test-lib/javax.servlet-api-
> > 3.0.1.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-
> > trunk-Java8/solr/core/test-lib/jcl-over-slf4j-
> > 1.7.6.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-
> > trunk-Java8/solr/core/test-lib/jersey-core-
> > 1.9.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-
> >
> trunk-Java8/solr/core/test-lib/jetty-6.1.26.jar:/usr/home/jenkins/jenkins-
> >
> slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/test-lib/jetty-util-
> > 6.1.26.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-
> >
> trunk-Java8/solr/core/test-lib/objenesis-1.2.jar:/usr/home/jenkins/jenkins-
> > slave/workspace/Lucene-Solr-Tests-trunk-
> > Java8/lucene/build/analysis/icu/lucene-analyzers-icu-6.0.0-
> > SNAPSHOT.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-
> > Tests-trunk-Java8/solr/build/contrib/solr-analysis-
> > extras/classes/java:/usr/home/jenkins/jenkins-slave/workspace/Lucene-
> > Solr-Tests-trunk-Java8/solr/contrib/analysis-extras/lib/icu4j-
> > 54.1.jar:/home/jenkins/jenkins-
> > slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-
> > launcher.jar:/home/jenkins/.ant/lib/ivy-2.3.0.jar:/home/jenkins/jenkins-
> > slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-
> > jsch.jar:/home/jenkins/jenkins-
> > slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-
> > jai.jar:/home/jenkins/jenkins-
> > slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-
> > regexp.jar:/home/jenkins/jenkins-
> > slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-
> > netrexx.jar:/home/jenkins/jenkins-
> > slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-
> > antlr.jar:/home/jenkins/jenkins-
> > slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-
> > junit4.jar:/home/jenkins/jenkins-
> > slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-
> > junit.jar:/home/jenkins/jenkins-
> > slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-
> > bsf.jar:/home/jenkins/jenkins-
> > slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-
> > log4j.jar:/home/jenkins/jenkins-
> > slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-
> > bcel.jar:/home/jenkins/jenkins-
> > slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-
> > resolver.jar:/home/jenkins/jenkins-
> > slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-
> > swing.jar:/home/jenkins/jenkins-
> > slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-
> > oro.jar:/home/jenkins/jenkins-
> > slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-
> > xalan2.jar:/home/jenkins/jenkins-
> > slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-
> > jmf.jar:/home/jenkins/jenkins-
> > slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-
> > jdepend.jar:/home/jenkins/jenkins-
> > slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-
> > testutil.jar:/home/jenkins/jenkins-
> > slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-commons-
> > logging.jar:/home/jenkins/jenkins-
> > slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-commons-
> > net.jar:/home/jenkins/jenkins-
> > slave/tools/hudson.tasks.Ant_AntInstallation/ant-
> > 1.8.2/lib/ant.jar:/home/jenkins/jenkins-
> > slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-
> >
> javamail.jar:/usr/local/openjdk8/lib/tools.jar:/home/jenkins/.ivy2/cache/co
> > m.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.1.9.jar
> > com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe -eventsfile
> > /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-
> > Java8/solr/build/solr-core/test/temp/junit4-J2-20141121_173810_288.events
> > @/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-
> > Java8/solr/build/solr-core/test/temp/junit4-J2-20141121_173810_288.suites
> >    [junit4] ERROR: JVM J2 ended with an exception: Forked process
> returned
> > with error code: 1. Very likely a JVM crash.  Process output piped in
> logs
> > above.
> >    [junit4]   at
> > com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1481)
> >    [junit4]   at
> > com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:133)
> >    [junit4]   at
> > com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:964)
> >    [junit4]   at
> > com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:961)
> >    [junit4]   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> >    [junit4]   at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
> > a:1142)
> >    [junit4]   at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
> > va:617)
> >    [junit4]   at java.lang.Thread.run(Thread.java:745)
> >
> > BUILD FAILED
> > /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-
> > Java8/build.xml:515: The following error occurred while executing this
> line:
> > /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-
> > Java8/build.xml:463: The following error occurred while executing this
> line:
> > /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-
> > Java8/build.xml:61: The following error occurred while executing this
> line:
> > /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-
> > Java8/extra-targets.xml:39: The following error occurred while executing
> this
> > line:
> > /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-
> > Java8/solr/build.xml:186: The following error occurred while executing
> this
> > line:
> > /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-
> > Java8/solr/common-build.xml:508: The following error occurred while
> > executing this line:
> > /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-
> > Java8/lucene/common-build.xml:1345: The following error occurred while
> > executing this line:
> > /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-
> > Java8/lucene/common-build.xml:952: At least one slave process threw an
> > exception, first: Forked process returned with error code: 134. Very
> likely a
> > JVM crash.  Process output piped in logs above.
> >
> > Total time: 92 minutes 30 seconds
> > Build step 'Invoke Ant' marked build as failure
> > Archiving artifacts
> > No prior successful build to compare, so performing full copy of
> artifacts
> > Recording test results
> > Email was triggered for: Failure
> > Sending email for trigger: Failure
> >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>


-- 
Regards,
Shalin Shekhar Mangar.

RE: [JENKINS] Lucene-Solr-Tests-trunk-Java8 - Build # 5 - Still Failing

Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi,

you might have noticed, Lucene trunk is now on Java 8. Policeman Jenkins was already changed to handle that, builds are succeeding perfectly.

Unfortunately, FreeBSD Jenkins has a problem, because OpenJDK8, namely the FreeBSD Port, crushes always while running Solr tests. Reason seems to be a problem in the networking layer, which is heavy modified from default OpenJDK. So there seems to be a bug in the patch that FreBSD applys to OpenJDK.

I already updated to OpenJDK 8u25, but this did not help. I will open an issue at the FreeBSD website, but the problem is: The crush always looks different (sometimes sigbus, sometimes sigsegv, sometimes crushing inside free()) - I have also seen a failed malloc() call [caused by corrumpted heap].

For now I disabled the test builds (nightly-smoke, nightly-tests, and hourly-tests). The artifact builds are still enabled, so people can download nightly trunk artifacts. The maven build that publishes to ASF snapshot repo was modified to have "-DskipTests" passed to ANT/Maven. So it runs and deploys the artifacts.

I am not sure if we can solve the FreeBSD issues soon, so I would suggest that we ask ASF INFRA, if they could provide us with an Ubuntu VM machine so we can migrate our builds there. I have to check with them to get enough CPU power and free disk space, so we can run the heavy nightly tests. The other good thing would be: no more blackhole :-)

Comments?

Uwe

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de


> -----Original Message-----
> From: Apache Jenkins Server [mailto:jenkins@builds.apache.org]
> Sent: Friday, November 21, 2014 8:02 PM
> To: dev@lucene.apache.org
> Subject: [JENKINS] Lucene-Solr-Tests-trunk-Java8 - Build # 5 - Still Failing
> 
> Build: https://builds.apache.org/job/Lucene-Solr-Tests-trunk-Java8/5/
> 
> 2 tests failed.
> REGRESSION:  org.apache.solr.TestDistributedGrouping.testDistribSearch
> 
> Error Message:
> org.apache.solr.client.solrj.SolrServerException: No live SolrServers available
> to handle this request:[https://[ff01::114]:33332, https://[ff01::083]:33332,
> https://127.0.0.1:37975, https://[ff01::213]:33332]
> 
> Stack Trace:
> org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
> org.apache.solr.client.solrj.SolrServerException: No live SolrServers available
> to handle this request:[https://[ff01::114]:33332, https://[ff01::083]:33332,
> https://127.0.0.1:37975, https://[ff01::213]:33332]
> 	at
> __randomizedtesting.SeedInfo.seed([103A566D0461F871:91DCD875733E984
> D]:0)
> 	at
> org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrSer
> ver.java:569)
> 	at
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:
> 215)
> 	at
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:
> 211)
> 	at
> org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.ja
> va:91)
> 	at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
> 	at
> org.apache.solr.BaseDistributedSearchTestCase.queryServer(BaseDistribute
> dSearchTestCase.java:512)
> 	at
> org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearc
> hTestCase.java:560)
> 	at
> org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearc
> hTestCase.java:542)
> 	at
> org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearc
> hTestCase.java:521)
> 	at
> org.apache.solr.TestDistributedGrouping.doTest(TestDistributedGrouping.ja
> va:206)
> 	at
> org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistri
> butedSearchTestCase.java:875)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j
> ava:62)
> 	at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
> sorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:483)
> 	at
> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(Randomize
> dRunner.java:1618)
> 	at
> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(Rando
> mizedRunner.java:827)
> 	at
> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(Rando
> mizedRunner.java:863)
> 	at
> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(Rando
> mizedRunner.java:877)
> 	at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.
> evaluate(SystemPropertiesRestoreRule.java:53)
> 	at
> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRule
> SetupTeardownChained.java:50)
> 	at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeA
> fterRule.java:46)
> 	at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1
> .evaluate(SystemPropertiesInvariantRule.java:55)
> 	at
> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleTh
> readAndTestName.java:49)
> 	at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRule
> IgnoreAfterMaxFailures.java:65)
> 	at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure
> .java:48)
> 	at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stat
> ementAdapter.java:36)
> 	at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.
> run(ThreadLeakControl.java:365)
> 	at
> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask
> (ThreadLeakControl.java:798)
> 	at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadL
> eakControl.java:458)
> 	at
> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(Ran
> domizedRunner.java:836)
> 	at
> com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(Rando
> mizedRunner.java:738)
> 	at
> com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(Rando
> mizedRunner.java:772)
> 	at
> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(Rando
> mizedRunner.java:783)
> 	at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stat
> ementAdapter.java:36)
> 	at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.
> evaluate(SystemPropertiesRestoreRule.java:53)
> 	at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeA
> fterRule.java:46)
> 	at
> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreCl
> assName.java:42)
> 	at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1
> .evaluate(SystemPropertiesInvariantRule.java:55)
> 	at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMet
> hodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> 	at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMet
> hodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> 	at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stat
> ementAdapter.java:36)
> 	at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stat
> ementAdapter.java:36)
> 	at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stat
> ementAdapter.java:36)
> 	at
> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAss
> ertionsRequired.java:54)
> 	at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure
> .java:48)
> 	at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRule
> IgnoreAfterMaxFailures.java:65)
> 	at
> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnore
> TestSuites.java:55)
> 	at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stat
> ementAdapter.java:36)
> 	at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.
> run(ThreadLeakControl.java:365)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> 
> FAILED:
> org.apache.solr.handler.TestSolrConfigHandlerConcurrent.testDistribSearch
> 
> Error Message:
> [[size property not set, expected = 11, actual null, initialSize property not set,
> expected = 12, actual null, autowarmCount property not set, expected = 13,
> actual null], [], [], []]
> 
> Stack Trace:
> java.lang.AssertionError: [[size property not set, expected = 11, actual null,
> initialSize property not set, expected = 12, actual null, autowarmCount
> property not set, expected = 13, actual null], [], [], []]
> 	at
> __randomizedtesting.SeedInfo.seed([103A566D0461F871:91DCD875733E984
> D]:0)
> 	at org.junit.Assert.fail(Assert.java:93)
> 	at org.junit.Assert.assertTrue(Assert.java:43)
> 	at
> org.apache.solr.handler.TestSolrConfigHandlerConcurrent.doTest(TestSolrCo
> nfigHandlerConcurrent.java:112)
> 	at
> org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistri
> butedSearchTestCase.java:869)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j
> ava:62)
> 	at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
> sorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:483)
> 	at
> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(Randomize
> dRunner.java:1618)
> 	at
> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(Rando
> mizedRunner.java:827)
> 	at
> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(Rando
> mizedRunner.java:863)
> 	at
> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(Rando
> mizedRunner.java:877)
> 	at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.
> evaluate(SystemPropertiesRestoreRule.java:53)
> 	at
> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRule
> SetupTeardownChained.java:50)
> 	at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeA
> fterRule.java:46)
> 	at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1
> .evaluate(SystemPropertiesInvariantRule.java:55)
> 	at
> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleTh
> readAndTestName.java:49)
> 	at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRule
> IgnoreAfterMaxFailures.java:65)
> 	at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure
> .java:48)
> 	at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stat
> ementAdapter.java:36)
> 	at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.
> run(ThreadLeakControl.java:365)
> 	at
> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask
> (ThreadLeakControl.java:798)
> 	at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadL
> eakControl.java:458)
> 	at
> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(Ran
> domizedRunner.java:836)
> 	at
> com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(Rando
> mizedRunner.java:738)
> 	at
> com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(Rando
> mizedRunner.java:772)
> 	at
> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(Rando
> mizedRunner.java:783)
> 	at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stat
> ementAdapter.java:36)
> 	at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.
> evaluate(SystemPropertiesRestoreRule.java:53)
> 	at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeA
> fterRule.java:46)
> 	at
> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreCl
> assName.java:42)
> 	at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1
> .evaluate(SystemPropertiesInvariantRule.java:55)
> 	at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMet
> hodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> 	at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMet
> hodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> 	at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stat
> ementAdapter.java:36)
> 	at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stat
> ementAdapter.java:36)
> 	at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stat
> ementAdapter.java:36)
> 	at
> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAss
> ertionsRequired.java:54)
> 	at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure
> .java:48)
> 	at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRule
> IgnoreAfterMaxFailures.java:65)
> 	at
> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnore
> TestSuites.java:55)
> 	at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stat
> ementAdapter.java:36)
> 	at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.
> run(ThreadLeakControl.java:365)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> 
> 
> 
> Build Log:
> [...truncated 13391 lines...]
>    [junit4] Suite: org.apache.solr.handler.TestSolrConfigHandlerConcurrent
>    [junit4]   2> Creating dataDir: /usr/home/jenkins/jenkins-
> slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-
> core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> 103A566D0461F871-001/init-core-data-001
>    [junit4]   2> 409504 T2325 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl
> (false) and clientAuth (false)
>    [junit4]   2> 409505 T2325
> oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext
> system property: /orln/
>    [junit4]   2> 409508 T2325 oas.SolrTestCaseJ4.setUp ###Starting
> testDistribSearch
>    [junit4]   2> 409509 T2325 oasc.ZkTestServer.run STARTING ZK TEST SERVER
>    [junit4]   1> client port:0.0.0.0/0.0.0.0:0
>    [junit4]   2> 409510 T2326 oasc.ZkTestServer$ZKServerMain.runFromConfig
> Starting server
>    [junit4]   2> 409610 T2325 oasc.ZkTestServer.run start zk server on
> port:47646
>    [junit4]   2> 409611 T2325
> oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default
> ZkCredentialsProvider
>    [junit4]   2> 409612 T2325 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
>    [junit4]   2> 409616 T2333 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@3d627ba3
> name:ZooKeeperConnection Watcher:127.0.0.1:47646 got event
> WatchedEvent state:SyncConnected type:None path:null path:null
> type:None
>    [junit4]   2> 409617 T2325 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
>    [junit4]   2> 409617 T2325 oascc.SolrZkClient.createZkACLProvider Using
> default ZkACLProvider
>    [junit4]   2> 409617 T2325 oascc.SolrZkClient.makePath makePath: /solr
>    [junit4]   2> 409620 T2325
> oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default
> ZkCredentialsProvider
>    [junit4]   2> 409621 T2325 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
>    [junit4]   2> 409622 T2336 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@2458f3dd
> name:ZooKeeperConnection Watcher:127.0.0.1:47646/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null
> type:None
>    [junit4]   2> 409622 T2325 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
>    [junit4]   2> 409623 T2325 oascc.SolrZkClient.createZkACLProvider Using
> default ZkACLProvider
>    [junit4]   2> 409623 T2325 oascc.SolrZkClient.makePath makePath:
> /collections/collection1
>    [junit4]   2> 409625 T2325 oascc.SolrZkClient.makePath makePath:
> /collections/collection1/shards
>    [junit4]   2> 409626 T2325 oascc.SolrZkClient.makePath makePath:
> /collections/control_collection
>    [junit4]   2> 409627 T2325 oascc.SolrZkClient.makePath makePath:
> /collections/control_collection/shards
>    [junit4]   2> 409629 T2325 oasc.AbstractZkTestCase.putConfig put
> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-
> Java8/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to
> /configs/conf1/solrconfig.xml
>    [junit4]   2> 409630 T2325 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/solrconfig.xml
>    [junit4]   2> 409632 T2325 oasc.AbstractZkTestCase.putConfig put
> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-
> Java8/solr/core/src/test-files/solr/collection1/conf/schema.xml to
> /configs/conf1/schema.xml
>    [junit4]   2> 409632 T2325 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/schema.xml
>    [junit4]   2> 409734 T2325 oasc.AbstractZkTestCase.putConfig put
> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-
> Java8/solr/core/src/test-
> files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to
> /configs/conf1/solrconfig.snippet.randomindexconfig.xml
>    [junit4]   2> 409735 T2325 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/solrconfig.snippet.randomindexconfig.xml
>    [junit4]   2> 409736 T2325 oasc.AbstractZkTestCase.putConfig put
> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-
> Java8/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to
> /configs/conf1/stopwords.txt
>    [junit4]   2> 409737 T2325 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/stopwords.txt
>    [junit4]   2> 409738 T2325 oasc.AbstractZkTestCase.putConfig put
> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-
> Java8/solr/core/src/test-files/solr/collection1/conf/protwords.txt to
> /configs/conf1/protwords.txt
>    [junit4]   2> 409739 T2325 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/protwords.txt
>    [junit4]   2> 409740 T2325 oasc.AbstractZkTestCase.putConfig put
> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-
> Java8/solr/core/src/test-files/solr/collection1/conf/currency.xml to
> /configs/conf1/currency.xml
>    [junit4]   2> 409741 T2325 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/currency.xml
>    [junit4]   2> 409742 T2325 oasc.AbstractZkTestCase.putConfig put
> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-
> Java8/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to
> /configs/conf1/enumsConfig.xml
>    [junit4]   2> 409743 T2325 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/enumsConfig.xml
>    [junit4]   2> 409745 T2325 oasc.AbstractZkTestCase.putConfig put
> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-
> Java8/solr/core/src/test-files/solr/collection1/conf/open-exchange-
> rates.json to /configs/conf1/open-exchange-rates.json
>    [junit4]   2> 409745 T2325 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/open-exchange-rates.json
>    [junit4]   2> 409747 T2325 oasc.AbstractZkTestCase.putConfig put
> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-
> Java8/solr/core/src/test-files/solr/collection1/conf/mapping-
> ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
>    [junit4]   2> 409747 T2325 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/mapping-ISOLatin1Accent.txt
>    [junit4]   2> 409749 T2325 oasc.AbstractZkTestCase.putConfig put
> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-
> Java8/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to
> /configs/conf1/old_synonyms.txt
>    [junit4]   2> 409750 T2325 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/old_synonyms.txt
>    [junit4]   2> 409751 T2325 oasc.AbstractZkTestCase.putConfig put
> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-
> Java8/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to
> /configs/conf1/synonyms.txt
>    [junit4]   2> 409751 T2325 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/synonyms.txt
>    [junit4]   2> 410973 T2325 oejs.Server.doStart jetty-8.1.10.v20130312
>    [junit4]   2> 410994 T2325 oejs.AbstractConnector.doStart Started
> SocketConnector@127.0.0.1:37737
>    [junit4]   2> 410994 T2325 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
>    [junit4]   2> 410994 T2325 oasc.SolrResourceLoader.locateSolrHome JNDI
> not configured for solr (NoInitialContextEx)
>    [junit4]   2> 410995 T2325 oasc.SolrResourceLoader.locateSolrHome using
> system property solr.solr.home:
> ../../../../../../../../../../../usr/home/jenkins/jenkins-
> slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-
> core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> 103A566D0461F871-001/tempDir-002
>    [junit4]   2> 410995 T2325 oasc.SolrResourceLoader.<init> new
> SolrResourceLoader for directory:
> '../../../../../../../../../../../usr/home/jenkins/jenkins-
> slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-
> core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> 103A566D0461F871-001/tempDir-002/'
>    [junit4]   2> 411025 T2325 oasc.ConfigSolr.fromFile Loading container
> configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-
> Solr-Tests-trunk-Java8/solr/build/solr-
> core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-
> slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-
> core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> 103A566D0461F871-001/tempDir-002/solr.xml
>    [junit4]   2> 411038 T2325 oasc.CoreContainer.<init> New CoreContainer
> 1421426377
>    [junit4]   2> 411039 T2325 oasc.CoreContainer.load Loading cores into
> CoreContainer
> [instanceDir=../../../../../../../../../../../usr/home/jenkins/jenkins-
> slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-
> core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> 103A566D0461F871-001/tempDir-002/]
>    [junit4]   2> 411039 T2325 oashc.HttpShardHandlerFactory.getParameter
> Setting socketTimeout to: 90000
>    [junit4]   2> 411040 T2325 oashc.HttpShardHandlerFactory.getParameter
> Setting urlScheme to:
>    [junit4]   2> 411040 T2325 oashc.HttpShardHandlerFactory.getParameter
> Setting connTimeout to: 15000
>    [junit4]   2> 411040 T2325 oashc.HttpShardHandlerFactory.getParameter
> Setting maxConnectionsPerHost to: 20
>    [junit4]   2> 411040 T2325 oashc.HttpShardHandlerFactory.getParameter
> Setting maxConnections to: 10000
>    [junit4]   2> 411040 T2325 oashc.HttpShardHandlerFactory.getParameter
> Setting corePoolSize to: 0
>    [junit4]   2> 411041 T2325 oashc.HttpShardHandlerFactory.getParameter
> Setting maximumPoolSize to: 2147483647
>    [junit4]   2> 411041 T2325 oashc.HttpShardHandlerFactory.getParameter
> Setting maxThreadIdleTime to: 5
>    [junit4]   2> 411041 T2325 oashc.HttpShardHandlerFactory.getParameter
> Setting sizeOfQueue to: -1
>    [junit4]   2> 411041 T2325 oashc.HttpShardHandlerFactory.getParameter
> Setting fairnessPolicy to: false
>    [junit4]   2> 411042 T2325 oasu.UpdateShardHandler.<init> Creating
> UpdateShardHandler HTTP client with params:
> socketTimeout=340000&connTimeout=45000&retry=false
>    [junit4]   2> 411042 T2325 oasl.LogWatcher.createWatcher SLF4J impl is
> org.slf4j.impl.Log4jLoggerFactory
>    [junit4]   2> 411042 T2325 oasl.LogWatcher.newRegisteredLogWatcher
> Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>    [junit4]   2> 411043 T2325 oasc.CoreContainer.load Host Name: 127.0.0.1
>    [junit4]   2> 411043 T2325 oasc.ZkContainer.initZooKeeper Zookeeper
> client=127.0.0.1:47646/solr
>    [junit4]   2> 411043 T2325 oasc.ZkController.checkChrootPath zkHost
> includes chroot
>    [junit4]   2> 411044 T2325
> oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default
> ZkCredentialsProvider
>    [junit4]   2> 411044 T2325 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
>    [junit4]   2> 411046 T2348 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@5640bd9
> name:ZooKeeperConnection Watcher:127.0.0.1:47646 got event
> WatchedEvent state:SyncConnected type:None path:null path:null
> type:None
>    [junit4]   2> 411047 T2325 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
>    [junit4]   2> 411047 T2325 oascc.SolrZkClient.createZkACLProvider Using
> default ZkACLProvider
>    [junit4]   2> 411049 T2325 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
>    [junit4]   2> 411050 T2351 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@286c5bed
> name:ZooKeeperConnection Watcher:127.0.0.1:47646/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null
> type:None
>    [junit4]   2> 411051 T2325 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
>    [junit4]   2> 411052 T2325 oascc.SolrZkClient.makePath makePath:
> /overseer/queue
>    [junit4]   2> 411053 T2325 oascc.SolrZkClient.makePath makePath:
> /overseer/collection-queue-work
>    [junit4]   2> 411055 T2325 oascc.SolrZkClient.makePath makePath:
> /overseer/collection-map-running
>    [junit4]   2> 411056 T2325 oascc.SolrZkClient.makePath makePath:
> /overseer/collection-map-completed
>    [junit4]   2> 411058 T2325 oascc.SolrZkClient.makePath makePath:
> /overseer/collection-map-failure
>    [junit4]   2> 411060 T2325 oascc.SolrZkClient.makePath makePath:
> /live_nodes
>    [junit4]   2> 411060 T2325 oasc.ZkController.createEphemeralLiveNode
> Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37737_orln
>    [junit4]   2> 411061 T2325 oascc.SolrZkClient.makePath makePath:
> /live_nodes/127.0.0.1:37737_orln
>    [junit4]   2> 411062 T2325 oascc.SolrZkClient.makePath makePath:
> /overseer_elect
>    [junit4]   2> 411063 T2325 oascc.SolrZkClient.makePath makePath:
> /overseer_elect/election
>    [junit4]   2> 411064 T2325 oasc.Overseer.close Overseer (id=null) closing
>    [junit4]   2> 411065 T2325 oasc.OverseerElectionContext.runLeaderProcess I
> am going to be the leader 127.0.0.1:37737_orln
>    [junit4]   2> 411066 T2325 oascc.SolrZkClient.makePath makePath:
> /overseer_elect/leader
>    [junit4]   2> 411067 T2325 oasc.Overseer.start Overseer
> (id=92837767038042115-127.0.0.1:37737_orln-n_0000000000) starting
>    [junit4]   2> 411068 T2325 oascc.SolrZkClient.makePath makePath:
> /overseer/queue-work
>    [junit4]   2> 411072 T2325 oasc.OverseerAutoReplicaFailoverThread.<init>
> Starting OverseerAutoReplicaFailoverThread
> autoReplicaFailoverWorkLoopDelay=10000
> autoReplicaFailoverWaitAfterExpiration=30000
> autoReplicaFailoverBadNodeExpiration=60000
>    [junit4]   2> 411073 T2353 oasc.OverseerCollectionProcessor.run Process
> current queue of collection creations
>    [junit4]   2> 411073 T2325 oascc.SolrZkClient.makePath makePath:
> /clusterstate.json
>    [junit4]   2> 411075 T2325 oascc.SolrZkClient.makePath makePath:
> /aliases.json
>    [junit4]   2> 411075 T2325
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating
> cluster state from ZooKeeper...
>    [junit4]   2> 411077 T2352 oasc.Overseer$ClusterStateUpdater.run Starting
> to work on the main queue
>    [junit4]   2> 411079 T2355 oasc.ZkController.publish publishing
> core=collection1 state=down collection=control_collection
>    [junit4]   2> 411080 T2355 oasc.ZkController.publish numShards not found
> on descriptor - reading it from system property
>    [junit4]   2> 411080 T2351 oasc.DistributedQueue$LatchWatcher.process
> NodeChildrenChanged fired on path /overseer/queue state SyncConnected
>    [junit4]   2> 411080 T2355 oasc.ZkController.waitForCoreNodeName look for
> our core node name
>    [junit4]   2> 411082 T2352 oasc.Overseer$ClusterStateUpdater.updateState
> Update state numShards=1 message={
>    [junit4]   2> 	  "core":"collection1",
>    [junit4]   2> 	  "roles":null,
>    [junit4]   2> 	  "base_url":"http://127.0.0.1:37737/orln",
>    [junit4]   2> 	  "node_name":"127.0.0.1:37737_orln",
>    [junit4]   2> 	  "numShards":"1",
>    [junit4]   2> 	  "state":"down",
>    [junit4]   2> 	  "shard":null,
>    [junit4]   2> 	  "collection":"control_collection",
>    [junit4]   2> 	  "operation":"state"}
>    [junit4]   2> 411082 T2352
> oasc.Overseer$ClusterStateUpdater.createCollection Create collection
> control_collection with shards [shard1]
>    [junit4]   2> 411082 T2352
> oasc.Overseer$ClusterStateUpdater.createCollection state version
> control_collection 1
>    [junit4]   2> 411082 T2352 oasc.Overseer$ClusterStateUpdater.updateState
> Assigning new node to shard shard=shard1
>    [junit4]   2> 411084 T2351 oascc.ZkStateReader$2.process A cluster state
> change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>    [junit4]   2> 412081 T2355 oasc.ZkController.waitForShardId waiting to find
> shard id in clusterstate for collection1
>    [junit4]   2> 412082 T2355 oasc.ZkController.createCollectionZkNode Check
> for collection zkNode:control_collection
>    [junit4]   2> 412082 T2355 oasc.ZkController.createCollectionZkNode
> Collection zkNode exists
>    [junit4]   2> 412083 T2355 oascc.ZkStateReader.readConfigName Load
> collection config from:/collections/control_collection
>    [junit4]   2> 412083 T2355 oascc.ZkStateReader.readConfigName
> path=/collections/control_collection configName=conf1 specified config
> exists in ZooKeeper
>    [junit4]   2> 412083 T2355 oasc.SolrResourceLoader.<init> new
> SolrResourceLoader for directory:
> '../../../../../../../../../../../usr/home/jenkins/jenkins-
> slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-
> core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> 103A566D0461F871-001/tempDir-002/collection1/'
>    [junit4]   2> 412085 T2355 oasc.SolrResourceLoader.replaceClassLoader
> Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-
> trunk-Java8/solr/build/solr-
> core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> 103A566D0461F871-001/tempDir-002/collection1/lib/.svn/' to classloader
>    [junit4]   2> 412086 T2355 oasc.SolrResourceLoader.replaceClassLoader
> Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-
> trunk-Java8/solr/build/solr-
> core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> 103A566D0461F871-001/tempDir-002/collection1/lib/README' to classloader
>    [junit4]   2> 412087 T2355 oasc.SolrResourceLoader.replaceClassLoader
> Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-
> trunk-Java8/solr/build/solr-
> core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> 103A566D0461F871-001/tempDir-002/collection1/lib/classes/' to classloader
>    [junit4]   2> 412119 T2355 oasc.ZkController.watchZKConfDir watch zkdir
> /configs/conf1
>    [junit4]   2> 412120 T2355 oasc.Config.<init> loaded config solrconfig.xml
> with version 0
>    [junit4]   2> 412134 T2355 oasc.SolrConfig.<init> Using Lucene
> MatchVersion: 6.0.0
>    [junit4]   2> 412145 T2355 oasc.SolrConfig.<init> Loaded SolrConfig:
> solrconfig.xml
>    [junit4]   2> 412246 T2355 oass.IndexSchema.readSchema Reading Solr
> Schema from /configs/conf1/schema.xml
>    [junit4]   2> 412252 T2355 oass.IndexSchema.readSchema [collection1]
> Schema name=test
>    [junit4]   2> 412366 T2355 oass.OpenExchangeRatesOrgProvider.init
> Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
>    [junit4]   2> 412376 T2355 oass.IndexSchema.readSchema default search
> field in schema is text
>    [junit4]   2> 412378 T2355 oass.IndexSchema.readSchema unique key field:
> id
>    [junit4]   2> 412385 T2355 oass.FileExchangeRateProvider.reload Reloading
> exchange rates from file currency.xml
>    [junit4]   2> 412388 T2355 oass.FileExchangeRateProvider.reload Reloading
> exchange rates from file currency.xml
>    [junit4]   2> 412390 T2355 oass.OpenExchangeRatesOrgProvider.reload
> Reloading exchange rates from open-exchange-rates.json
>    [junit4]   2> 412391 T2355
> oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN
> Unknown key IMPORTANT NOTE
>    [junit4]   2> 412392 T2355
> oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN
> Expected key, got STRING
>    [junit4]   2> 412392 T2355 oass.OpenExchangeRatesOrgProvider.reload
> Reloading exchange rates from open-exchange-rates.json
>    [junit4]   2> 412392 T2355
> oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN
> Unknown key IMPORTANT NOTE
>    [junit4]   2> 412393 T2355
> oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN
> Expected key, got STRING
>    [junit4]   2> 412393 T2355 oasc.CoreContainer.create Creating SolrCore
> 'collection1' using configuration from collection control_collection
>    [junit4]   2> 412393 T2355 oasc.SolrCore.initDirectoryFactory
> org.apache.solr.core.MockDirectoryFactory
>    [junit4]   2> 412393 T2355 oasc.SolrCore.<init> [collection1] Opening new
> SolrCore at ../../../../../../../../../../../usr/home/jenkins/jenkins-
> slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-
> core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> 103A566D0461F871-001/tempDir-002/collection1/,
> dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-
> trunk-Java8/solr/build/solr-
> core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> 103A566D0461F871-001/tempDir-001/control/data/
>    [junit4]   2> 412394 T2355 oasc.JmxMonitoredMap.<init> JMX monitoring is
> enabled. Adding Solr mbeans to JMX Server:
> com.sun.jmx.mbeanserver.JmxMBeanServer@330fda4c
>    [junit4]   2> 412394 T2355 oasc.CachingDirectoryFactory.get return new
> directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-
> Tests-trunk-Java8/solr/build/solr-
> core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> 103A566D0461F871-001/tempDir-001/control/data
>    [junit4]   2> 412395 T2355 oasc.SolrCore.getNewIndexDir New index
> directory detected: old=null new=/usr/home/jenkins/jenkins-
> slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-
> core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> 103A566D0461F871-001/tempDir-001/control/data/index/
>    [junit4]   2> 412395 T2355 oasc.SolrCore.initIndex WARN [collection1] Solr
> index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-
> Tests-trunk-Java8/solr/build/solr-
> core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> 103A566D0461F871-001/tempDir-001/control/data/index' doesn't exist.
> Creating new index...
>    [junit4]   2> 412395 T2355 oasc.CachingDirectoryFactory.get return new
> directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-
> Tests-trunk-Java8/solr/build/solr-
> core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> 103A566D0461F871-001/tempDir-001/control/data/index
>    [junit4]   2> 412396 T2355 oasu.RandomMergePolicy.<init>
> RandomMergePolicy wrapping class
> org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy:
> maxMergeAtOnce=30, maxMergeAtOnceExplicit=41,
> maxMergedSegmentMB=64.537109375, floorSegmentMB=1.693359375,
> forceMergeDeletesPctAllowed=3.111209946074762, segmentsPerTier=41.0,
> maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
>    [junit4]   2> 412397 T2355 oasc.SolrDeletionPolicy.onCommit
> SolrDeletionPolicy.onCommit: commits: num=1
>    [junit4]   2>
> 	commit{dir=MockDirectoryWrapper(RAMDirectory@17bfecc0
> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@3d315b92),
> segFN=segments_1,generation=1}
>    [junit4]   2> 412397 T2355 oasc.SolrDeletionPolicy.updateCommits newest
> commit generation = 1
>    [junit4]   2> 412400 T2355 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "nodistrib"
>    [junit4]   2> 412400 T2355 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "dedupe"
>    [junit4]   2> 412401 T2355 oasup.UpdateRequestProcessorChain.init
> inserting DistributedUpdateProcessorFactory into
> updateRequestProcessorChain "dedupe"
>    [junit4]   2> 412401 T2355 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "stored_sig"
>    [junit4]   2> 412401 T2355 oasup.UpdateRequestProcessorChain.init
> inserting DistributedUpdateProcessorFactory into
> updateRequestProcessorChain "stored_sig"
>    [junit4]   2> 412401 T2355 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
>    [junit4]   2> 412402 T2355 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>    [junit4]   2> 412402 T2355 oasup.UpdateRequestProcessorChain.init
> inserting DistributedUpdateProcessorFactory into
> updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>    [junit4]   2> 412402 T2355 oasc.SolrCore.loadUpdateProcessorChains no
> updateRequestProcessorChain defined as default, creating implicit default
>    [junit4]   2> 412403 T2355 oasc.RequestHandlers.initHandlersFromConfig
> created /config: org.apache.solr.handler.SolrConfigHandler
>    [junit4]   2> 412404 T2355 oasc.RequestHandlers.initHandlersFromConfig
> created /update/csv: org.apache.solr.handler.UpdateRequestHandler
>    [junit4]   2> 412404 T2355 oasc.RequestHandlers.initHandlersFromConfig
> created /update/json: org.apache.solr.handler.UpdateRequestHandler
>    [junit4]   2> 412404 T2355 oasc.RequestHandlers.initHandlersFromConfig
> created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
>    [junit4]   2> 412405 T2355 oasc.RequestHandlers.initHandlersFromConfig
> created standard: solr.StandardRequestHandler
>    [junit4]   2> 412405 T2355 oasc.RequestHandlers.initHandlersFromConfig
> created /get: solr.RealTimeGetHandler
>    [junit4]   2> 412405 T2355 oasc.RequestHandlers.initHandlersFromConfig
> adding lazy requestHandler: solr.ReplicationHandler
>    [junit4]   2> 412406 T2355 oasc.RequestHandlers.initHandlersFromConfig
> created /replication: solr.ReplicationHandler
>    [junit4]   2> 412406 T2355 oasc.RequestHandlers.initHandlersFromConfig
> created /update: solr.UpdateRequestHandler
>    [junit4]   2> 412406 T2355 oasc.RequestHandlers.initHandlersFromConfig
> created /admin/: org.apache.solr.handler.admin.AdminHandlers
>    [junit4]   2> 412407 T2355 oasc.RequestHandlers.initHandlersFromConfig
> created /admin/file: solr.admin.ShowFileRequestHandler
>    [junit4]   2> 412407 T2355 oasc.RequestHandlers.initHandlersFromConfig
> created /admin/fileedit: solr.admin.EditFileRequestHandler
>    [junit4]   2> 412417 T2355 oashl.XMLLoader.init
> xsltCacheLifetimeSeconds=60
>    [junit4]   2> 412418 T2355 oashl.XMLLoader.init
> xsltCacheLifetimeSeconds=60
>    [junit4]   2> 412420 T2355 oashl.XMLLoader.init
> xsltCacheLifetimeSeconds=60
>    [junit4]   2> 412421 T2355 oashl.XMLLoader.init
> xsltCacheLifetimeSeconds=60
>    [junit4]   2> 412425 T2355 oasu.CommitTracker.<init> Hard AutoCommit:
> disabled
>    [junit4]   2> 412425 T2355 oasu.CommitTracker.<init> Soft AutoCommit:
> disabled
>    [junit4]   2> 412425 T2355 oasu.RandomMergePolicy.<init>
> RandomMergePolicy wrapping class
> org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy:
> maxMergeAtOnce=16, maxMergeAtOnceExplicit=15,
> maxMergedSegmentMB=60.369140625, floorSegmentMB=1.87109375,
> forceMergeDeletesPctAllowed=16.64506491494333, segmentsPerTier=38.0,
> maxCFSSegmentSizeMB=8.796093022207999E12,
> noCFSRatio=0.3115152765900482
>    [junit4]   2> 412426 T2355 oasc.SolrDeletionPolicy.onInit
> SolrDeletionPolicy.onInit: commits: num=1
>    [junit4]   2>
> 	commit{dir=MockDirectoryWrapper(RAMDirectory@17bfecc0
> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@3d315b92),
> segFN=segments_1,generation=1}
>    [junit4]   2> 412426 T2355 oasc.SolrDeletionPolicy.updateCommits newest
> commit generation = 1
>    [junit4]   2> 412427 T2355 oass.SolrIndexSearcher.<init> Opening
> Searcher@48d7329e[collection1] main
>    [junit4]   2> 412427 T2355 oascc.ZkStateReader.readConfigName Load
> collection config from:/collections/control_collection
>    [junit4]   2> 412428 T2355 oascc.ZkStateReader.readConfigName
> path=/collections/control_collection configName=conf1 specified config
> exists in ZooKeeper
>    [junit4]   2> 412428 T2355 oasr.ManagedResourceStorage.newStorageIO
> Setting up ZooKeeper-based storage for the RestManager with znodeBase:
> /configs/conf1
>    [junit4]   2> 412428 T2355
> oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured
> ZooKeeperStorageIO with znodeBase: /configs/conf1
>    [junit4]   2> 412428 T2355 oasr.RestManager.init Initializing RestManager
> with initArgs: {}
>    [junit4]   2> 412429 T2355 oasr.ManagedResourceStorage.load Reading
> _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
>    [junit4]   2> 412429 T2355
> oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No
> data found for znode /configs/conf1/_rest_managed.json
>    [junit4]   2> 412429 T2355 oasr.ManagedResourceStorage.load Loaded null
> at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
>    [junit4]   2> 412430 T2355 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/_rest_managed.json
>    [junit4]   2> 412431 T2355
> oasr.ManagedResourceStorage$ZooKeeperStorageIO$1.close Wrote 38
> bytes to new znode /configs/conf1/_rest_managed.json
>    [junit4]   2> 412432 T2355
> oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to
> path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
>    [junit4]   2> 412432 T2355 oasr.RestManager.init Initializing 0 registered
> ManagedResources
>    [junit4]   2> 412434 T2356 oasc.SolrCore.registerSearcher [collection1]
> Registered new searcher Searcher@48d7329e[collection1]
> main{ExitableDirectoryReader(UninvertingDirectoryReader())}
>    [junit4]   2> 412435 T2355 oasc.CoreContainer.registerCore registering core:
> collection1
>    [junit4]   2> 412436 T2359 oasc.ZkController.register Register replica -
> core:collection1 address:http://127.0.0.1:37737/orln
> collection:control_collection shard:shard1
>    [junit4]   2> 412436 T2325 oass.SolrDispatchFilter.init
> user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-
> trunk-Java8/solr/build/solr-core/test/J3
>    [junit4]   2> 412436 T2325 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
> done
>    [junit4]   2> 412436 T2359 oascc.SolrZkClient.makePath makePath:
> /collections/control_collection/leader_elect/shard1/election
>    [junit4]   2> 412437 T2325
> oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default
> ZkCredentialsProvider
>    [junit4]   2> 412438 T2325 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
>    [junit4]   2> 412440 T2362 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@7e3464f1
> name:ZooKeeperConnection Watcher:127.0.0.1:47646/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null
> type:None
>    [junit4]   2> 412440 T2325 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
>    [junit4]   2> 412440 T2325 oascc.SolrZkClient.createZkACLProvider Using
> default ZkACLProvider
>    [junit4]   2> 412441 T2325
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating
> cluster state from ZooKeeper...
>    [junit4]   2> 412441 T2359
> oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader
> process for shard shard1
>    [junit4]   2> 412443 T2351 oasc.DistributedQueue$LatchWatcher.process
> NodeChildrenChanged fired on path /overseer/queue state SyncConnected
>    [junit4]   2> 412443 T2359
> oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough
> replicas found to continue.
>    [junit4]   2> 412443 T2359
> oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader
> - try and sync
>    [junit4]   2> ASYNC  NEW_CORE C136 name=collection1
> org.apache.solr.core.SolrCore@3982b644
> url=http://127.0.0.1:37737/orln/collection1 node=127.0.0.1:37737_orln
> C136_STATE=coll:control_collection core:collection1 props:{core=collection1,
> base_url=http://127.0.0.1:37737/orln, node_name=127.0.0.1:37737_orln,
> state=down}
>    [junit4]   2> 412443 T2359 C136 P37737 oasc.SyncStrategy.sync Sync replicas
> to http://127.0.0.1:37737/orln/collection1/
>    [junit4]   2> 412444 T2359 C136 P37737 oasc.SyncStrategy.syncReplicas Sync
> Success - now sync replicas to me
>    [junit4]   2> 412444 T2325 oasc.ChaosMonkey.monkeyLog monkey: init -
> expire sessions:false cause connection loss:false
>    [junit4]   2> 412444 T2359 C136 P37737 oasc.SyncStrategy.syncToMe
> http://127.0.0.1:37737/orln/collection1/ has no replicas
>    [junit4]   2> 412445 T2325 oasc.AbstractFullDistribZkTestBase.createJettys
> Creating collection1 with stateFormat=2
>    [junit4]   2> 412445 T2359
> oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader:
> http://127.0.0.1:37737/orln/collection1/ shard1
>    [junit4]   2> 412446 T2325
> oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default
> ZkCredentialsProvider
>    [junit4]   2> 412446 T2359 oascc.SolrZkClient.makePath makePath:
> /collections/control_collection/leaders/shard1
>    [junit4]   2> 412447 T2325 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
>    [junit4]   2> 412448 T2365 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@5249c5fa
> name:ZooKeeperConnection Watcher:127.0.0.1:47646/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null
> type:None
>    [junit4]   2> 412448 T2325 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
>    [junit4]   2> 412449 T2325 oascc.SolrZkClient.createZkACLProvider Using
> default ZkACLProvider
>    [junit4]   2> 412450 T2351 oasc.DistributedQueue$LatchWatcher.process
> NodeChildrenChanged fired on path /overseer/queue state SyncConnected
>    [junit4]   2> 412453 T2352
> oasc.Overseer$ClusterStateUpdater.buildCollection building a new
> collection: collection1
>    [junit4]   2> 412454 T2352
> oasc.Overseer$ClusterStateUpdater.createCollection Create collection
> collection1 with shards [shard1, shard2]
>    [junit4]   2> 412454 T2352
> oasc.Overseer$ClusterStateUpdater.createCollection state version
> collection1 2
>    [junit4]   2> 412455 T2351 oasc.DistributedQueue$LatchWatcher.process
> NodeChildrenChanged fired on path /overseer/queue state SyncConnected
>    [junit4]   2> 412455 T2352
> oasc.Overseer$ClusterStateUpdater.updateZkStates going to
> create_collection /collections/collection1/state.json
>    [junit4]   2> 412456 T2351 oascc.ZkStateReader$2.process A cluster state
> change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>    [junit4]   2> 412456 T2362 oascc.ZkStateReader$2.process A cluster state
> change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>    [junit4]   2> 412502 T2359 oasc.ZkController.register We are
> http://127.0.0.1:37737/orln/collection1/ and leader is
> http://127.0.0.1:37737/orln/collection1/
>    [junit4]   2> 412503 T2359 oasc.ZkController.register No LogReplay needed
> for core=collection1 baseURL=http://127.0.0.1:37737/orln
>    [junit4]   2> 412503 T2359 oasc.ZkController.checkRecovery I am the leader,
> no recovery necessary
>    [junit4]   2> 412503 T2359 oasc.ZkController.publish publishing
> core=collection1 state=active collection=control_collection
>    [junit4]   2> 412503 T2359 oasc.ZkController.publish numShards not found
> on descriptor - reading it from system property
>    [junit4]   2> 412504 T2351 oasc.DistributedQueue$LatchWatcher.process
> NodeChildrenChanged fired on path /overseer/queue state SyncConnected
>    [junit4]   2> 412506 T2352 oasc.Overseer$ClusterStateUpdater.updateState
> Update state numShards=2 message={
>    [junit4]   2> 	  "core":"collection1",
>    [junit4]   2> 	  "core_node_name":"core_node1",
>    [junit4]   2> 	  "roles":null,
>    [junit4]   2> 	  "base_url":"http://127.0.0.1:37737/orln",
>    [junit4]   2> 	  "node_name":"127.0.0.1:37737_orln",
>    [junit4]   2> 	  "numShards":"2",
>    [junit4]   2> 	  "state":"active",
>    [junit4]   2> 	  "shard":"shard1",
>    [junit4]   2> 	  "collection":"control_collection",
>    [junit4]   2> 	  "operation":"state"}
>    [junit4]   2> 412609 T2362 oascc.ZkStateReader$2.process A cluster state
> change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>    [junit4]   2> 412609 T2351 oascc.ZkStateReader$2.process A cluster state
> change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>    [junit4]   2> 413587 T2325 oasc.AbstractFullDistribZkTestBase.createJettys
> create jetty 1
>    [junit4]   2> 413587 T2325 oejs.Server.doStart jetty-8.1.10.v20130312
>    [junit4]   2> 413590 T2325 oejs.AbstractConnector.doStart Started
> SocketConnector@127.0.0.1:35189
>    [junit4]   2> 413590 T2325 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
>    [junit4]   2> 413591 T2325 oasc.SolrResourceLoader.locateSolrHome JNDI
> not configured for solr (NoInitialContextEx)
>    [junit4]   2> 413591 T2325 oasc.SolrResourceLoader.locateSolrHome using
> system property solr.solr.home: /usr/home/jenkins/jenkins-
> slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-
> core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> 103A566D0461F871-001/tempDir-003
>    [junit4]   2> 413591 T2325 oasc.SolrResourceLoader.<init> new
> SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-
> slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-
> core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> 103A566D0461F871-001/tempDir-003/'
>    [junit4]   2> 413620 T2325 oasc.ConfigSolr.fromFile Loading container
> configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-
> Solr-Tests-trunk-Java8/solr/build/solr-
> core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> 103A566D0461F871-001/tempDir-003/solr.xml
>    [junit4]   2> 413633 T2325 oasc.CoreContainer.<init> New CoreContainer
> 751864104
>    [junit4]   2> 413634 T2325 oasc.CoreContainer.load Loading cores into
> CoreContainer [instanceDir=/usr/home/jenkins/jenkins-
> slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-
> core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> 103A566D0461F871-001/tempDir-003/]
>    [junit4]   2> 413634 T2325 oashc.HttpShardHandlerFactory.getParameter
> Setting socketTimeout to: 90000
>    [junit4]   2> 413634 T2325 oashc.HttpShardHandlerFactory.getParameter
> Setting urlScheme to:
>    [junit4]   2> 413635 T2325 oashc.HttpShardHandlerFactory.getParameter
> Setting connTimeout to: 15000
>    [junit4]   2> 413635 T2325 oashc.HttpShardHandlerFactory.getParameter
> Setting maxConnectionsPerHost to: 20
>    [junit4]   2> 413635 T2325 oashc.HttpShardHandlerFactory.getParameter
> Setting maxConnections to: 10000
>    [junit4]   2> 413635 T2325 oashc.HttpShardHandlerFactory.getParameter
> Setting corePoolSize to: 0
>    [junit4]   2> 413635 T2325 oashc.HttpShardHandlerFactory.getParameter
> Setting maximumPoolSize to: 2147483647
>    [junit4]   2> 413636 T2325 oashc.HttpShardHandlerFactory.getParameter
> Setting maxThreadIdleTime to: 5
>    [junit4]   2> 413636 T2325 oashc.HttpShardHandlerFactory.getParameter
> Setting sizeOfQueue to: -1
>    [junit4]   2> 413636 T2325 oashc.HttpShardHandlerFactory.getParameter
> Setting fairnessPolicy to: false
>    [junit4]   2> 413637 T2325 oasu.UpdateShardHandler.<init> Creating
> UpdateShardHandler HTTP client with params:
> socketTimeout=340000&connTimeout=45000&retry=false
>    [junit4]   2> 413637 T2325 oasl.LogWatcher.createWatcher SLF4J impl is
> org.slf4j.impl.Log4jLoggerFactory
>    [junit4]   2> 413637 T2325 oasl.LogWatcher.newRegisteredLogWatcher
> Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>    [junit4]   2> 413637 T2325 oasc.CoreContainer.load Host Name: 127.0.0.1
>    [junit4]   2> 413638 T2325 oasc.ZkContainer.initZooKeeper Zookeeper
> client=127.0.0.1:47646/solr
>    [junit4]   2> 413638 T2325 oasc.ZkController.checkChrootPath zkHost
> includes chroot
>    [junit4]   2> 413638 T2325
> oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default
> ZkCredentialsProvider
>    [junit4]   2> 413639 T2325 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
>    [junit4]   2> 413641 T2377 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@3b3bc7d
> name:ZooKeeperConnection Watcher:127.0.0.1:47646 got event
> WatchedEvent state:SyncConnected type:None path:null path:null
> type:None
>    [junit4]   2> 413641 T2325 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
>    [junit4]   2> 413642 T2325 oascc.SolrZkClient.createZkACLProvider Using
> default ZkACLProvider
>    [junit4]   2> 413644 T2325 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
>    [junit4]   2> 413645 T2380 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@4ed9fdbd
> name:ZooKeeperConnection Watcher:127.0.0.1:47646/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null
> type:None
>    [junit4]   2> 413645 T2325 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
>    [junit4]   2> 413649 T2325
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating
> cluster state from ZooKeeper...
>    [junit4]   2> 414653 T2325 oasc.ZkController.createEphemeralLiveNode
> Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35189_orln
>    [junit4]   2> 414654 T2325 oascc.SolrZkClient.makePath makePath:
> /live_nodes/127.0.0.1:35189_orln
>    [junit4]   2> 414656 T2325 oasc.Overseer.close Overseer (id=null) closing
>    [junit4]   2> 414662 T2381 oasc.ZkController.publish publishing
> core=collection1 state=down collection=collection1
>    [junit4]   2> 414662 T2381 oasc.ZkController.publish numShards not found
> on descriptor - reading it from system property
>    [junit4]   2> 414663 T2351 oasc.DistributedQueue$LatchWatcher.process
> NodeChildrenChanged fired on path /overseer/queue state SyncConnected
>    [junit4]   2> 414663 T2381 oasc.ZkController.preRegister Registering watch
> for external collection collection1
>    [junit4]   2> 414663 T2381 oascc.ZkStateReader.addZkWatch addZkWatch
> collection1
>    [junit4]   2> 414664 T2381 oascc.ZkStateReader.addZkWatch Updating
> collection state at /collections/collection1/state.json from ZooKeeper...
>    [junit4]   2> 414665 T2381 oascc.ZkStateReader.updateWatchedCollection
> Updating data for collection1 to ver 0
>    [junit4]   2> 414665 T2381 oasc.ZkController.waitForCoreNodeName look for
> our core node name
>    [junit4]   2> 414668 T2352 oasc.Overseer$ClusterStateUpdater.updateState
> Update state numShards=2 message={
>    [junit4]   2> 	  "core":"collection1",
>    [junit4]   2> 	  "roles":null,
>    [junit4]   2> 	  "base_url":"http://127.0.0.1:35189/orln",
>    [junit4]   2> 	  "node_name":"127.0.0.1:35189_orln",
>    [junit4]   2> 	  "numShards":"2",
>    [junit4]   2> 	  "state":"down",
>    [junit4]   2> 	  "shard":null,
>    [junit4]   2> 	  "collection":"collection1",
>    [junit4]   2> 	  "operation":"state"}
>    [junit4]   2> 414670 T2352 oasc.Overseer$ClusterStateUpdater.updateState
> Collection already exists with numShards=2
>    [junit4]   2> 414671 T2352 oasc.Overseer$ClusterStateUpdater.updateState
> Assigning new node to shard shard=shard2
>    [junit4]   2> 414673 T2351 oasc.DistributedQueue$LatchWatcher.process
> NodeChildrenChanged fired on path /overseer/queue state SyncConnected
>    [junit4]   2> 414674 T2352
> oasc.Overseer$ClusterStateUpdater.updateZkStates going to
> update_collection /collections/collection1/state.json
>    [junit4]   2> 414674 T2380 oascc.ZkStateReader$7.process A cluster state
> change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/collections/collection1/state.json, has occurred - updating...
>    [junit4]   2> 414675 T2380 oascc.ZkStateReader.updateWatchedCollection
> Updating data for collection1 to ver 1
>    [junit4]   2> 415666 T2381 oasc.ZkController.waitForShardId waiting to find
> shard id in clusterstate for collection1
>    [junit4]   2> 415667 T2381 oasc.ZkController.createCollectionZkNode Check
> for collection zkNode:collection1
>    [junit4]   2> 415667 T2381 oasc.ZkController.createCollectionZkNode
> Collection zkNode exists
>    [junit4]   2> 415667 T2381 oascc.ZkStateReader.readConfigName Load
> collection config from:/collections/collection1
>    [junit4]   2> 415668 T2381 oascc.ZkStateReader.readConfigName
> path=/collections/collection1 configName=conf1 specified config exists in
> ZooKeeper
>    [junit4]   2> 415668 T2381 oasc.SolrResourceLoader.<init> new
> SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-
> slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-
> core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> 103A566D0461F871-001/tempDir-003/collection1/'
>    [junit4]   2> 415670 T2381 oasc.SolrResourceLoader.replaceClassLoader
> Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-
> trunk-Java8/solr/build/solr-
> core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> 103A566D0461F871-001/tempDir-003/collection1/lib/classes/' to classloader
>    [junit4]   2> 415670 T2381 oasc.SolrResourceLoader.replaceClassLoader
> Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-
> trunk-Java8/solr/build/solr-
> core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> 103A566D0461F871-001/tempDir-003/collection1/lib/README' to classloader
>    [junit4]   2> 415671 T2381 oasc.SolrResourceLoader.replaceClassLoader
> Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-
> trunk-Java8/solr/build/solr-
> core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> 103A566D0461F871-001/tempDir-003/collection1/lib/.svn/' to classloader
>    [junit4]   2> 415702 T2381 oasc.ZkController.watchZKConfDir watch zkdir
> /configs/conf1
>    [junit4]   2> 415704 T2381 oasc.Config.<init> loaded config solrconfig.xml
> with version 0
>    [junit4]   2> 415718 T2381 oasc.SolrConfig.<init> Using Lucene
> MatchVersion: 6.0.0
>    [junit4]   2> 415729 T2381 oasc.SolrConfig.<init> Loaded SolrConfig:
> solrconfig.xml
>    [junit4]   2> 415831 T2381 oass.IndexSchema.readSchema Reading Solr
> Schema from /configs/conf1/schema.xml
>    [junit4]   2> 415837 T2381 oass.IndexSchema.readSchema [collection1]
> Schema name=test
>    [junit4]   2> 415952 T2381 oass.OpenExchangeRatesOrgProvider.init
> Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
>    [junit4]   2> 415962 T2381 oass.IndexSchema.readSchema default search
> field in schema is text
>    [junit4]   2> 415964 T2381 oass.IndexSchema.readSchema unique key field:
> id
>    [junit4]   2> 415971 T2381 oass.FileExchangeRateProvider.reload Reloading
> exchange rates from file currency.xml
>    [junit4]   2> 415974 T2381 oass.FileExchangeRateProvider.reload Reloading
> exchange rates from file currency.xml
>    [junit4]   2> 415976 T2381 oass.OpenExchangeRatesOrgProvider.reload
> Reloading exchange rates from open-exchange-rates.json
>    [junit4]   2> 415977 T2381
> oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN
> Unknown key IMPORTANT NOTE
>    [junit4]   2> 415978 T2381
> oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN
> Expected key, got STRING
>    [junit4]   2> 415978 T2381 oass.OpenExchangeRatesOrgProvider.reload
> Reloading exchange rates from open-exchange-rates.json
>    [junit4]   2> 415978 T2381
> oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN
> Unknown key IMPORTANT NOTE
>    [junit4]   2> 415979 T2381
> oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN
> Expected key, got STRING
>    [junit4]   2> 415979 T2381 oasc.CoreContainer.create Creating SolrCore
> 'collection1' using configuration from collection collection1
>    [junit4]   2> 415979 T2381 oasc.SolrCore.initDirectoryFactory
> org.apache.solr.core.MockDirectoryFactory
>    [junit4]   2> 415979 T2381 oasc.SolrCore.<init> [collection1] Opening new
> SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-
> trunk-Java8/solr/build/solr-
> core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> 103A566D0461F871-001/tempDir-003/collection1/,
> dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-
> trunk-Java8/solr/build/solr-
> core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> 103A566D0461F871-001/tempDir-001/jetty1/
>    [junit4]   2> 415980 T2381 oasc.JmxMonitoredMap.<init> JMX monitoring is
> enabled. Adding Solr mbeans to JMX Server:
> com.sun.jmx.mbeanserver.JmxMBeanServer@330fda4c
>    [junit4]   2> 415981 T2381 oasc.CachingDirectoryFactory.get return new
> directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-
> Tests-trunk-Java8/solr/build/solr-
> core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> 103A566D0461F871-001/tempDir-001/jetty1
>    [junit4]   2> 415981 T2381 oasc.SolrCore.getNewIndexDir New index
> directory detected: old=null new=/usr/home/jenkins/jenkins-
> slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-
> core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> 103A566D0461F871-001/tempDir-001/jetty1/index/
>    [junit4]   2> 415981 T2381 oasc.SolrCore.initIndex WARN [collection1] Solr
> index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-
> Tests-trunk-Java8/solr/build/solr-
> core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> 103A566D0461F871-001/tempDir-001/jetty1/index' doesn't exist. Creating
> new index...
>    [junit4]   2> 415981 T2381 oasc.CachingDirectoryFactory.get return new
> directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-
> Tests-trunk-Java8/solr/build/solr-
> core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> 103A566D0461F871-001/tempDir-001/jetty1/index
>    [junit4]   2> 415982 T2381 oasu.RandomMergePolicy.<init>
> RandomMergePolicy wrapping class
> org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy:
> maxMergeAtOnce=30, maxMergeAtOnceExplicit=41,
> maxMergedSegmentMB=64.537109375, floorSegmentMB=1.693359375,
> forceMergeDeletesPctAllowed=3.111209946074762, segmentsPerTier=41.0,
> maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
>    [junit4]   2> 415983 T2381 oasc.SolrDeletionPolicy.onCommit
> SolrDeletionPolicy.onCommit: commits: num=1
>    [junit4]   2>
> 	commit{dir=MockDirectoryWrapper(RAMDirectory@1e365fa1
> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@66e9bbe),
> segFN=segments_1,generation=1}
>    [junit4]   2> 415983 T2381 oasc.SolrDeletionPolicy.updateCommits newest
> commit generation = 1
>    [junit4]   2> 415987 T2381 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "nodistrib"
>    [junit4]   2> 415987 T2381 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "dedupe"
>    [junit4]   2> 415987 T2381 oasup.UpdateRequestProcessorChain.init
> inserting DistributedUpdateProcessorFactory into
> updateRequestProcessorChain "dedupe"
>    [junit4]   2> 415988 T2381 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "stored_sig"
>    [junit4]   2> 415988 T2381 oasup.UpdateRequestProcessorChain.init
> inserting DistributedUpdateProcessorFactory into
> updateRequestProcessorChain "stored_sig"
>    [junit4]   2> 415988 T2381 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
>    [junit4]   2> 415989 T2381 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>    [junit4]   2> 415989 T2381 oasup.UpdateRequestProcessorChain.init
> inserting DistributedUpdateProcessorFactory into
> updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>    [junit4]   2> 415989 T2381 oasc.SolrCore.loadUpdateProcessorChains no
> updateRequestProcessorChain defined as default, creating implicit default
>    [junit4]   2> 415990 T2381 oasc.RequestHandlers.initHandlersFromConfig
> created /config: org.apache.solr.handler.SolrConfigHandler
>    [junit4]   2> 415990 T2381 oasc.RequestHandlers.initHandlersFromConfig
> created /update/csv: org.apache.solr.handler.UpdateRequestHandler
>    [junit4]   2> 415990 T2381 oasc.RequestHandlers.initHandlersFromConfig
> created /update/json: org.apache.solr.handler.UpdateRequestHandler
>    [junit4]   2> 415991 T2381 oasc.RequestHandlers.initHandlersFromConfig
> created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
>    [junit4]   2> 415991 T2381 oasc.RequestHandlers.initHandlersFromConfig
> created standard: solr.StandardRequestHandler
>    [junit4]   2> 415992 T2381 oasc.RequestHandlers.initHandlersFromConfig
> created /get: solr.RealTimeGetHandler
>    [junit4]   2> 415992 T2381 oasc.RequestHandlers.initHandlersFromConfig
> adding lazy requestHandler: solr.ReplicationHandler
>    [junit4]   2> 415992 T2381 oasc.RequestHandlers.initHandlersFromConfig
> created /replication: solr.ReplicationHandler
>    [junit4]   2> 415993 T2381 oasc.RequestHandlers.initHandlersFromConfig
> created /update: solr.UpdateRequestHandler
>    [junit4]   2> 415993 T2381 oasc.RequestHandlers.initHandlersFromConfig
> created /admin/: org.apache.solr.handler.admin.AdminHandlers
>    [junit4]   2> 415993 T2381 oasc.RequestHandlers.initHandlersFromConfig
> created /admin/file: solr.admin.ShowFileRequestHandler
>    [junit4]   2> 415994 T2381 oasc.RequestHandlers.initHandlersFromConfig
> created /admin/fileedit: solr.admin.EditFileRequestHandler
>    [junit4]   2> 416004 T2381 oashl.XMLLoader.init
> xsltCacheLifetimeSeconds=60
>    [junit4]   2> 416005 T2381 oashl.XMLLoader.init
> xsltCacheLifetimeSeconds=60
>    [junit4]   2> 416006 T2381 oashl.XMLLoader.init
> xsltCacheLifetimeSeconds=60
>    [junit4]   2> 416008 T2381 oashl.XMLLoader.init
> xsltCacheLifetimeSeconds=60
>    [junit4]   2> 416011 T2381 oasu.CommitTracker.<init> Hard AutoCommit:
> disabled
>    [junit4]   2> 416011 T2381 oasu.CommitTracker.<init> Soft AutoCommit:
> disabled
>    [junit4]   2> 416012 T2381 oasu.RandomMergePolicy.<init>
> RandomMergePolicy wrapping class
> org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy:
> maxMergeAtOnce=16, maxMergeAtOnceExplicit=15,
> maxMergedSegmentMB=60.369140625, floorSegmentMB=1.87109375,
> forceMergeDeletesPctAllowed=16.64506491494333, segmentsPerTier=38.0,
> maxCFSSegmentSizeMB=8.796093022207999E12,
> noCFSRatio=0.3115152765900482
>    [junit4]   2> 416012 T2381 oasc.SolrDeletionPolicy.onInit
> SolrDeletionPolicy.onInit: commits: num=1
>    [junit4]   2>
> 	commit{dir=MockDirectoryWrapper(RAMDirectory@1e365fa1
> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@66e9bbe),
> segFN=segments_1,generation=1}
>    [junit4]   2> 416013 T2381 oasc.SolrDeletionPolicy.updateCommits newest
> commit generation = 1
>    [junit4]   2> 416013 T2381 oass.SolrIndexSearcher.<init> Opening
> Searcher@1e3921a8[collection1] main
>    [junit4]   2> 416013 T2381 oascc.ZkStateReader.readConfigName Load
> collection config from:/collections/collection1
>    [junit4]   2> 416014 T2381 oascc.ZkStateReader.readConfigName
> path=/collections/collection1 configName=conf1 specified config exists in
> ZooKeeper
>    [junit4]   2> 416014 T2381 oasr.ManagedResourceStorage.newStorageIO
> Setting up ZooKeeper-based storage for the RestManager with znodeBase:
> /configs/conf1
>    [junit4]   2> 416014 T2381
> oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured
> ZooKeeperStorageIO with znodeBase: /configs/conf1
>    [junit4]   2> 416015 T2381 oasr.RestManager.init Initializing RestManager
> with initArgs: {}
>    [junit4]   2> 416015 T2381 oasr.ManagedResourceStorage.load Reading
> _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
>    [junit4]   2> 416016 T2381
> oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream
> Read 38 bytes from znode /configs/conf1/_rest_managed.json
>    [junit4]   2> 416016 T2381 oasr.ManagedResourceStorage.load Loaded
> LinkedHashMap at path _rest_managed.json using
> ZooKeeperStorageIO:path=/configs/conf1
>    [junit4]   2> 416016 T2381 oasr.ManagedResource.reloadFromStorage
> Loaded initArgs {} for /rest/managed
>    [junit4]   2> 416016 T2381 oasr.RestManager.init Initializing 0 registered
> ManagedResources
>    [junit4]   2> 416019 T2382 oasc.SolrCore.registerSearcher [collection1]
> Registered new searcher Searcher@1e3921a8[collection1]
> main{ExitableDirectoryReader(UninvertingDirectoryReader())}
>    [junit4]   2> 416020 T2381 oasc.CoreContainer.registerCore registering core:
> collection1
>    [junit4]   2> 416020 T2385 oasc.ZkController.register Register replica -
> core:collection1 address:http://127.0.0.1:35189/orln collection:collection1
> shard:shard2
>    [junit4]   2> 416021 T2325 oass.SolrDispatchFilter.init
> user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-
> trunk-Java8/solr/build/solr-core/test/J3
>    [junit4]   2> 416021 T2325 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
> done
>    [junit4]   2> 416021 T2385 oascc.SolrZkClient.makePath makePath:
> /collections/collection1/leader_elect/shard2/election
>    [junit4]   2> 416025 T2385
> oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader
> process for shard shard2
>    [junit4]   2> 416026 T2351 oasc.DistributedQueue$LatchWatcher.process
> NodeChildrenChanged fired on path /overseer/queue state SyncConnected
>    [junit4]   2> 416026 T2385
> oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough
> replicas found to continue.
>    [junit4]   2> 416026 T2385
> oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader
> - try and sync
>    [junit4]   2> ASYNC  NEW_CORE C137 name=collection1
> org.apache.solr.core.SolrCore@4e55ac0b
> url=http://127.0.0.1:35189/orln/collection1 node=127.0.0.1:35189_orln
> C137_STATE=coll:collection1 core:collection1 props:{core=collection1,
> base_url=http://127.0.0.1:35189/orln, node_name=127.0.0.1:35189_orln,
> state=down}
>    [junit4]   2> 416026 T2385 C137 P35189 oasc.SyncStrategy.sync Sync replicas
> to http://127.0.0.1:35189/orln/collection1/
>    [junit4]   2> 416027 T2385 C137 P35189 oasc.SyncStrategy.syncReplicas Sync
> Success - now sync replicas to me
>    [junit4]   2> 416027 T2385 C137 P35189 oasc.SyncStrategy.syncToMe
> http://127.0.0.1:35189/orln/collection1/ has no replicas
>    [junit4]   2> 416027 T2385
> oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader:
> http://127.0.0.1:35189/orln/collection1/ shard2
>    [junit4]   2> 416027 T2385 oascc.SolrZkClient.makePath makePath:
> /collections/collection1/leaders/shard2
>    [junit4]   2> 416030 T2352
> oasc.Overseer$ClusterStateUpdater.updateZkStates going to
> update_collection /collections/collection1/state.json
>    [junit4]   2> 416030 T2380 oascc.ZkStateReader$7.process A cluster state
> change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/collections/collection1/state.json, has occurred - updating...
>    [junit4]   2> 416031 T2380 oascc.ZkStateReader.updateWatchedCollection
> Updating data for collection1 to ver 2
>    [junit4]   2> 416035 T2351 oasc.DistributedQueue$LatchWatcher.process
> NodeChildrenChanged fired on path /overseer/queue state SyncConnected
>    [junit4]   2> 416035 T2352
> oasc.Overseer$ClusterStateUpdater.updateZkStates going to
> update_collection /collections/collection1/state.json
>    [junit4]   2> 416035 T2380 oascc.ZkStateReader$7.process A cluster state
> change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/collections/collection1/state.json, has occurred - updating...
>    [junit4]   2> 416036 T2380 oascc.ZkStateReader.updateWatchedCollection
> Updating data for collection1 to ver 3
>    [junit4]   2> 416081 T2385 oasc.ZkController.register We are
> http://127.0.0.1:35189/orln/collection1/ and leader is
> http://127.0.0.1:35189/orln/collection1/
>    [junit4]   2> 416082 T2385 oasc.ZkController.register No LogReplay needed
> for core=collection1 baseURL=http://127.0.0.1:35189/orln
>    [junit4]   2> 416082 T2385 oasc.ZkController.checkRecovery I am the leader,
> no recovery necessary
>    [junit4]   2> 416082 T2385 oasc.ZkController.publish publishing
> core=collection1 state=active collection=collection1
>    [junit4]   2> 416082 T2385 oasc.ZkController.publish numShards not found
> on descriptor - reading it from system property
>    [junit4]   2> 416083 T2351 oasc.DistributedQueue$LatchWatcher.process
> NodeChildrenChanged fired on path /overseer/queue state SyncConnected
>    [junit4]   2> 416085 T2385 oascc.ZkStateReader.updateWatchedCollection
> Updating data for collection1 to ver 3
>    [junit4]   2> 416085 T2352 oasc.Overseer$ClusterStateUpdater.updateState
> Update state numShards=2 message={
>    [junit4]   2> 	  "core":"collection1",
>    [junit4]   2> 	  "core_node_name":"core_node1",
>    [junit4]   2> 	  "roles":null,
>    [junit4]   2> 	  "base_url":"http://127.0.0.1:35189/orln",
>    [junit4]   2> 	  "node_name":"127.0.0.1:35189_orln",
>    [junit4]   2> 	  "numShards":"2",
>    [junit4]   2> 	  "state":"active",
>    [junit4]   2> 	  "shard":"shard2",
>    [junit4]   2> 	  "collection":"collection1",
>    [junit4]   2> 	  "operation":"state"}
>    [junit4]   2> 416088 T2352
> oasc.Overseer$ClusterStateUpdater.updateZkStates going to
> update_collection /collections/collection1/state.json
>    [junit4]   2> 416089 T2380 oascc.ZkStateReader$7.process A cluster state
> change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/collections/collection1/state.json, has occurred - updating...
>    [junit4]   2> 416089 T2380 oascc.ZkStateReader.updateWatchedCollection
> Updating data for collection1 to ver 4
>    [junit4]   2> 417139 T2325 oasc.AbstractFullDistribZkTestBase.createJettys
> create jetty 2
>    [junit4]   2> 417140 T2325 oejs.Server.doStart jetty-8.1.10.v20130312
>    [junit4]   2> 417142 T2325 oejs.AbstractConnector.doStart Started
> SocketConnector@127.0.0.1:28438
>    [junit4]   2> 417143 T2325 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
>    [junit4]   2> 417143 T2325 oasc.SolrResourceLoader.locateSolrHome JNDI
> not configured for solr (NoInitialContextEx)
>    [junit4]   2> 417143 T2325 oasc.SolrResourceLoader.locateSolrHome using
> system property solr.solr.home:
> ../../../../../../../../../../../usr/home/jenkins/jenkins-
> slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-
> core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> 103A566D0461F871-001/tempDir-004
>    [junit4]   2> 417143 T2325 oasc.SolrResourceLoader.<init> new
> SolrResourceLoader for directory:
> '../../../../../../../../../../../usr/home/jenkins/jenkins-
> slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-
> core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> 103A566D0461F871-001/tempDir-004/'
>    [junit4]   2> 417173 T2325 oasc.ConfigSolr.fromFile Loading container
> configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-
> Solr-Tests-trunk-Java8/solr/build/solr-
> core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-
> slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-
> core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> 103A566D0461F871-001/tempDir-004/solr.xml
>    [junit4]   2> 417186 T2325 oasc.CoreContainer.<init> New CoreContainer
> 967093745
>    [junit4]   2> 417187 T2325 oasc.CoreContainer.load Loading cores into
> CoreContainer
> [instanceDir=../../../../../../../../../../../usr/home/jenkins/jenkins-
> slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-
> core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> 103A566D0461F871-001/tempDir-004/]
>    [junit4]   2> 417187 T2325 oashc.HttpShardHandlerFactory.getParameter
> Setting socketTimeout to: 90000
>    [junit4]   2> 417187 T2325 oashc.HttpShardHandlerFactory.getParameter
> Setting urlScheme to:
>    [junit4]   2> 417188 T2325 oashc.HttpShardHandlerFactory.getParameter
> Setting connTimeout to: 15000
>    [junit4]   2> 417188 T2325 oashc.HttpShardHandlerFactory.getParameter
> Setting maxConnectionsPerHost to: 20
>    [junit4]   2> 417188 T2325 oashc.HttpShardHandlerFactory.getParameter
> Setting maxConnections to: 10000
>    [junit4]   2> 417188 T2325 oashc.HttpShardHandlerFactory.getParameter
> Setting corePoolSize to: 0
>    [junit4]   2> 417189 T2325 oashc.HttpShardHandlerFactory.getParameter
> Setting maximumPoolSize to: 2147483647
>    [junit4]   2> 417189 T2325 oashc.HttpShardHandlerFactory.getParameter
> Setting maxThreadIdleTime to: 5
>    [junit4]   2> 417189 T2325 oashc.HttpShardHandlerFactory.getParameter
> Setting sizeOfQueue to: -1
>    [junit4]   2> 417189 T2325 oashc.HttpShardHandlerFactory.getParameter
> Setting fairnessPolicy to: false
>    [junit4]   2> 417190 T2325 oasu.UpdateShardHandler.<init> Creating
> UpdateShardHandler HTTP client with params:
> socketTimeout=340000&connTimeout=45000&retry=false
>    [junit4]   2> 417190 T2325 oasl.LogWatcher.createWatcher SLF4J impl is
> org.slf4j.impl.Log4jLoggerFactory
>    [junit4]   2> 417190 T2325 oasl.LogWatcher.newRegisteredLogWatcher
> Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>    [junit4]   2> 417190 T2325 oasc.CoreContainer.load Host Name: 127.0.0.1
>    [junit4]   2> 417191 T2325 oasc.ZkContainer.initZooKeeper Zookeeper
> client=127.0.0.1:47646/solr
>    [junit4]   2> 417191 T2325 oasc.ZkController.checkChrootPath zkHost
> includes chroot
>    [junit4]   2> 417191 T2325
> oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default
> ZkCredentialsProvider
>    [junit4]   2> 417192 T2325 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
>    [junit4]   2> 417194 T2397 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@d035dfa
> name:ZooKeeperConnection Watcher:127.0.0.1:47646 got event
> WatchedEvent state:SyncConnected type:None path:null path:null
> type:None
>    [junit4]   2> 417195 T2325 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
>    [junit4]   2> 417195 T2325 oascc.SolrZkClient.createZkACLProvider Using
> default ZkACLProvider
>    [junit4]   2> 417197 T2325 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
>    [junit4]   2> 417199 T2400 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@5f3d26d0
> name:ZooKeeperConnection Watcher:127.0.0.1:47646/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null
> type:None
>    [junit4]   2> 417199 T2325 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
>    [junit4]   2> 417203 T2325
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating
> cluster state from ZooKeeper...
>    [junit4]   2> 418206 T2325 oasc.ZkController.createEphemeralLiveNode
> Register node as live in ZooKeeper:/live_nodes/127.0.0.1:28438_orln
>    [junit4]   2> 418207 T2325 oascc.SolrZkClient.makePath makePath:
> /live_nodes/127.0.0.1:28438_orln
>    [junit4]   2> 418210 T2325 oasc.Overseer.close Overseer (id=null) closing
>    [junit4]   2> 418215 T2401 oasc.ZkController.publish publishing
> core=collection1 state=down collection=collection1
>    [junit4]   2> 418215 T2401 oasc.ZkController.publish numShards not found
> on descriptor - reading it from system property
>    [junit4]   2> 418216 T2351 oasc.DistributedQueue$LatchWatcher.process
> NodeChildrenChanged fired on path /overseer/queue state SyncConnected
>    [junit4]   2> 418216 T2401 oasc.ZkController.preRegister Registering watch
> for external collection collection1
>    [junit4]   2> 418217 T2401 oascc.ZkStateReader.addZkWatch addZkWatch
> collection1
>    [junit4]   2> 418217 T2401 oascc.ZkStateReader.addZkWatch Updating
> collection state at /collections/collection1/state.json from ZooKeeper...
>    [junit4]   2> 418218 T2401 oascc.ZkStateReader.updateWatchedCollection
> Updating data for collection1 to ver 4
>    [junit4]   2> 418218 T2401 oasc.ZkController.waitForCoreNodeName look for
> our core node name
>    [junit4]   2> 418218 T2352 oasc.Overseer$ClusterStateUpdater.updateState
> Update state numShards=2 message={
>    [junit4]   2> 	  "core":"collection1",
>    [junit4]   2> 	  "roles":null,
>    [junit4]   2> 	  "base_url":"http://127.0.0.1:28438/orln",
>    [junit4]   2> 	  "node_name":"127.0.0.1:28438_orln",
>    [junit4]   2> 	  "numShards":"2",
>    [junit4]   2> 	  "state":"down",
>    [junit4]   2> 	  "shard":null,
>    [junit4]   2> 	  "collection":"collection1",
>    [junit4]   2> 	  "operation":"state"}
>    [junit4]   2> 418220 T2352 oasc.Overseer$ClusterStateUpdater.updateState
> Collection already exists with numShards=2
>    [junit4]   2> 418220 T2352 oasc.Overseer$ClusterStateUpdater.updateState
> Assigning new node to shard shard=shard1
>    [junit4]   2> 418223 T2352
> oasc.Overseer$ClusterStateUpdater.updateZkStates going to
> update_collection /collections/collection1/state.json
>    [junit4]   2> 418224 T2400 oascc.ZkStateReader$7.process A cluster state
> change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/collections/collection1/state.json, has occurred - updating...
>    [junit4]   2> 418224 T2380 oascc.ZkStateReader$7.process A cluster state
> change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/collections/collection1/state.json, has occurred - updating...
>    [junit4]   2> 418224 T2400 oascc.ZkStateReader.updateWatchedCollection
> Updating data for collection1 to ver 5
>    [junit4]   2> 418225 T2380 oascc.ZkStateReader.updateWatchedCollection
> Updating data for collection1 to ver 5
>    [junit4]   2> 419218 T2401 oasc.ZkController.waitForShardId waiting to find
> shard id in clusterstate for collection1
>    [junit4]   2> 419219 T2401 oasc.ZkController.createCollectionZkNode Check
> for collection zkNode:collection1
>    [junit4]   2> 419219 T2401 oasc.ZkController.createCollectionZkNode
> Collection zkNode exists
>    [junit4]   2> 419220 T2401 oascc.ZkStateReader.readConfigName Load
> collection config from:/collections/collection1
>    [junit4]   2> 419220 T2401 oascc.ZkStateReader.readConfigName
> path=/collections/collection1 configName=conf1 specified config exists in
> ZooKeeper
>    [junit4]   2> 419220 T2401 oasc.SolrResourceLoader.<init> new
> SolrResourceLoader for directory:
> '../../../../../../../../../../../usr/home/jenkins/jenkins-
> slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-
> core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> 103A566D0461F871-001/tempDir-004/collection1/'
>    [junit4]   2> 419222 T2401 oasc.SolrResourceLoader.replaceClassLoader
> Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-
> trunk-Java8/solr/build/solr-
> core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> 103A566D0461F871-001/tempDir-004/collection1/lib/README' to classloader
>    [junit4]   2> 419223 T2401 oasc.SolrResourceLoader.replaceClassLoader
> Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-
> trunk-Java8/solr/build/solr-
> core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> 103A566D0461F871-001/tempDir-004/collection1/lib/.svn/' to classloader
>    [junit4]   2> 419224 T2401 oasc.SolrResourceLoader.replaceClassLoader
> Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-
> trunk-Java8/solr/build/solr-
> core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> 103A566D0461F871-001/tempDir-004/collection1/lib/classes/' to classloader
>    [junit4]   2> 419256 T2401 oasc.ZkController.watchZKConfDir watch zkdir
> /configs/conf1
>    [junit4]   2> 419258 T2401 oasc.Config.<init> loaded config solrconfig.xml
> with version 0
>    [junit4]   2> 419273 T2401 oasc.SolrConfig.<init> Using Lucene
> MatchVersion: 6.0.0
>    [junit4]   2> 419285 T2401 oasc.SolrConfig.<init> Loaded SolrConfig:
> solrconfig.xml
>    [junit4]   2> 419386 T2401 oass.IndexSchema.readSchema Reading Solr
> Schema from /configs/conf1/schema.xml
>    [junit4]   2> 419392 T2401 oass.IndexSchema.readSchema [collection1]
> Schema name=test
>    [junit4]   2> 419507 T2401 oass.OpenExchangeRatesOrgProvider.init
> Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
>    [junit4]   2> 419517 T2401 oass.IndexSchema.readSchema default search
> field in schema is text
>    [junit4]   2> 419518 T2401 oass.IndexSchema.readSchema unique key field:
> id
>    [junit4]   2> 419526 T2401 oass.FileExchangeRateProvider.reload Reloading
> exchange rates from file currency.xml
>    [junit4]   2> 419529 T2401 oass.FileExchangeRateProvider.reload Reloading
> exchange rates from file currency.xml
>    [junit4]   2> 419531 T2401 oass.OpenExchangeRatesOrgProvider.reload
> Reloading exchange rates from open-exchange-rates.json
>    [junit4]   2> 419532 T2401
> oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN
> Unknown key IMPORTANT NOTE
>    [junit4]   2> 419533 T2401
> oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN
> Expected key, got STRING
>    [junit4]   2> 419533 T2401 oass.OpenExchangeRatesOrgProvider.reload
> Reloading exchange rates from open-exchange-rates.json
>    [junit4]   2> 419533 T2401
> oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN
> Unknown key IMPORTANT NOTE
>    [junit4]   2> 419534 T2401
> oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN
> Expected key, got STRING
>    [junit4]   2> 419534 T2401 oasc.CoreContainer.create Creating SolrCore
> 'collection1' using configuration from collection collection1
>    [junit4]   2> 419534 T2401 oasc.SolrCore.initDirectoryFactory
> org.apache.solr.core.MockDirectoryFactory
>    [junit4]   2> 419534 T2401 oasc.SolrCore.<init> [collection1] Opening new
> SolrCore at ../../../../../../../../../../../usr/home/jenkins/jenkins-
> slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-
> core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> 103A566D0461F871-001/tempDir-004/collection1/,
> dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-
> trunk-Java8/solr/build/solr-
> core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> 103A566D0461F871-001/tempDir-001/jetty2/
>    [junit4]   2> 419535 T2401 oasc.JmxMonitoredMap.<init> JMX monitoring is
> enabled. Adding Solr mbeans to JMX Server:
> com.sun.jmx.mbeanserver.JmxMBeanServer@330fda4c
>    [junit4]   2> 419536 T2401 oasc.CachingDirectoryFactory.get return new
> directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-
> Tests-trunk-Java8/solr/build/solr-
> core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> 103A566D0461F871-001/tempDir-001/jetty2
>    [junit4]   2> 419536 T2401 oasc.SolrCore.getNewIndexDir New index
> directory detected: old=null new=/usr/home/jenkins/jenkins-
> slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-
> core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> 103A566D0461F871-001/tempDir-001/jetty2/index/
>    [junit4]   2> 419536 T2401 oasc.SolrCore.initIndex WARN [collection1] Solr
> index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-
> Tests-trunk-Java8/solr/build/solr-
> core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> 103A566D0461F871-001/tempDir-001/jetty2/index' doesn't exist. Creating
> new index...
>    [junit4]   2> 419537 T2401 oasc.CachingDirectoryFactory.get return new
> directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-
> Tests-trunk-Java8/solr/build/solr-
> core/test/J3/temp/solr.handler.TestSolrConfigHandlerConcurrent-
> 103A566D0461F871-001/tempDir-001/jetty2/index
>    [junit4]   2> 419537 T2401 oasu.RandomMergePolicy.<init>
> RandomMergePolicy wrapping class
> org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy:
> maxMergeAtOnce=30, maxMergeAtOnceExplicit=41,
> maxMergedSegmentMB=64.537109375, floorSegmentMB=1.693359375,
> forceMergeDeletesPctAllowed=3.111209946074762, segmentsPerTier=41.0,
> maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
>    [junit4]   2> 419538 T2401 oasc.SolrDeletionPolicy.onCommit
> SolrDeletionPolicy.onCommit: commits: num=1
>    [junit4]   2>
> 	commit{dir=MockDirectoryWrapper(RAMDirectory@4ad441db
> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@505677e5),
> segFN=segments_1,generation=1}
>    [junit4]   2> 419538 T2401 oasc.SolrDeletionPolicy.updateCommits newest
> commit generation = 1
>    [junit4]   2> 419542 T2401 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "nodistrib"
>    [junit4]   2> 419543 T2401 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "dedupe"
>    [junit4]   2> 419543 T2401 oasup.UpdateRequestProcessorChain.init
> inserting DistributedUpdateProcessorFactory into
> updateRequestProcessorChain "dedupe"
>    [junit4]   2> 419543 T2401 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "stored_sig"
>    [junit4]   2> 419543 T2401 oasup.UpdateRequestProcessorChain.init
> inserting DistributedUpdateProcessorFactory into
> updateRequestProcessorChain "stored_sig"
>    [junit4]   2> 419544 T2401 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
>    [junit4]   2> 419544 T2401 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>    [junit4]   2> 419544 T2401 oasup.UpdateRequestProcessorChain.init
> inserting DistributedUpdateProcessorFactory into
> updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>    [junit4]   2> 419545 T2401 oasc.SolrCore.loadUpdateProcessorChains no
> updateRequestProcessorChain defined as default, creating implicit default
>    [junit4]   2> 419545 T2401 oasc.RequestHandlers.initHandlersFromConfig
> created /config: org.apache.solr.handler.SolrConfigHandler
>    [junit4]   2> 419546 T2401 oasc.RequestHandlers.initHandlersFromConfig
> created /update/csv: org.apache.solr.handler.UpdateRequestHandler
>    [junit4]   2> 419546 T2401 oasc.RequestHandlers.initHandlersFromConfig
> created /update/json: org.apache.solr.handler.UpdateRequestHandler
>    [junit4]   2> 419546 T2401 oasc.RequestHandlers.initHandlersFromConfig
> created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
>    [junit4]   2> 419547 T2401 oasc.RequestHandlers.initHandlersFromConfig
> created standard: solr.StandardRequestHandler
>    [junit4]   2> 419548 T2401 oasc.RequestHandlers.initHandlersFromConfig
> created /get: solr.RealTimeGetHandler
>    [junit4]   2> 419548 T2401 oasc.RequestHandlers.initHandlersFromConfig
> adding lazy requestHandler: solr.ReplicationHandler
>    [junit4]   2> 419548 T2401 oasc.RequestHandlers.initHandlersFromConfig
> created /replication: solr.ReplicationHandler
>    [junit4]   2> 419548 T2401 oasc.RequestHandlers.initHandlersFromConfig
> created /update: solr.UpdateRequestHandler
>    [junit4]   2> 419549 T2401 oasc.RequestHandlers.initHandlersFromConfig
> created /admin/: org.apache.solr.handler.admin.AdminHandlers
>    [junit4]   2> 419549 T2401 oasc.RequestHandlers.initHandlersFromConfig
> created /admin/file: solr.admin.ShowFileRequestHandler
>    [junit4]   2> 419550 T2401 oasc.RequestHandlers.initHandlersFromConfig
> created /admin/fileedit: solr.admin.EditFileRequestHandler
>    [junit4]   2> 419560 T2401 oashl.XMLLoader.init
> xsltCacheLifetimeSeconds=60
>    [junit4]   2> 419562 T2401 oashl.XMLLoader.init
> xsltCacheLifetimeSeconds=60
>    [junit4]   2> 419563 T2401 oashl.XMLLoader.init
> xsltCacheLifetimeSeconds=60
>    [junit4]   2> 419564 T2401 oashl.XMLLoader.init
> xsltCacheLifetimeSeconds=60
>    [junit4]   2> 419567 T2401 oasu.CommitTracker.<init> Hard AutoCommit:
> disabled
>    [junit4]   2> 419568 T2401 oasu.CommitTracker.<init> Soft AutoCommit:
> disabled
>    [junit4]   2> 419569 T2401 oasu.RandomMergePolicy.<init>
> RandomMergePolicy wrapping class
> org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy:
> maxMergeAtOnce=16, maxMergeAtOnceExplicit=15,
> maxMergedSegmentMB=60.369140625, floorSegmentMB=1.87109375,
> forceMergeDeletesPctAllowed=16.64506491494333, segmentsPerTier=38.0,
> maxCFSSegmentSizeMB=8.796093022207999E12,
> noCFSRatio=0.3115152765900482
>    [junit4]   2> 419569 T2401 oasc.SolrDeletionPolicy.onInit
> SolrDeletionPolicy.onInit: commits: num=1
>    [junit4]   2>
> 	commit{dir=MockDirectoryWrapper(RAMDirectory@4ad441db
> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@505677e5),
> segFN=segments_1,generation=1}
>    [junit4]   2> 419570 T2401 oasc.SolrDeletionPolicy.updateCommits newest
> commit generation = 1
>    [junit4]   2> 419570 T2401 oass.SolrIndexSearcher.<init> Opening
> Searcher@62dc7557[collection1] main
>    [junit4]   2> 419570 T2401 oascc.ZkStateReader.readConfigName Load
> collection config from:/collections/collection1
>    [junit4]   2> 419571 T2401 oascc.ZkStateReader.readConfigName
> path=/collections/collection1 configName=conf1 specified config exists in
> ZooKeeper
>    [junit4]   2> 419571 T2401 oasr.ManagedResourceStorage.newStorageIO
> Setting up ZooKeeper-based storage for the RestManager with znodeBase:
> /configs/conf1
>    [junit4]   2> 419572 T2401
> oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured
> ZooKeeperStorageIO with znodeBase: /configs/conf1
>    [junit4]   2> 419572 T2401 oasr.RestManager.init Initializing RestManager
> with initArgs: {}
>    [junit4]   2> 419572 T2401 oasr.ManagedResourceStorage.load Reading
> _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
>    [junit4]   2> 419573 T2401
> oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream
> Read 38 bytes from znode /configs/conf1/_rest_managed.json
>    [junit4]   2> 419573 T2401 oasr.ManagedResourceStorage.load Loaded
> LinkedHashMap at path _rest_managed.json using
> ZooKeeperStorageIO:path=/configs/conf1
>    [junit4]   2> 419573 T2401 oasr.ManagedResource.reloadFromStorage
> Loaded initArgs {} for /rest/managed
>    [junit4]   2> 419573 T2401 oasr.RestManager.init Initializing 0 registered
> ManagedResources
>    [junit4]   2> 419577 T2402 oasc.SolrCore.registerSearcher [collection1]
> Registered new searcher Searcher@62dc7557[collection1]
> main{ExitableDirectoryReader(UninvertingDirectoryReader())}
>    [junit4]   2> 419577 T2401 oasc.CoreContainer.registerCore registering core:
> collection1
>    [junit4]   2> 419578 T2405 oasc.ZkController.register Register replica -
> core:collection1 address:http://127.0.0.1:28438/orln collection:co
> 
> [...truncated too long message...]
> 
> -Tests-trunk-Java8/solr/core/lib/org.restlet.ext.servlet-
> 2.1.1.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-
> trunk-Java8/solr/core/lib/protobuf-java-
> 2.5.0.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-
> trunk-Java8/solr/core/lib/slf4j-log4j12-1.7.6.jar:/usr/home/jenkins/jenkins-
> slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/lib/spatial4j-
> 0.4.1.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-
> trunk-Java8/solr/solrj/lib/commons-io-2.4.jar:/usr/home/jenkins/jenkins-
> slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/solrj/lib/httpclient-
> 4.3.1.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-
> trunk-Java8/solr/solrj/lib/httpcore-4.3.jar:/usr/home/jenkins/jenkins-
> slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/solrj/lib/httpmime-
> 4.3.1.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-
> trunk-Java8/solr/solrj/lib/noggit-0.6.jar:/usr/home/jenkins/jenkins-
> slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/solrj/lib/slf4j-api-
> 1.7.6.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-
> trunk-Java8/solr/solrj/lib/wstx-asl-3.2.7.jar:/usr/home/jenkins/jenkins-
> slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/solrj/lib/zookeeper-
> 3.4.6.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-
> trunk-Java8/solr/server/lib/jetty-continuation-
> 8.1.10.v20130312.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-
> Solr-Tests-trunk-Java8/solr/server/lib/jetty-deploy-
> 8.1.10.v20130312.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-
> Solr-Tests-trunk-Java8/solr/server/lib/jetty-http-
> 8.1.10.v20130312.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-
> Solr-Tests-trunk-Java8/solr/server/lib/jetty-io-
> 8.1.10.v20130312.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-
> Solr-Tests-trunk-Java8/solr/server/lib/jetty-jmx-
> 8.1.10.v20130312.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-
> Solr-Tests-trunk-Java8/solr/server/lib/jetty-security-
> 8.1.10.v20130312.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-
> Solr-Tests-trunk-Java8/solr/server/lib/jetty-server-
> 8.1.10.v20130312.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-
> Solr-Tests-trunk-Java8/solr/server/lib/jetty-servlet-
> 8.1.10.v20130312.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-
> Solr-Tests-trunk-Java8/solr/server/lib/jetty-util-
> 8.1.10.v20130312.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-
> Solr-Tests-trunk-Java8/solr/server/lib/jetty-webapp-
> 8.1.10.v20130312.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-
> Solr-Tests-trunk-Java8/solr/server/lib/jetty-xml-
> 8.1.10.v20130312.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-
> Solr-Tests-trunk-Java8/solr/server/lib/servlet-api-
> 3.0.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-
> trunk-Java8/solr/example/example-DIH/solr/db/lib/derby-
> 10.9.1.0.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-
> trunk-Java8/solr/example/example-DIH/solr/db/lib/hsqldb-
> 1.8.0.10.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-
> trunk-Java8/lucene/build/core/classes/java:/usr/home/jenkins/jenkins-
> slave/workspace/Lucene-Solr-Tests-trunk-Java8/lucene/test-
> framework/lib/junit-4.10.jar:/usr/home/jenkins/jenkins-
> slave/workspace/Lucene-Solr-Tests-trunk-Java8/lucene/test-
> framework/lib/randomizedtesting-runner-
> 2.1.9.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-
> trunk-Java8/solr/core/test-lib/cglib-nodep-
> 2.2.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-
> trunk-Java8/solr/core/test-lib/commons-collections-
> 3.2.1.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-
> trunk-Java8/solr/core/test-lib/easymock-3.0.jar:/usr/home/jenkins/jenkins-
> slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/test-lib/hadoop-
> common-2.2.0-tests.jar:/usr/home/jenkins/jenkins-
> slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/test-lib/hadoop-
> hdfs-2.2.0-tests.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-
> Solr-Tests-trunk-Java8/solr/core/test-lib/javax.servlet-api-
> 3.0.1.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-
> trunk-Java8/solr/core/test-lib/jcl-over-slf4j-
> 1.7.6.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-
> trunk-Java8/solr/core/test-lib/jersey-core-
> 1.9.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-
> trunk-Java8/solr/core/test-lib/jetty-6.1.26.jar:/usr/home/jenkins/jenkins-
> slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/test-lib/jetty-util-
> 6.1.26.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-
> trunk-Java8/solr/core/test-lib/objenesis-1.2.jar:/usr/home/jenkins/jenkins-
> slave/workspace/Lucene-Solr-Tests-trunk-
> Java8/lucene/build/analysis/icu/lucene-analyzers-icu-6.0.0-
> SNAPSHOT.jar:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-
> Tests-trunk-Java8/solr/build/contrib/solr-analysis-
> extras/classes/java:/usr/home/jenkins/jenkins-slave/workspace/Lucene-
> Solr-Tests-trunk-Java8/solr/contrib/analysis-extras/lib/icu4j-
> 54.1.jar:/home/jenkins/jenkins-
> slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-
> launcher.jar:/home/jenkins/.ant/lib/ivy-2.3.0.jar:/home/jenkins/jenkins-
> slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-
> jsch.jar:/home/jenkins/jenkins-
> slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-
> jai.jar:/home/jenkins/jenkins-
> slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-
> regexp.jar:/home/jenkins/jenkins-
> slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-
> netrexx.jar:/home/jenkins/jenkins-
> slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-
> antlr.jar:/home/jenkins/jenkins-
> slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-
> junit4.jar:/home/jenkins/jenkins-
> slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-
> junit.jar:/home/jenkins/jenkins-
> slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-
> bsf.jar:/home/jenkins/jenkins-
> slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-
> log4j.jar:/home/jenkins/jenkins-
> slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-
> bcel.jar:/home/jenkins/jenkins-
> slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-
> resolver.jar:/home/jenkins/jenkins-
> slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-
> swing.jar:/home/jenkins/jenkins-
> slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-
> oro.jar:/home/jenkins/jenkins-
> slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-
> xalan2.jar:/home/jenkins/jenkins-
> slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-
> jmf.jar:/home/jenkins/jenkins-
> slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-
> jdepend.jar:/home/jenkins/jenkins-
> slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-
> testutil.jar:/home/jenkins/jenkins-
> slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-commons-
> logging.jar:/home/jenkins/jenkins-
> slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-commons-
> net.jar:/home/jenkins/jenkins-
> slave/tools/hudson.tasks.Ant_AntInstallation/ant-
> 1.8.2/lib/ant.jar:/home/jenkins/jenkins-
> slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-
> javamail.jar:/usr/local/openjdk8/lib/tools.jar:/home/jenkins/.ivy2/cache/co
> m.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.1.9.jar
> com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe -eventsfile
> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-
> Java8/solr/build/solr-core/test/temp/junit4-J2-20141121_173810_288.events
> @/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-
> Java8/solr/build/solr-core/test/temp/junit4-J2-20141121_173810_288.suites
>    [junit4] ERROR: JVM J2 ended with an exception: Forked process returned
> with error code: 1. Very likely a JVM crash.  Process output piped in logs
> above.
>    [junit4] 	at
> com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1481)
>    [junit4] 	at
> com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:133)
>    [junit4] 	at
> com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:964)
>    [junit4] 	at
> com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:961)
>    [junit4] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>    [junit4] 	at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
> a:1142)
>    [junit4] 	at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
> va:617)
>    [junit4] 	at java.lang.Thread.run(Thread.java:745)
> 
> BUILD FAILED
> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-
> Java8/build.xml:515: The following error occurred while executing this line:
> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-
> Java8/build.xml:463: The following error occurred while executing this line:
> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-
> Java8/build.xml:61: The following error occurred while executing this line:
> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-
> Java8/extra-targets.xml:39: The following error occurred while executing this
> line:
> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-
> Java8/solr/build.xml:186: The following error occurred while executing this
> line:
> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-
> Java8/solr/common-build.xml:508: The following error occurred while
> executing this line:
> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-
> Java8/lucene/common-build.xml:1345: The following error occurred while
> executing this line:
> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-
> Java8/lucene/common-build.xml:952: At least one slave process threw an
> exception, first: Forked process returned with error code: 134. Very likely a
> JVM crash.  Process output piped in logs above.
> 
> Total time: 92 minutes 30 seconds
> Build step 'Invoke Ant' marked build as failure
> Archiving artifacts
> No prior successful build to compare, so performing full copy of artifacts
> Recording test results
> Email was triggered for: Failure
> Sending email for trigger: Failure
> 



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org