You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2014/10/28 13:20:25 UTC

[JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 669 - Still Failing

Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/669/

2 tests failed.
FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testDistribSearch

Error Message:
Captured an uncaught exception in thread: Thread[id=7272, name=Thread-2448, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=7272, name=Thread-2448, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
Caused by: java.lang.NullPointerException
	at __randomizedtesting.SeedInfo.seed([359299200057C864]:0)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:1044)


FAILED:  org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testDistribSearch

Error Message:
IOException occured when talking to server at: http://127.0.0.1:16199/fjsd/a/collection1

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://127.0.0.1:16199/fjsd/a/collection1
	at __randomizedtesting.SeedInfo.seed([359299200057C864:B47417387708A858]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:583)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
	at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:91)
	at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
	at org.apache.solr.cloud.CloudInspectUtil.compareResults(CloudInspectUtil.java:223)
	at org.apache.solr.cloud.CloudInspectUtil.compareResults(CloudInspectUtil.java:165)
	at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testIndexingBatchPerRequestWithHttpSolrServer(FullSolrCloudDistribCmdsTest.java:414)
	at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.doTest(FullSolrCloudDistribCmdsTest.java:144)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketException: Broken pipe
	at java.net.SocketOutputStream.socketWrite0(Native Method)
	at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
	at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
	at org.apache.http.impl.io.AbstractSessionOutputBuffer.flushBuffer(AbstractSessionOutputBuffer.java:159)
	at org.apache.http.impl.io.AbstractSessionOutputBuffer.writeLine(AbstractSessionOutputBuffer.java:258)
	at org.apache.http.impl.io.HttpRequestWriter.writeHeadLine(HttpRequestWriter.java:59)
	at org.apache.http.impl.io.HttpRequestWriter.writeHeadLine(HttpRequestWriter.java:46)
	at org.apache.http.impl.io.AbstractMessageWriter.write(AbstractMessageWriter.java:109)
	at org.apache.http.impl.AbstractHttpClientConnection.sendRequestHeader(AbstractHttpClientConnection.java:254)
	at org.apache.http.impl.conn.DefaultClientConnection.sendRequestHeader(DefaultClientConnection.java:270)
	at org.apache.http.impl.conn.ManagedClientConnectionImpl.sendRequestHeader(ManagedClientConnectionImpl.java:209)
	at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:203)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:121)
	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:682)
	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:486)
	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:465)
	... 49 more




Build Log:
[...truncated 11437 lines...]
   [junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
   [junit4]   2> Creating dataDir: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest-359299200057C864-001/init-core-data-001
   [junit4]   2> 1344741 T5913 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (true)
   [junit4]   2> 1344742 T5913 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 1344746 T5913 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 1344746 T5913 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1344747 T5914 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1344847 T5913 oasc.ZkTestServer.run start zk server on port:24469
   [junit4]   2> 1344848 T5913 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1344848 T5913 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1344851 T5920 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2de8313a name:ZooKeeperConnection Watcher:127.0.0.1:24469 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1344851 T5913 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1344851 T5913 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1344851 T5913 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1344853 T5913 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1344854 T5913 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1344855 T5922 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@57265f83 name:ZooKeeperConnection Watcher:127.0.0.1:24469/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1344855 T5913 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1344855 T5913 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1344855 T5913 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 1344857 T5913 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 1344858 T5913 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 1344859 T5913 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 1344860 T5913 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1344860 T5913 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1344862 T5913 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 1344863 T5913 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 1344864 T5913 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1344864 T5913 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1344866 T5913 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1344866 T5913 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1344867 T5913 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1344868 T5913 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1344869 T5913 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1344869 T5913 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 1344871 T5913 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1344871 T5913 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1344872 T5913 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1344873 T5913 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1344874 T5913 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1344874 T5913 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1344876 T5913 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1344876 T5913 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1344877 T5913 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1344878 T5913 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1344879 T5913 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1344880 T5913 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1344880 T5924 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7ec7189a name:ZooKeeperConnection Watcher:127.0.0.1:24469/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1344881 T5913 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1344881 T5913 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> Using new style solr.xml
   [junit4]   2> 1345859 T5913 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1345863 T5913 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1345865 T5913 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:24473
   [junit4]   2> 1345867 T5913 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1345868 T5913 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1345868 T5913 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest-359299200057C864-001/tempDir-002
   [junit4]   2> 1345868 T5913 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest-359299200057C864-001/tempDir-002/'
   [junit4]   2> 1345898 T5913 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J0/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest-359299200057C864-001/tempDir-002/solr.xml
   [junit4]   2> 1345951 T5913 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J0/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest-359299200057C864-001/tempDir-002
   [junit4]   2> 1345951 T5913 oasc.CoreContainer.<init> New CoreContainer 1860977573
   [junit4]   2> 1345952 T5913 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest-359299200057C864-001/tempDir-002/]
   [junit4]   2> 1345952 T5913 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1345953 T5913 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1345953 T5913 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1345953 T5913 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1345953 T5913 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 1345954 T5913 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1345954 T5913 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1345954 T5913 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1345954 T5913 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1345954 T5913 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1345957 T5913 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 1345958 T5913 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1345958 T5913 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1345959 T5913 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1345959 T5913 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:24469/solr
   [junit4]   2> 1345959 T5913 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1345959 T5913 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1345960 T5913 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1345961 T5935 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3a02e632 name:ZooKeeperConnection Watcher:127.0.0.1:24469 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1345962 T5913 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1345962 T5913 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1345964 T5913 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1345964 T5937 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2be396ac name:ZooKeeperConnection Watcher:127.0.0.1:24469/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1345964 T5913 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1345965 T5913 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1345967 T5913 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1345968 T5913 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 1345970 T5913 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 1345971 T5913 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 1345973 T5913 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1345974 T5913 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:24473_
   [junit4]   2> 1345974 T5913 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:24473_
   [junit4]   2> 1345975 T5913 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 1345976 T5913 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1345977 T5913 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1345978 T5913 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:24473_
   [junit4]   2> 1345979 T5913 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1345980 T5913 oasc.Overseer.start Overseer (id=92700303572860932-127.0.0.1:24473_-n_0000000000) starting
   [junit4]   2> 1345981 T5913 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1345985 T5913 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=10000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 1345986 T5939 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1345986 T5913 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1345988 T5913 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1346000 T5913 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1346003 T5938 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1346006 T5913 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J0/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest-359299200057C864-001/tempDir-002
   [junit4]   2> 1346930 T5913 oasc.CorePropertiesLocator.discoverUnder Found core conf in /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J0/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest-359299200057C864-001/tempDir-002/conf/
   [junit4]   2> 1346949 T5913 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1346950 T5913 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J0
   [junit4]   2> 1346951 T5913 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1346954 T5913 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1346955 T5913 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1346956 T5943 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@62901579 name:ZooKeeperConnection Watcher:127.0.0.1:24469/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1346956 T5913 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1346956 T5913 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1346957 T5913 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1346959 T5913 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1347869 T5913 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 1347869 T5913 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1347874 T5913 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1347875 T5913 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:24477
   [junit4]   2> 1347877 T5913 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1347877 T5913 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1347877 T5913 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest-359299200057C864-001/tempDir-003
   [junit4]   2> 1347877 T5913 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest-359299200057C864-001/tempDir-003/'
   [junit4]   2> 1347906 T5913 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest-359299200057C864-001/tempDir-003/solr.xml
   [junit4]   2> 1347947 T5913 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest-359299200057C864-001/tempDir-003
   [junit4]   2> 1347947 T5913 oasc.CoreContainer.<init> New CoreContainer 880789601
   [junit4]   2> 1347947 T5913 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest-359299200057C864-001/tempDir-003/]
   [junit4]   2> 1347948 T5913 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1347949 T5913 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1347949 T5913 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1347949 T5913 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1347949 T5913 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 1347949 T5913 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1347950 T5913 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1347950 T5913 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1347950 T5913 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1347950 T5913 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1347952 T5913 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 1347954 T5913 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1347954 T5913 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1347954 T5913 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1347954 T5913 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:24469/solr
   [junit4]   2> 1347955 T5913 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1347955 T5913 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1347956 T5913 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1347957 T5954 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3baa5ee4 name:ZooKeeperConnection Watcher:127.0.0.1:24469 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1347958 T5913 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1347958 T5913 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1347959 T5913 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1347960 T5956 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@37f684e5 name:ZooKeeperConnection Watcher:127.0.0.1:24469/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1347960 T5913 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1347964 T5913 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1348967 T5913 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:24477_
   [junit4]   2> 1348968 T5913 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:24477_
   [junit4]   2> 1348971 T5913 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1348974 T5913 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest-359299200057C864-001/tempDir-003
   [junit4]   2> 1348985 T5913 oasc.CorePropertiesLocator.discoverUnder Found core conf in /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest-359299200057C864-001/tempDir-003/conf/
   [junit4]   2> 1349692 T5913 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1349693 T5913 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J0
   [junit4]   2> 1349693 T5913 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1350628 T5913 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 1350629 T5913 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1350633 T5913 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1350634 T5913 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:24480
   [junit4]   2> 1350636 T5913 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1350636 T5913 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1350636 T5913 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest-359299200057C864-001/tempDir-004
   [junit4]   2> 1350637 T5913 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest-359299200057C864-001/tempDir-004/'
   [junit4]   2> 1350665 T5913 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest-359299200057C864-001/tempDir-004/solr.xml
   [junit4]   2> 1350708 T5913 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest-359299200057C864-001/tempDir-004
   [junit4]   2> 1350708 T5913 oasc.CoreContainer.<init> New CoreContainer 549974263
   [junit4]   2> 1350709 T5913 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest-359299200057C864-001/tempDir-004/]
   [junit4]   2> 1350709 T5913 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1350710 T5913 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1350710 T5913 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1350710 T5913 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1350710 T5913 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 1350711 T5913 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1350711 T5913 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1350711 T5913 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1350711 T5913 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1350711 T5913 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1350713 T5913 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 1350715 T5913 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1350715 T5913 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1350715 T5913 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1350716 T5913 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:24469/solr
   [junit4]   2> 1350716 T5913 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1350716 T5913 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1350717 T5913 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1350718 T5971 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5f86e1fa name:ZooKeeperConnection Watcher:127.0.0.1:24469 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1350719 T5913 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1350719 T5913 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1350720 T5913 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1350721 T5973 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@fc2799 name:ZooKeeperConnection Watcher:127.0.0.1:24469/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1350721 T5913 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1350725 T5913 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1351728 T5913 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:24480_
   [junit4]   2> 1351729 T5913 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:24480_
   [junit4]   2> 1351731 T5913 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1351734 T5913 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest-359299200057C864-001/tempDir-004
   [junit4]   2> 1352295 T5913 oasc.CorePropertiesLocator.discoverUnder Found core conf in /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest-359299200057C864-001/tempDir-004/conf/
   [junit4]   2> 1352414 T5913 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1352415 T5913 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J0
   [junit4]   2> 1352416 T5913 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1353331 T5913 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 1353332 T5913 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1353336 T5913 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1353337 T5913 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:24483
   [junit4]   2> 1353339 T5913 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1353339 T5913 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1353340 T5913 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest-359299200057C864-001/tempDir-005
   [junit4]   2> 1353340 T5913 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest-359299200057C864-001/tempDir-005/'
   [junit4]   2> 1353369 T5913 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest-359299200057C864-001/tempDir-005/solr.xml
   [junit4]   2> 1353410 T5913 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest-359299200057C864-001/tempDir-005
   [junit4]   2> 1353411 T5913 oasc.CoreContainer.<init> New CoreContainer 71179863
   [junit4]   2> 1353411 T5913 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest-359299200057C864-001/tempDir-005/]
   [junit4]   2> 1353412 T5913 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1353412 T5913 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1353413 T5913 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1353413 T5913 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1353413 T5913 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 1353413 T5913 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1353413 T5913 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1353414 T5913 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1353414 T5913 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1353414 T5913 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1353416 T5913 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 1353417 T5913 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1353418 T5913 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1353418 T5913 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1353418 T5913 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:24469/solr
   [junit4]   2> 1353418 T5913 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1353419 T5913 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1353419 T5913 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1353421 T5986 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7c253f28 name:ZooKeeperConnection Watcher:127.0.0.1:24469 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1353421 T5913 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1353422 T5913 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1353423 T5913 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1353424 T5988 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ba4c274 name:ZooKeeperConnection Watcher:127.0.0.1:24469/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1353424 T5913 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1353428 T5913 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1354430 T5913 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:24483_
   [junit4]   2> 1354431 T5913 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:24483_
   [junit4]   2> 1354433 T5913 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1354437 T5913 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest-359299200057C864-001/tempDir-005
   [junit4]   2> 1354442 T5913 oasc.CorePropertiesLocator.discoverUnder Found core conf in /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest-359299200057C864-001/tempDir-005/conf/
   [junit4]   2> 1355426 T5913 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1355427 T5913 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J0
   [junit4]   2> 1355427 T5913 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1356367 T5913 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 4
   [junit4]   2> 1356368 T5913 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1356372 T5913 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1356373 T5913 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:24486
   [junit4]   2> 1356375 T5913 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1356376 T5913 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1356376 T5913 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest-359299200057C864-001/tempDir-006
   [junit4]   2> 1356376 T5913 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest-359299200057C864-001/tempDir-006/'
   [junit4]   2> 1356404 T5913 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest-359299200057C864-001/tempDir-006/solr.xml
   [junit4]   2> 1356446 T5913 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest-359299200057C864-001/tempDir-006
   [junit4]   2> 1356446 T5913 oasc.CoreContainer.<init> New CoreContainer 1764873175
   [junit4]   2> 1356447 T5913 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest-359299200057C864-001/tempDir-006/]
   [junit4]   2> 1356447 T5913 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1356448 T5913 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1356448 T5913 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1356448 T5913 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1356448 T5913 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 1356448 T5913 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1356449 T5913 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1356449 T5913 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1356449 T5913 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1356449 T5913 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1356451 T5913 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 1356453 T5913 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1356453 T5913 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1356453 T5913 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1356453 T5913 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:24469/solr
   [junit4]   2> 1356454 T5913 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1356454 T5913 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1356455 T5913 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1356456 T6001 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@69ee989b name:ZooKeeperConnection Watcher:127.0.0.1:24469 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1356457 T5913 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1356457 T5913 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1356458 T5913 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1356459 T6003 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3c8cf62c name:ZooKeeperConnection Watcher:127.0.0.1:24469/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1356459 T5913 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1356463 T5913 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1357466 T5913 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:24486_
   [junit4]   2> 1357467 T5913 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:24486_
   [junit4]   2> 1357469 T5913 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1357474 T5913 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest-359299200057C864-001/tempDir-006
   [junit4]   2> 1358369 T5913 oasc.CorePropertiesLocator.discoverUnder Found core conf in /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest-359299200057C864-001/tempDir-006/conf/
   [junit4]   2> 1358457 T5913 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1358458 T5913 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J0
   [junit4]   2> 1358458 T5913 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1358587 T5947 oasha.CollectionsHandler.handleCreateAction Creating Collection : numShards=2&version=2&router.field=myOwnField&action=CREATE&wt=javabin&replicationFactor=2&collection.configName=conf1&name=solrj_collection
   [junit4]   2> 1358588 T5959 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1358589 T5939 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"create",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"solrj_collection",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "collection.configName":"conf1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "router.field":"myOwnField"}
   [junit4]   2> 1358589 T6007 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : create , {
   [junit4]   2> 	  "operation":"create",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"solrj_collection",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "collection.configName":"conf1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "router.field":"myOwnField"}
   [junit4]   2> 1358590 T6007 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/solrj_collection 
   [junit4]   2> 1358590 T6007 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection
   [junit4]   2> 1358592 T5959 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1358593 T5938 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: solrj_collection
   [junit4]   2> 1358593 T5938 oasc.Overseer$ClusterStateUpdater.createCollection Create collection solrj_collection with shards [shard1, shard2]
   [junit4]   2> 1358593 T5938 oasc.Overseer$ClusterStateUpdater.createCollection state version solrj_collection 1
   [junit4]   2> 1358595 T5989 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1358595 T5974 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1358595 T5958 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1358595 T5959 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1358595 T5957 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1358595 T6004 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1358692 T6007 oasc.OverseerCollectionProcessor.createCollection Creating SolrCores for new collection solrj_collection, shardNames [shard1, shard2] , replicationFactor : 2
   [junit4]   2> 1358693 T6007 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard1_replica1 as part of slice shard1 of collection solrj_collection on 127.0.0.1:24477_
   [junit4]   2> 1358693 T6007 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard1_replica2 as part of slice shard1 of collection solrj_collection on 127.0.0.1:24486_
   [junit4]   2> 1358695 T6007 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard2_replica1 as part of slice shard2 of collection solrj_collection on 127.0.0.1:24483_
   [junit4]   2> 1358696 T6007 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard2_replica2 as part of slice shard2 of collection solrj_collection on 127.0.0.1:24473_
   [junit4]   2> 1358813 T5948 oasha.CoreAdminHandler.handleCreateAction core create command numShards=2&shard=shard1&version=2&action=CREATE&collection=solrj_collection&qt=/admin/cores&wt=javabin&collection.configName=conf1&name=solrj_collection_shard1_replica1
   [junit4]   2> 1358814 T5993 oasha.CoreAdminHandler.handleCreateAction core create command numShards=2&shard=shard1&version=2&action=CREATE&collection=solrj_collection&qt=/admin/cores&wt=javabin&collection.configName=conf1&name=solrj_collection_shard1_replica2
   [junit4]   2> 1358815 T5948 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica1 state=down collection=solrj_collection
   [junit4]   2> 1358816 T5993 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica2 state=down collection=solrj_collection
   [junit4]   2> 1358816 T5948 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1358816 T5993 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1358816 T5959 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1358818 T5938 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:24477",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "node_name":"127.0.0.1:24477_",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "core":"solrj_collection_shard1_replica1"}
   [junit4]   2> 1358820 T5938 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:24486",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "node_name":"127.0.0.1:24486_",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "core":"solrj_collection_shard1_replica2"}
   [junit4]   2> 1358821 T5959 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1358821 T5989 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1358821 T5959 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1358821 T6004 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1358821 T5958 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1358821 T5957 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1358821 T5974 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1358822 T5981 oasha.CoreAdminHandler.handleCreateAction core create command numShards=2&shard=shard2&version=2&action=CREATE&collection=solrj_collection&qt=/admin/cores&wt=javabin&collection.configName=conf1&name=solrj_collection_shard2_replica1
   [junit4]   2> 1358824 T5930 oasha.CoreAdminHandler.handleCreateAction core create command numShards=2&shard=shard2&version=2&action=CREATE&collection=solrj_collection&qt=/admin/cores&wt=javabin&collection.configName=conf1&name=solrj_collection_shard2_replica2
   [junit4]   2> 1358825 T5981 oasc.ZkController.publish publishing core=solrj_collection_shard2_replica1 state=down collection=solrj_collection
   [junit4]   2> 1358825 T5959 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1358825 T5981 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1358826 T5930 oasc.ZkController.publish publishing core=solrj_collection_shard2_replica2 state=down collection=solrj_collection
   [junit4]   2> 1358827 T5930 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1358827 T5938 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:24483",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "node_name":"127.0.0.1:24483_",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "core":"solrj_collection_shard2_replica1"}
   [junit4]   2> 1358828 T5938 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:24473",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "node_name":"127.0.0.1:24473_",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "core":"solrj_collection_shard2_replica2"}
   [junit4]   2> 1358829 T5959 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1358830 T5989 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1358830 T5959 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1358830 T6004 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1358830 T5974 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1358830 T5957 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1358830 T5958 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1359817 T5993 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard1_replica2
   [junit4]   2> 1359817 T5948 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard1_replica1
   [junit4]   2> 1359818 T5993 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 1359818 T5948 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 1359819 T5993 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1359819 T5993 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 1359819 T5948 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1359820 T5948 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 1359820 T5993 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1359820 T5993 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest-359299200057C864-001/tempDir-006/solrj_collection_shard1_replica2/'
   [junit4]   2> 1359820 T5948 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1359821 T5948 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest-359299200057C864-001/tempDir-003/solrj_collection_shard1_replica1/'
   [junit4]   2> 1359826 T5981 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard2_replica1
   [junit4]   2> 1359827 T5981 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 1359828 T5930 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard2_replica2
   [junit4]   2> 1359829 T5981 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1359829 T5930 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 1359829 T5981 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 1359830 T5930 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1359830 T5930 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 1359830 T5981 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1359832 T5981 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest-359299200057C864-001/tempDir-005/solrj_collection_shard2_replica1/'
   [junit4]   2> 1359832 T5930 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1359832 T5930 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest-359299200057C864-001/tempDir-002/solrj_collection_shard2_replica2/'
   [junit4]   2> 1359936 T5993 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 1359936 T5948 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 1359946 T5981 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 1359946 T5930 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 1359984 T5993 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1359985 T5993 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 1359993 T5993 oass.IndexSchema.readSchema [solrj_collection_shard1_replica2] Schema name=test
   [junit4]   2> 1359996 T5948 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1359997 T5948 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 1360009 T5948 oass.IndexSchema.readSchema [solrj_collection_shard1_replica1] Schema name=test
   [junit4]   2> 1360010 T5981 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1360010 T5930 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1360011 T5981 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 1360011 T5930 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 1360023 T5930 oass.IndexSchema.readSchema [solrj_collection_shard2_replica2] Schema name=test
   [junit4]   2> 1360023 T5981 oass.IndexSchema.readSchema [solrj_collection_shard2_replica1] Schema name=test
   [junit4]   2> 1360321 T5993 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1360333 T5993 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1360337 T5993 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1360355 T5993 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1360359 T5993 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1360362 T5993 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1360363 T5993 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1360364 T5993 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1360365 T5993 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1360365 T5993 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1360366 T5993 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1360366 T5993 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard1_replica2' using configuration from collection solrj_collection
   [junit4]   2> 1360366 T5993 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1360366 T5993 oasc.SolrCore.<init> [solrj_collection_shard1_replica2] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest-359299200057C864-001/tempDir-006/solrj_collection_shard1_replica2/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest-359299200057C864-001/tempDir-006/solrj_collection_shard1_replica2/data/
   [junit4]   2> 1360367 T5993 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3d634c52
   [junit4]   2> 1360368 T5993 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest-359299200057C864-001/tempDir-006/solrj_collection_shard1_replica2/data
   [junit4]   2> 1360368 T5993 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest-359299200057C864-001/tempDir-006/solrj_collection_shard1_replica2/data/index/
   [junit4]   2> 1360368 T5993 oasc.SolrCore.initIndex WARN [solrj_collection_shard1_replica2] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest-359299200057C864-001/tempDir-006/solrj_collection_shard1_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1360368 T5993 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest-359299200057C864-001/tempDir-006/solrj_collection_shard1_replica2/data/index
   [junit4]   2> 1360369 T5993 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@3d73f907
   [junit4]   2> 1360369 T5993 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RateLimitedDirectoryWrapper(RAMDirectory(id=lucene-19f2c17e))),segFN=segments_1,generation=1}
   [junit4]   2> 1360370 T5993 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1360373 T5993 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1360373 T5993 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1360373 T5993 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1360374 T5993 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1360374 T5993 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1360374 T5993 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1360374 T5948 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1360375 T5993 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1360375 T5993 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1360375 T5993 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1360376 T5993 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1360376 T5993 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1360377 T5993 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1360377 T5993 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1360378 T5993 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1360378 T5993 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1360378 T5993 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1360378 T5993 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1360379 T5993 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1360379 T5993 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1360379 T5993 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 1360382 T5948 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1360386 T5948 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1360395 T5993 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1360397 T5948 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1360397 T5993 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1360399 T5993 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1360401 T5993 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1360402 T5948 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1360406 T5993 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1360406 T5993 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1360406 T5948 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1360407 T5993 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=16, maxMergeAtOnceExplicit=25, maxMergedSegmentMB=0.4345703125, floorSegmentMB=0.55859375, forceMergeDeletesPctAllowed=25.564099666783477, segmentsPerTier=40.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1360407 T5948 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1360407 T5993 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RateLimitedDirectoryWrapper(RAMDirectory(id=lucene-19f2c17e))),segFN=segments_1,generation=1}
   [junit4]   2> 1360408 T5948 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1360408 T5993 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1360408 T5948 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1360409 T5993 oass.SolrIndexSearcher.<init> Opening Searcher@2d3c36b1[solrj_collection_shard1_replica2] main
   [junit4]   2> 1360409 T5993 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 1360410 T5948 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1360410 T5948 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1360410 T5993 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1360410 T5948 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard1_replica1' using configuration from collection solrj_collection
   [junit4]   2> 1360410 T5993 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1360411 T5948 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1360411 T5993 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1360412 T5948 oasc.SolrCore.<init> [solrj_collection_shard1_replica1] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest-359299200057C864-001/tempDir-003/solrj_collection_shard1_replica1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest-359299200057C864-001/tempDir-003/solrj_collection_shard1_replica1/data/
   [junit4]   2> 1360412 T5993 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 1360412 T5948 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3d634c52
   [junit4]   2> 1360412 T5993 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1360413 T5993 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1360413 T5948 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest-359299200057C864-001/tempDir-003/solrj_collection_shard1_replica1/data
   [junit4]   2> 1360413 T5993 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1360414 T5948 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest-359299200057C864-001/tempDir-003/solrj_collection_shard1_replica1/data/index/
   [junit4]   2> 1360414 T5993 oascc.SolrZkClient.makePath makePath: /configs/conf1/_rest_managed.json
   [junit4]   2> 1360414 T5948 oasc.SolrCore.initIndex WARN [solrj_collection_shard1_replica1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest-359299200057C864-001/tempDir-003/solrj_collection_shard1_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1360415 T5948 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest-359299200057C864-001/tempDir-003/solrj_collection_shard1_replica1/data/index
   [junit4]   2> 1360416 T5948 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@b4e29f
   [junit4]   2> 1360416 T5993 oasr.ManagedResourceStorage$ZooKeeperStorageIO$1.close Wrote 38 bytes to new znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1360416 T5993 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1360417 T5993 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 1360417 T5948 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RateLimitedDirectoryWrapper(RAMDirectory(id=lucene-17a6dbd7))),segFN=segments_1,generation=1}
   [junit4]   2> 1360417 T5948 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1360419 T6016 oasc.SolrCore.registerSearcher [solrj_collection_shard1_replica2] Registered new searcher Searcher@2d3c36b1[solrj_collection_shard1_replica2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1360421 T5993 oasc.CoreContainer.registerCore registering core: solrj_collection_shard1_replica2
   [junit4]   2> 1360422 T5993 oasc.ZkController.register Register replica - core:solrj_collection_shard1_replica2 address:https://127.0.0.1:24486 collection:solrj_collection shard:shard1
   [junit4]   2> 1360422 T5948 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1360422 T5993 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection/leader_elect/shard1/election
   [junit4]   2> 1360423 T5948 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1360423 T5948 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1360424 T5948 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1360424 T5981 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1360424 T5948 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1360425 T5948 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1360425 T5948 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1360426 T5948 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1360426 T5948 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1360427 T5993 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 1360427 T5948 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1360427 T5948 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1360428 T5948 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1360428 T5959 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1360428 T5993 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Waiting until we see more replicas up for shard shard1: total=2 found=1 timeoutin=179999ms
   [junit4]   2> 1360428 T5948 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1360428 T5981 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1360429 T5948 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1360430 T5948 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1360430 T5948 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1360431 T5948 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1360431 T5948 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1360431 T5981 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1360432 T5948 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1360432 T5948 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 1360432 T5989 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1360432 T6004 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1360432 T5958 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1360432 T5959 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1360432 T5974 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1360432 T5957 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1360435 T5948 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1360437 T5948 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1360439 T5948 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1360440 T5981 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1360440 T5930 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1360441 T5948 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1360443 T5981 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1360447 T5981 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1360447 T5930 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1360448 T5948 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1360448 T5981 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1360448 T5981 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1360448 T5948 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1360448 T5981 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1360449 T5948 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=16, maxMergeAtOnceExplicit=25, maxMergedSegmentMB=0.4345703125, floorSegmentMB=0.55859375, forceMergeDeletesPctAllowed=25.564099666783477, segmentsPerTier=40.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1360449 T5981 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1360450 T5981 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1360450 T5930 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1360450 T5948 oasc.SolrDeletionPolicy.onInit SolrDeleti

[...truncated too long message...]

cFreqInterval), 2grand0_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), 6parent_f2_s=PostingsFormat(name=Memory doPackFST= false), 28parent_f2_s=PostingsFormat(name=Memory doPackFST= false), 30parent_f1_s=PostingsFormat(name=Memory doPackFST= false), 21parent_f1_s=FST50, 8parent_f2_s=FST50, 1parent_f1_s=PostingsFormat(name=Memory doPackFST= false), 11grand8_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), 1child0_s=FST50, 10grand1_s=PostingsFormat(name=Direct), 6grand2_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), 3grand0_s=PostingsFormat(name=Memory doPackFST= false), 7grand2_s=FST50, 25parent_f1_s=FST50, 40parent_f2_s=FST50, 19parent_f2_s=FST50, 11grand7_s=PostingsFormat(name=Memory doPackFST= false), 0parent_f2_s=FST50, 26parent_f2_s=FST50, 5parent_f2_s=PostingsFormat(name=Direct), 8grand2_s=PostingsFormat(name=Direct), 38parent_f2_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), 6child1_s=PostingsFormat(name=Memory doPackFST= false), 5grand1_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), 11grand0_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), 3parent_f1_s=FST50, 27parent_f2_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), 7parent_f2_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), type_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), 3grand2_s=FST50, 27parent_f1_s=PostingsFormat(name=Memory doPackFST= false), 6parent_f1_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), 33parent_f1_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), 2grand1_s=PostingsFormat(name=Memory doPackFST= false), 10parent_f1_s=FST50, 2parent_f2_s=PostingsFormat(name=Memory doPackFST= false), 3child1_s=PostingsFormat(name=Direct), 13parent_f1_s=PostingsFormat(name=Direct), 9grand6_s=PostingsFormat(name=Memory doPackFST= false), 18parent_f1_s=FST50, 1parent_f2_s=PostingsFormat(name=Direct), id=PostingsFormat(name=Memory doPackFST= false), 10grand0_s=PostingsFormat(name=Memory doPackFST= false), 30parent_f2_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), 9grand5_s=PostingsFormat(name=Direct), 7parent_f1_s=FST50, 10grand7_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), 3child0_s=PostingsFormat(name=Memory doPackFST= false), 10grand5_s=PostingsFormat(name=Direct), 2parent_f1_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), 8grand5_s=FST50, 42parent_f2_s=PostingsFormat(name=Memory doPackFST= false), 3parent_f2_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), 10grand6_s=FST50, 29parent_f2_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), 23parent_f2_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), 39parent_f2_s=PostingsFormat(name=Memory doPackFST= false), 11grand9_s=FST50, 22parent_f1_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), 12parent_f2_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), 25parent_f2_s=PostingsFormat(name=Direct), 4grand1_s=FST50, 2child0_s=PostingsFormat(name=Direct), 5child2_s=PostingsFormat(name=Memory doPackFST= false), 4grand3_s=PostingsFormat(name=Memory doPackFST= false), 26parent_f1_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), 5grand4_s=FST50, 15parent_f2_s=FST50, 9grand3_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), 35parent_f2_s=PostingsFormat(name=Memory doPackFST= false), 37parent_f2_s=FST50, 7grand6_s=FST50, 9parent_f2_s=PostingsFormat(name=Direct), 6child4_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), 4child1_s=FST50, 11parent_f2_s=FST50, 39parent_f1_s=PostingsFormat(name=Direct), 4grand0_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), 1grand0_s=FST50, 18parent_f2_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), 17parent_f1_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), 15parent_f1_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), 4parent_f2_s=FST50, 5grand2_s=PostingsFormat(name=Memory doPackFST= false), text_t=PostingsFormat(name=LuceneVarGapDocFreqInterval), 3grand1_s=PostingsFormat(name=Direct), 5grand3_s=PostingsFormat(name=Direct), 10grand4_s=PostingsFormat(name=Memory doPackFST= false), 23parent_f1_s=PostingsFormat(name=Memory doPackFST= false), 31parent_f2_s=PostingsFormat(name=Memory doPackFST= false), 6grand3_s=FST50, 34parent_f2_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), 6child5_s=PostingsFormat(name=Memory doPackFST= false), 6child0_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), 10grand2_s=FST50, 32parent_f1_s=FST50, 16parent_f2_s=PostingsFormat(name=Direct), 6grand4_s=PostingsFormat(name=Direct), 7grand5_s=PostingsFormat(name=Direct), 9parent_f1_s=PostingsFormat(name=Memory doPackFST= false), 11grand5_s=FST50, a_t=PostingsFormat(name=Direct), 22parent_f2_s=FST50, 28parent_f1_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), 11grand1_s=FST50, 4grand2_s=PostingsFormat(name=Direct), 33parent_f2_s=FST50, 9grand0_s=FST50, 20parent_f1_s=PostingsFormat(name=Direct), 34parent_f1_s=PostingsFormat(name=Memory doPackFST= false), 13parent_f2_s=PostingsFormat(name=Memory doPackFST= false), 7grand0_s=PostingsFormat(name=Memory doPackFST= false), 5child4_s=FST50, 11grand2_s=PostingsFormat(name=Direct), 10parent_f2_s=PostingsFormat(name=Direct), 35parent_f1_s=PostingsFormat(name=Direct), 8grand1_s=FST50, 29parent_f1_s=FST50, 5child1_s=PostingsFormat(name=Direct), 24parent_f1_s=PostingsFormat(name=Direct), 37parent_f1_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), 38parent_f1_s=PostingsFormat(name=Memory doPackFST= false), 17parent_f2_s=PostingsFormat(name=Memory doPackFST= false), 9grand7_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), 16parent_f1_s=PostingsFormat(name=Memory doPackFST= false), 21parent_f2_s=PostingsFormat(name=Direct), 5child3_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), 5child0_s=FST50, 8grand0_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), 9grand4_s=FST50, 24parent_f2_s=PostingsFormat(name=Memory doPackFST= false), _root_=PostingsFormat(name=LuceneVarGapDocFreqInterval), 4child3_s=PostingsFormat(name=Memory doPackFST= false), 12parent_f1_s=PostingsFormat(name=Memory doPackFST= false), 11grand10_s=PostingsFormat(name=Memory doPackFST= false), 4child2_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), 11grand6_s=PostingsFormat(name=Direct), 41parent_f2_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), 0parent_f1_s=PostingsFormat(name=Direct), 42parent_f1_s=PostingsFormat(name=Direct), 19parent_f1_s=PostingsFormat(name=Direct), 10grand3_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), 20parent_f2_s=PostingsFormat(name=Memory doPackFST= false), 7grand1_s=PostingsFormat(name=Direct), 36parent_f1_s=FST50, 7grand3_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), 10grand8_s=PostingsFormat(name=Memory doPackFST= false), 36parent_f2_s=PostingsFormat(name=Direct), 4parent_f1_s=PostingsFormat(name=Direct), 4child0_s=PostingsFormat(name=Direct), 8parent_f1_s=PostingsFormat(name=Direct), 11grand3_s=PostingsFormat(name=Memory doPackFST= false), 6grand1_s=PostingsFormat(name=Memory doPackFST= false), 9grand2_s=PostingsFormat(name=Memory doPackFST= false), 11grand4_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), 11parent_f1_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), 6grand0_s=PostingsFormat(name=Direct), 2child1_s=PostingsFormat(name=Memory doPackFST= false), 3child2_s=FST50, 9grand8_s=FST50, 7grand4_s=PostingsFormat(name=Memory doPackFST= false), 6child2_s=PostingsFormat(name=Direct), 14parent_f1_s=FST50, 10grand9_s=PostingsFormat(name=Direct)}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=false,coord=crazy): {}, locale=en_US, timezone=America/Tegucigalpa
   [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_65 (64-bit)/cpus=16,threads=1,free=241167952,total=457179136
   [junit4]   2> NOTE: All tests run in this JVM: [SuggesterFSTTest, TestDocumentBuilder, OpenCloseCoreStressTest, TestReload, TestCustomSort, TestCloudSchemaless, HdfsSyncSliceTest, TestRecoveryHdfs, TestCodecSupport, TriLevelCompositeIdRoutingTest, ZkControllerTest, TestIntervalFaceting, LukeRequestHandlerTest, AnalyticsQueryTest, TestConfig, SortByFunctionTest, TestReRankQParserPlugin, ShowFileRequestHandlerTest, BlockDirectoryTest, TestOrdValues, TestDistribDocBasedVersion, AutoCommitTest, TestBinaryField, ClusterStateTest, TestBulkSchemaConcurrent, TestBulkSchemaAPI, JSONWriterTest, HdfsDirectoryTest, FullSolrCloudDistribCmdsTest]
   [junit4] Completed on J1 in 161.52s, 1 test, 1 error <<< FAILURES!

[...truncated 886 lines...]
BUILD FAILED
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/build.xml:532: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/build.xml:473: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/build.xml:61: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/extra-targets.xml:39: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build.xml:189: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/common-build.xml:496: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/lucene/common-build.xml:1358: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/lucene/common-build.xml:965: There were test failures: 438 suites, 1797 tests, 2 errors, 54 ignored (9 assumptions)

Total time: 223 minutes 16 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-NightlyTests-trunk #664
Archived 5 artifacts
Archive block size is 32768
Received 0 blocks and 703119135 bytes
Compression is 0.0%
Took 1 min 57 sec
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 674 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/674/

2 tests failed.
REGRESSION:  org.apache.solr.cloud.CollectionsAPIAsyncDistributedZkTest.testDistribSearch

Error Message:
Shard split did not complete. Last recorded state: running expected:<[completed]> but was:<[running]>

Stack Trace:
org.junit.ComparisonFailure: Shard split did not complete. Last recorded state: running expected:<[completed]> but was:<[running]>
	at __randomizedtesting.SeedInfo.seed([80B8668EE9622AC1:15EE8969E3D4AFD]:0)
	at org.junit.Assert.assertEquals(Assert.java:125)
	at org.apache.solr.cloud.CollectionsAPIAsyncDistributedZkTest.testSolrJAPICalls(CollectionsAPIAsyncDistributedZkTest.java:114)
	at org.apache.solr.cloud.CollectionsAPIAsyncDistributedZkTest.doTest(CollectionsAPIAsyncDistributedZkTest.java:64)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


REGRESSION:  org.apache.solr.search.mlt.SimpleMLTQParserTest.doTest

Error Message:
Exception during query

Stack Trace:
java.lang.RuntimeException: Exception during query
	at __randomizedtesting.SeedInfo.seed([80B8668EE9622AC1:27FCDE2A84D93978]:0)
	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:723)
	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:690)
	at org.apache.solr.search.mlt.SimpleMLTQParserTest.doTest(SimpleMLTQParserTest.java:71)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: REQUEST FAILED: xpath=//result/doc[2]/int[@name='id'][.='13']
	xml response was: <?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">0</int><int name="QTime">35</int></lst><result name="response" numFound="14" start="0"><doc><int name="id">17</int><arr name="range_facet_l"><long>17</long></arr><arr name="lowerfilt"><str>The quote red fox jumped moon over the lazy brown dogs moon. Of course moon. Foxes and moon come back to the foxes and moon</str></arr><arr name="lowerfilt1and2"><str>The quote red fox jumped moon over the lazy brown dogs moon. Of course moon. Foxes and moon come back to the foxes and moon</str></arr><long name="_version_">1483718590381162496</long><arr name="multiDefault"><str>muLti-Default</str></arr><int name="intDefault">42</int><date name="timestamp">2014-11-03T03:11:52.42Z</date></doc><doc><int name="id">18</int><arr name="range_facet_l"><long>18</long></arr><arr name="lowerfilt"><str>The quote red fox jumped over the lazy brown dogs.</str></arr><arr name="lowerfilt1and2"><str>The quote red fox jumped over the lazy brown dogs.</str></arr><long name="_version_">1483718590383259648</long><arr name="multiDefault"><str>muLti-Default</str></arr><int name="intDefault">42</int><date name="timestamp">2014-11-03T03:11:52.423Z</date></doc><doc><int name="id">20</int><arr name="range_facet_l"><long>20</long></arr><arr name="lowerfilt"><str>The quote red fox jumped over the lazy brown dogs.</str></arr><arr name="lowerfilt1and2"><str>The quote red fox jumped over the lazy brown dogs.</str></arr><long name="_version_">1483718590388502528</long><arr name="multiDefault"><str>muLti-Default</str></arr><int name="intDefault">42</int><date name="timestamp">2014-11-03T03:11:52.428Z</date></doc><doc><int name="id">22</int><arr name="range_facet_l"><long>22</long></arr><arr name="lowerfilt"><str>The quote red fox jumped over the lazy brown dogs.</str></arr><arr name="lowerfilt1and2"><str>The quote red fox jumped over the lazy brown dogs.</str></arr><long name="_version_">1483718590392696832</long><arr name="multiDefault"><str>muLti-Default</str></arr><int name="intDefault">42</int><date name="timestamp">2014-11-03T03:11:52.432Z</date></doc><doc><int name="id">23</int><arr name="range_facet_l"><long>23</long></arr><arr name="lowerfilt"><str>The quote red fox jumped over the lazy brown dogs.</str></arr><arr name="lowerfilt1and2"><str>The quote red fox jumped over the lazy brown dogs.</str></arr><long name="_version_">1483718590394793984</long><arr name="multiDefault"><str>muLti-Default</str></arr><int name="intDefault">42</int><date name="timestamp">2014-11-03T03:11:52.434Z</date></doc><doc><int name="id">13</int><arr name="range_facet_l"><long>13</long></arr><arr name="lowerfilt"><str>The quote red fox jumped over the lazy brown dogs.</str></arr><arr name="lowerfilt1and2"><str>The quote red fox jumped over the lazy brown dogs.</str></arr><long name="_version_">1483718590364385280</long><arr name="multiDefault"><str>muLti-Default</str></arr><int name="intDefault">42</int><date name="timestamp">2014-11-03T03:11:52.405Z</date></doc><doc><int name="id">14</int><arr name="range_facet_l"><long>14</long></arr><arr name="lowerfilt"><str>The quote red fox jumped over the lazy brown dogs.</str></arr><arr name="lowerfilt1and2"><str>The quote red fox jumped over the lazy brown dogs.</str></arr><long name="_version_">1483718590366482432</long><arr name="multiDefault"><str>muLti-Default</str></arr><int name="intDefault">42</int><date name="timestamp">2014-11-03T03:11:52.407Z</date></doc><doc><int name="id">9</int><arr name="range_facet_l"><long>9</long></arr><arr name="lowerfilt"><str>The quick red fox jumped over the lazy big and large brown dogs.</str></arr><arr name="lowerfilt1and2"><str>The quick red fox jumped over the lazy big and large brown dogs.</str></arr><long name="_version_">1483718590349705216</long><arr name="multiDefault"><str>muLti-Default</str></arr><int name="intDefault">42</int><date name="timestamp">2014-11-03T03:11:52.391Z</date></doc><doc><int name="id">7</int><arr name="range_facet_l"><long>7</long></arr><arr name="lowerfilt"><str>mclaren moon or the moon and moon moon shine and the moon but moon was good foxes too</str></arr><arr name="lowerfilt1and2"><str>mclaren moon or the moon and moon moon shine and the moon but moon was good foxes too</str></arr><long name="_version_">1483718590345510912</long><arr name="multiDefault"><str>muLti-Default</str></arr><int name="intDefault">42</int><date name="timestamp">2014-11-03T03:11:52.387Z</date></doc><doc><int name="id">15</int><arr name="range_facet_l"><long>15</long></arr><arr name="lowerfilt"><str>The fat red fox jumped over the lazy brown dogs.</str></arr><arr name="lowerfilt1and2"><str>The fat red fox jumped over the lazy brown dogs.</str></arr><long name="_version_">1483718590375919616</long><arr name="multiDefault"><str>muLti-Default</str></arr><int name="intDefault">42</int><date name="timestamp">2014-11-03T03:11:52.415Z</date></doc></result>
</response>

	request was:q=%7B%21mlt+qf%3Dlowerfilt%7D17
	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:716)
	... 42 more




Build Log:
[...truncated 11925 lines...]
   [junit4] Suite: org.apache.solr.search.mlt.SimpleMLTQParserTest
   [junit4]   2> Creating dataDir: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.search.mlt.SimpleMLTQParserTest-80B8668EE9622AC1-001/init-core-data-001
   [junit4]   2> 2168892 T3971 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (true)
   [junit4]   2> 2168894 T3971 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> 2168894 T3971 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/'
   [junit4]   2> 2168896 T3971 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/lib/.svn/' to classloader
   [junit4]   2> 2168896 T3971 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 2168897 T3971 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 2169062 T3971 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 2169129 T3971 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2169131 T3971 oass.IndexSchema.readSchema Reading Solr Schema from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/schema.xml
   [junit4]   2> 2169141 T3971 oass.IndexSchema.readSchema [null] Schema name=test
   [junit4]   2> 2169582 T3971 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 2169594 T3971 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2169597 T3971 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2169607 T3971 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2169612 T3971 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2169616 T3971 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2169617 T3971 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2169618 T3971 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2169618 T3971 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2169619 T3971 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2169619 T3971 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2169619 T3971 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2169620 T3971 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr
   [junit4]   2> 2169620 T3971 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/'
   [junit4]   2> 2169713 T3971 oasc.CoreContainer.<init> New CoreContainer 1504592265
   [junit4]   2> 2169713 T3971 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/]
   [junit4]   2> 2169714 T3971 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
   [junit4]   2> 2169715 T3971 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 2169715 T3971 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
   [junit4]   2> 2169715 T3971 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2169715 T3971 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 2169716 T3971 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2169716 T3971 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2169716 T3971 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2169717 T3971 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2169717 T3971 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2169717 T3971 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=30000&connTimeout=30000&retry=false
   [junit4]   2> 2169718 T3971 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2169718 T3971 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2169718 T3971 oasc.CoreContainer.load Host Name: 
   [junit4]   2> 2169723 T3972 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/'
   [junit4]   2> 2169725 T3972 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/lib/.svn/' to classloader
   [junit4]   2> 2169725 T3972 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 2169725 T3972 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 2169801 T3972 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 2169894 T3972 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2169894 T3972 oass.IndexSchema.readSchema Reading Solr Schema from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/schema.xml
   [junit4]   2> 2169906 T3972 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 2170452 T3972 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 2170463 T3972 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2170466 T3972 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2170475 T3972 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2170479 T3972 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2170483 T3972 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2170484 T3972 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2170484 T3972 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2170484 T3972 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2170485 T3972 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2170485 T3972 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2170485 T3972 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from instancedir /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/
   [junit4]   2> 2170486 T3972 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 2170486 T3972 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.search.mlt.SimpleMLTQParserTest-80B8668EE9622AC1-001/init-core-data-001/
   [junit4]   2> 2170486 T3972 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2ca6e9c5
   [junit4]   2> 2170488 T3972 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.search.mlt.SimpleMLTQParserTest-80B8668EE9622AC1-001/init-core-data-001
   [junit4]   2> 2170489 T3972 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.search.mlt.SimpleMLTQParserTest-80B8668EE9622AC1-001/init-core-data-001/index/
   [junit4]   2> 2170489 T3972 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.search.mlt.SimpleMLTQParserTest-80B8668EE9622AC1-001/init-core-data-001/index' doesn't exist. Creating new index...
   [junit4]   2> 2170490 T3972 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.search.mlt.SimpleMLTQParserTest-80B8668EE9622AC1-001/init-core-data-001/index
   [junit4]   2> 2170490 T3972 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=26, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 2170491 T3972 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(NRTCachingDirectory(RAMDirectory(id=lucene-798c44a0); maxCacheMB=0.9677734375 maxMergeSizeMB=0.14453125))),segFN=segments_1,generation=1}
   [junit4]   2> 2170491 T3972 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2170492 T3972 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
   [junit4]   2> 2170497 T3972 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 2170497 T3972 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 2170497 T3972 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 2170498 T3972 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 2170498 T3972 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2170498 T3972 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2170498 T3972 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 2170499 T3972 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 2170499 T3972 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 2170499 T3972 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2170500 T3972 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2170500 T3972 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2170500 T3972 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2170501 T3972 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2170501 T3972 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2170502 T3972 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 2170502 T3972 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 2170503 T3972 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 2170503 T3972 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 2170504 T3972 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
   [junit4]   2> 2170504 T3972 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
   [junit4]   2> 2170505 T3972 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 2170505 T3972 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 2170506 T3972 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 2170506 T3972 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
   [junit4]   2> 2170506 T3972 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
   [junit4]   2> 2170506 T3972 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
   [junit4]   2> 2170507 T3972 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 2170507 T3972 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2170508 T3972 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2170508 T3972 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2170509 T3972 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2170509 T3972 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2170510 T3972 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2170510 T3972 oasc.RequestHandlers.initHandlersFromConfig created mltrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2170511 T3972 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2170511 T3972 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
   [junit4]   2> 2170512 T3972 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
   [junit4]   2> 2170530 T3972 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2170532 T3972 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2170534 T3972 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2170536 T3972 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2170538 T3972 oasc.SolrCore.initDeprecatedSupport WARN solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
   [junit4]   2> 2170540 T3972 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 2170541 T3972 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 2170541 T3972 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@5338fc42
   [junit4]   2> 2170542 T3972 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(NRTCachingDirectory(RAMDirectory(id=lucene-798c44a0); maxCacheMB=0.9677734375 maxMergeSizeMB=0.14453125))),segFN=segments_1,generation=1}
   [junit4]   2> 2170542 T3972 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2170543 T3972 oass.SolrIndexSearcher.<init> Opening Searcher@4802039f[collection1] main
   [junit4]   2> 2170543 T3972 oasr.RestManager.init Initializing RestManager with initArgs: {storageIO=org.apache.solr.rest.ManagedResourceStorage$InMemoryStorageIO}
   [junit4]   2> 2170543 T3972 oasr.ManagedResourceStorage.load Reading _rest_managed.json using InMemoryStorage
   [junit4]   2> 2170544 T3972 oasr.ManagedResource.reloadFromStorage WARN No stored data found for /rest/managed
   [junit4]   2> 2170544 T3972 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using InMemoryStorage
   [junit4]   2> 2170544 T3972 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 2170544 T3972 oashc.SpellCheckComponent.inform Initializing spell checkers
   [junit4]   2> 2170553 T3972 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 2170709 T3973 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4802039f[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2170718 T3972 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 2170731 T3971 oas.SolrTestCaseJ4.initCore ####initCore end
   [junit4]   2> 2170736 T3971 oas.SolrTestCaseJ4.setUp ###Starting doTest
   [junit4]   2> ASYNC  NEW_CORE C1732 name=collection1 org.apache.solr.core.SolrCore@2f77b967
   [junit4]   2> 2170739 T3971 C1732 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {add=[1 (1483718590329782272)]} 0 2
   [junit4]   2> 2170744 T3971 C1732 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {add=[2 (1483718590336073728)]} 0 1
   [junit4]   2> 2170745 T3971 C1732 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {add=[3 (1483718590338170880)]} 0 0
   [junit4]   2> 2170747 T3971 C1732 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {add=[4 (1483718590340268032)]} 0 0
   [junit4]   2> 2170749 T3971 C1732 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {add=[5 (1483718590341316608)]} 0 1
   [junit4]   2> 2170750 T3971 C1732 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {add=[6 (1483718590343413760)]} 0 0
   [junit4]   2> 2170753 T3971 C1732 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {add=[7 (1483718590345510912)]} 0 1
   [junit4]   2> 2170754 T3971 C1732 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {add=[8 (1483718590347608064)]} 0 0
   [junit4]   2> 2170757 T3971 C1732 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {add=[9 (1483718590349705216)]} 0 1
   [junit4]   2> 2170767 T3971 C1732 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {add=[10 (1483718590361239552)]} 0 1
   [junit4]   2> 2170768 T3971 C1732 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {add=[12 (1483718590362288128)]} 0 0
   [junit4]   2> 2170770 T3971 C1732 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {add=[13 (1483718590364385280)]} 0 0
   [junit4]   2> 2170779 T3971 C1732 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {add=[14 (1483718590366482432)]} 0 7
   [junit4]   2> 2170782 T3971 C1732 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {add=[15 (1483718590375919616)]} 0 2
   [junit4]   2> 2170784 T3971 C1732 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {add=[16 (1483718590378016768)]} 0 1
   [junit4]   2> 2170787 T3971 C1732 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {add=[17 (1483718590381162496)]} 0 2
   [junit4]   2> 2170789 T3971 C1732 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {add=[18 (1483718590383259648)]} 0 1
   [junit4]   2> 2170791 T3971 C1732 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {add=[19 (1483718590386405376)]} 0 1
   [junit4]   2> 2170794 T3971 C1732 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {add=[20 (1483718590388502528)]} 0 1
   [junit4]   2> 2170796 T3971 C1732 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {add=[21 (1483718590390599680)]} 0 1
   [junit4]   2> 2170798 T3971 C1732 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {add=[22 (1483718590392696832)]} 0 1
   [junit4]   2> 2170800 T3971 C1732 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {add=[23 (1483718590394793984)]} 0 1
   [junit4]   2> 2170802 T3971 C1732 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {add=[24 (1483718590396891136)]} 0 1
   [junit4]   2> 2170804 T3971 C1732 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {add=[25 (1483718590398988288)]} 0 1
   [junit4]   2> 2170805 T3971 C1732 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 2170838 T3971 C1732 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(NRTCachingDirectory(RAMDirectory(id=lucene-798c44a0); maxCacheMB=0.9677734375 maxMergeSizeMB=0.14453125))),segFN=segments_1,generation=1}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(NRTCachingDirectory(RAMDirectory(id=lucene-798c44a0); maxCacheMB=0.9677734375 maxMergeSizeMB=0.14453125))),segFN=segments_2,generation=2}
   [junit4]   2> 2170839 T3971 C1732 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
   [junit4]   2> 2170842 T3971 C1732 oass.SolrIndexSearcher.<init> Opening Searcher@6f93fae6[collection1] main
   [junit4]   2> 2170843 T3971 C1732 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 2170845 T3973 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6f93fae6[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_2(6.0.0):C24)))}
   [junit4]   2> 2170846 T3971 C1732 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {commit=} 0 41
   [junit4]   2> 2170882 T3971 C1732 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=%7B%21mlt+qf%3Dlowerfilt%7D17} hits=14 status=0 QTime=35 
   [junit4]   2> 2170889 T3971 oas.SolrTestCaseJ4.assertQ ERROR REQUEST FAILED: xpath=//result/doc[2]/int[@name='id'][.='13']
   [junit4]   2> 		xml response was: <?xml version="1.0" encoding="UTF-8"?>
   [junit4]   2> 	<response>
   [junit4]   2> 	<lst name="responseHeader"><int name="status">0</int><int name="QTime">35</int></lst><result name="response" numFound="14" start="0"><doc><int name="id">17</int><arr name="range_facet_l"><long>17</long></arr><arr name="lowerfilt"><str>The quote red fox jumped moon over the lazy brown dogs moon. Of course moon. Foxes and moon come back to the foxes and moon</str></arr><arr name="lowerfilt1and2"><str>The quote red fox jumped moon over the lazy brown dogs moon. Of course moon. Foxes and moon come back to the foxes and moon</str></arr><long name="_version_">1483718590381162496</long><arr name="multiDefault"><str>muLti-Default</str></arr><int name="intDefault">42</int><date name="timestamp">2014-11-03T03:11:52.42Z</date></doc><doc><int name="id">18</int><arr name="range_facet_l"><long>18</long></arr><arr name="lowerfilt"><str>The quote red fox jumped over the lazy brown dogs.</str></arr><arr name="lowerfilt1and2"><str>The quote red fox jumped over the lazy brown dogs.</str></arr><long name="_version_">1483718590383259648</long><arr name="multiDefault"><str>muLti-Default</str></arr><int name="intDefault">42</int><date name="timestamp">2014-11-03T03:11:52.423Z</date></doc><doc><int name="id">20</int><arr name="range_facet_l"><long>20</long></arr><arr name="lowerfilt"><str>The quote red fox jumped over the lazy brown dogs.</str></arr><arr name="lowerfilt1and2"><str>The quote red fox jumped over the lazy brown dogs.</str></arr><long name="_version_">1483718590388502528</long><arr name="multiDefault"><str>muLti-Default</str></arr><int name="intDefault">42</int><date name="timestamp">2014-11-03T03:11:52.428Z</date></doc><doc><int name="id">22</int><arr name="range_facet_l"><long>22</long></arr><arr name="lowerfilt"><str>The quote red fox jumped over the lazy brown dogs.</str></arr><arr name="lowerfilt1and2"><str>The quote red fox jumped over the lazy brown dogs.</str></arr><long name="_version_">1483718590392696832</long><arr name="multiDefault"><str>muLti-Default</str></arr><int name="intDefault">42</int><date name="timestamp">2014-11-03T03:11:52.432Z</date></doc><doc><int name="id">23</int><arr name="range_facet_l"><long>23</long></arr><arr name="lowerfilt"><str>The quote red fox jumped over the lazy brown dogs.</str></arr><arr name="lowerfilt1and2"><str>The quote red fox jumped over the lazy brown dogs.</str></arr><long name="_version_">1483718590394793984</long><arr name="multiDefault"><str>muLti-Default</str></arr><int name="intDefault">42</int><date name="timestamp">2014-11-03T03:11:52.434Z</date></doc><doc><int name="id">13</int><arr name="range_facet_l"><long>13</long></arr><arr name="lowerfilt"><str>The quote red fox jumped over the lazy brown dogs.</str></arr><arr name="lowerfilt1and2"><str>The quote red fox jumped over the lazy brown dogs.</str></arr><long name="_version_">1483718590364385280</long><arr name="multiDefault"><str>muLti-Default</str></arr><int name="intDefault">42</int><date name="timestamp">2014-11-03T03:11:52.405Z</date></doc><doc><int name="id">14</int><arr name="range_facet_l"><long>14</long></arr><arr name="lowerfilt"><str>The quote red fox jumped over the lazy brown dogs.</str></arr><arr name="lowerfilt1and2"><str>The quote red fox jumped over the lazy brown dogs.</str></arr><long name="_version_">1483718590366482432</long><arr name="multiDefault"><str>muLti-Default</str></arr><int name="intDefault">42</int><date name="timestamp">2014-11-03T03:11:52.407Z</date></doc><doc><int name="id">9</int><arr name="range_facet_l"><long>9</long></arr><arr name="lowerfilt"><str>The quick red fox jumped over the lazy big and large brown dogs.</str></arr><arr name="lowerfilt1and2"><str>The quick red fox jumped over the lazy big and large brown dogs.</str></arr><long name="_version_">1483718590349705216</long><arr name="multiDefault"><str>muLti-Default</str></arr><int name="intDefault">42</int><date name="timestamp">2014-11-03T03:11:52.391Z</date></doc><doc><int name="id">7</int><arr name="range_facet_l"><long>7</long></arr><arr name="lowerfilt"><str>mclaren moon or the moon and moon moon shine and the moon but moon was good foxes too</str></arr><arr name="lowerfilt1and2"><str>mclaren moon or the moon and moon moon shine and the moon but moon was good foxes too</str></arr><long name="_version_">1483718590345510912</long><arr name="multiDefault"><str>muLti-Default</str></arr><int name="intDefault">42</int><date name="timestamp">2014-11-03T03:11:52.387Z</date></doc><doc><int name="id">15</int><arr name="range_facet_l"><long>15</long></arr><arr name="lowerfilt"><str>The fat red fox jumped over the lazy brown dogs.</str></arr><arr name="lowerfilt1and2"><str>The fat red fox jumped over the lazy brown dogs.</str></arr><long name="_version_">1483718590375919616</long><arr name="multiDefault"><str>muLti-Default</str></arr><int name="intDefault">42</int><date name="timestamp">2014-11-03T03:11:52.415Z</date></doc></result>
   [junit4]   2> 	</response>
   [junit4]   2> 	
   [junit4]   2> 		request was:q=%7B%21mlt+qf%3Dlowerfilt%7D17
   [junit4]   2> 2170892 T3971 oasc.SolrException.log ERROR REQUEST FAILED: q=%7B%21mlt+qf%3Dlowerfilt%7D17:java.lang.RuntimeException: REQUEST FAILED: xpath=//result/doc[2]/int[@name='id'][.='13']
   [junit4]   2> 		xml response was: <?xml version="1.0" encoding="UTF-8"?>
   [junit4]   2> 	<response>
   [junit4]   2> 	<lst name="responseHeader"><int name="status">0</int><int name="QTime">35</int></lst><result name="response" numFound="14" start="0"><doc><int name="id">17</int><arr name="range_facet_l"><long>17</long></arr><arr name="lowerfilt"><str>The quote red fox jumped moon over the lazy brown dogs moon. Of course moon. Foxes and moon come back to the foxes and moon</str></arr><arr name="lowerfilt1and2"><str>The quote red fox jumped moon over the lazy brown dogs moon. Of course moon. Foxes and moon come back to the foxes and moon</str></arr><long name="_version_">1483718590381162496</long><arr name="multiDefault"><str>muLti-Default</str></arr><int name="intDefault">42</int><date name="timestamp">2014-11-03T03:11:52.42Z</date></doc><doc><int name="id">18</int><arr name="range_facet_l"><long>18</long></arr><arr name="lowerfilt"><str>The quote red fox jumped over the lazy brown dogs.</str></arr><arr name="lowerfilt1and2"><str>The quote red fox jumped over the lazy brown dogs.</str></arr><long name="_version_">1483718590383259648</long><arr name="multiDefault"><str>muLti-Default</str></arr><int name="intDefault">42</int><date name="timestamp">2014-11-03T03:11:52.423Z</date></doc><doc><int name="id">20</int><arr name="range_facet_l"><long>20</long></arr><arr name="lowerfilt"><str>The quote red fox jumped over the lazy brown dogs.</str></arr><arr name="lowerfilt1and2"><str>The quote red fox jumped over the lazy brown dogs.</str></arr><long name="_version_">1483718590388502528</long><arr name="multiDefault"><str>muLti-Default</str></arr><int name="intDefault">42</int><date name="timestamp">2014-11-03T03:11:52.428Z</date></doc><doc><int name="id">22</int><arr name="range_facet_l"><long>22</long></arr><arr name="lowerfilt"><str>The quote red fox jumped over the lazy brown dogs.</str></arr><arr name="lowerfilt1and2"><str>The quote red fox jumped over the lazy brown dogs.</str></arr><long name="_version_">1483718590392696832</long><arr name="multiDefault"><str>muLti-Default</str></arr><int name="intDefault">42</int><date name="timestamp">2014-11-03T03:11:52.432Z</date></doc><doc><int name="id">23</int><arr name="range_facet_l"><long>23</long></arr><arr name="lowerfilt"><str>The quote red fox jumped over the lazy brown dogs.</str></arr><arr name="lowerfilt1and2"><str>The quote red fox jumped over the lazy brown dogs.</str></arr><long name="_version_">1483718590394793984</long><arr name="multiDefault"><str>muLti-Default</str></arr><int name="intDefault">42</int><date name="timestamp">2014-11-03T03:11:52.434Z</date></doc><doc><int name="id">13</int><arr name="range_facet_l"><long>13</long></arr><arr name="lowerfilt"><str>The quote red fox jumped over the lazy brown dogs.</str></arr><arr name="lowerfilt1and2"><str>The quote red fox jumped over the lazy brown dogs.</str></arr><long name="_version_">1483718590364385280</long><arr name="multiDefault"><str>muLti-Default</str></arr><int name="intDefault">42</int><date name="timestamp">2014-11-03T03:11:52.405Z</date></doc><doc><int name="id">14</int><arr name="range_facet_l"><long>14</long></arr><arr name="lowerfilt"><str>The quote red fox jumped over the lazy brown dogs.</str></arr><arr name="lowerfilt1and2"><str>The quote red fox jumped over the lazy brown dogs.</str></arr><long name="_version_">1483718590366482432</long><arr name="multiDefault"><str>muLti-Default</str></arr><int name="intDefault">42</int><date name="timestamp">2014-11-03T03:11:52.407Z</date></doc><doc><int name="id">9</int><arr name="range_facet_l"><long>9</long></arr><arr name="lowerfilt"><str>The quick red fox jumped over the lazy big and large brown dogs.</str></arr><arr name="lowerfilt1and2"><str>The quick red fox jumped over the lazy big and large brown dogs.</str></arr><long name="_version_">1483718590349705216</long><arr name="multiDefault"><str>muLti-Default</str></arr><int name="intDefault">42</int><date name="timestamp">2014-11-03T03:11:52.391Z</date></doc><doc><int name="id">7</int><arr name="range_facet_l"><long>7</long></arr><arr name="lowerfilt"><str>mclaren moon or the moon and moon moon shine and the moon but moon was good foxes too</str></arr><arr name="lowerfilt1and2"><str>mclaren moon or the moon and moon moon shine and the moon but moon was good foxes too</str></arr><long name="_version_">1483718590345510912</long><arr name="multiDefault"><str>muLti-Default</str></arr><int name="intDefault">42</int><date name="timestamp">2014-11-03T03:11:52.387Z</date></doc><doc><int name="id">15</int><arr name="range_facet_l"><long>15</long></arr><arr name="lowerfilt"><str>The fat red fox jumped over the lazy brown dogs.</str></arr><arr name="lowerfilt1and2"><str>The fat red fox jumped over the lazy brown dogs.</str></arr><long name="_version_">1483718590375919616</long><arr name="multiDefault"><str>muLti-Default</str></arr><int name="intDefault">42</int><date name="timestamp">2014-11-03T03:11:52.415Z</date></doc></result>
   [junit4]   2> 	</response>
   [junit4]   2> 	
   [junit4]   2> 		request was:q=%7B%21mlt+qf%3Dlowerfilt%7D17
   [junit4]   2> 		at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:716)
   [junit4]   2> 		at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:690)
   [junit4]   2> 		at org.apache.solr.search.mlt.SimpleMLTQParserTest.doTest(SimpleMLTQParserTest.java:71)
   [junit4]   2> 		at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2> 		at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   [junit4]   2> 		at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> 		at java.lang.reflect.Method.invoke(Method.java:606)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
   [junit4]   2> 		at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
   [junit4]   2> 		at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   2> 		at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
   [junit4]   2> 		at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   2> 		at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
   [junit4]   2> 		at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2> 		at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 		at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
   [junit4]   2> 		at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2> 		at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   2> 		at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 		at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
   [junit4]   2> 		at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 	
   [junit4]   2> 2170894 T3971 oas.SolrTestCaseJ4.tearDown ###Ending doTest
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=SimpleMLTQParserTest -Dtests.method=doTest -Dtests.seed=80B8668EE9622AC1 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=mk -Dtests.timezone=PST8PDT -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   0.20s J2 | SimpleMLTQParserTest.doTest <<<
   [junit4]    > Throwable #1: java.lang.RuntimeException: Exception during query
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([80B8668EE9622AC1:27FCDE2A84D93978]:0)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:723)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:690)
   [junit4]    > 	at org.apache.solr.search.mlt.SimpleMLTQParserTest.doTest(SimpleMLTQParserTest.java:71)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]    > Caused by: java.lang.RuntimeException: REQUEST FAILED: xpath=//result/doc[2]/int[@name='id'][.='13']
   [junit4]    > 	xml response was: <?xml version="1.0" encoding="UTF-8"?>
   [junit4]    > <response>
   [junit4]    > <lst name="responseHeader"><int name="status">0</int><int name="QTime">35</int></lst><result name="response" numFound="14" start="0"><doc><int name="id">17</int><arr name="range_facet_l"><long>17</long></arr><arr name="lowerfilt"><str>The quote red fox jumped moon over the lazy brown dogs moon. Of course moon. Foxes and moon come back to the foxes and moon</str></arr><arr name="lowerfilt1and2"><str>The quote red fox jumped moon over the lazy brown dogs moon. Of course moon. Foxes and moon come back to the foxes and moon</str></arr><long name="_version_">1483718590381162496</long><arr name="multiDefault"><str>muLti-Default</str></arr><int name="intDefault">42</int><date name="timestamp">2014-11-03T03:11:52.42Z</date></doc><doc><int name="id">18</int><arr name="range_facet_l"><long>18</long></arr><arr name="lowerfilt"><str>The quote red fox jumped over the lazy brown dogs.</str></arr><arr name="lowerfilt1and2"><str>The quote red fox jumped over the lazy brown dogs.</str></arr><long name="_version_">1483718590383259648</long><arr name="multiDefault"><str>muLti-Default</str></arr><int name="intDefault">42</int><date name="timestamp">2014-11-03T03:11:52.423Z</date></doc><doc><int name="id">20</int><arr name="range_facet_l"><long>20</long></arr><arr name="lowerfilt"><str>The quote red fox jumped over the lazy brown dogs.</str></arr><arr name="lowerfilt1and2"><str>The quote red fox jumped over the lazy brown dogs.</str></arr><long name="_version_">1483718590388502528</long><arr name="multiDefault"><str>muLti-Default</str></arr><int name="intDefault">42</int><date name="timestamp">2014-11-03T03:11:52.428Z</date></doc><doc><int name="id">22</int><arr name="range_facet_l"><long>22</long></arr><arr name="lowerfilt"><str>The quote red fox jumped over the lazy brown dogs.</str></arr><arr name="lowerfilt1and2"><str>The quote red fox jumped over the lazy brown dogs.</str></arr><long name="_version_">1483718590392696832</long><arr name="multiDefault"><str>muLti-Default</str></arr><int name="intDefault">42</int><date name="timestamp">2014-11-03T03:11:52.432Z</date></doc><doc><int name="id">23</int><arr name="range_facet_l"><long>23</long></arr><arr name="lowerfilt"><str>The quote red fox jumped over the lazy brown dogs.</str></arr><arr name="lowerfilt1and2"><str>The quote red fox jumped over the lazy brown dogs.</str></arr><long name="_version_">1483718590394793984</long><arr name="multiDefault"><str>muLti-Default</str></arr><int name="intDefault">42</int><date name="timestamp">2014-11-03T03:11:52.434Z</date></doc><doc><int name="id">13</int><arr name="range_facet_l"><long>13</long></arr><arr name="lowerfilt"><str>The quote red fox jumped over the lazy brown dogs.</str></arr><arr name="lowerfilt1and2"><str>The quote red fox jumped over the lazy brown dogs.</str></arr><long name="_version_">1483718590364385280</long><arr name="multiDefault"><str>muLti-Default</str></arr><int name="intDefault">42</int><date name="timestamp">2014-11-03T03:11:52.405Z</date></doc><doc><int name="id">14</int><arr name="range_facet_l"><long>14</long></arr><arr name="lowerfilt"><str>The quote red fox jumped over the lazy brown dogs.</str></arr><arr name="lowerfilt1and2"><str>The quote red fox jumped over the lazy brown dogs.</str></arr><long name="_version_">1483718590366482432</long><arr name="multiDefault"><str>muLti-Default</str></arr><int name="intDefault">42</int><date name="timestamp">2014-11-03T03:11:52.407Z</date></doc><doc><int name="id">9</int><arr name="range_facet_l"><long>9</long></arr><arr name="lowerfilt"><str>The quick red fox jumped over the lazy big and large brown dogs.</str></arr><arr name="lowerfilt1and2"><str>The quick red fox jumped over the lazy big and large brown dogs.</str></arr><long name="_version_">1483718590349705216</long><arr name="multiDefault"><str>muLti-Default</str></arr><int name="intDefault">42</int><date name="timestamp">2014-11-03T03:11:52.391Z</date></doc><doc><int name="id">7</int><arr name="range_facet_l"><long>7</long></arr><arr name="lowerfilt"><str>mclaren moon or the moon and moon moon shine and the moon but moon was good foxes too</str></arr><arr name="lowerfilt1and2"><str>mclaren moon or the moon and moon moon shine and the moon but moon was good foxes too</str></arr><long name="_version_">1483718590345510912</long><arr name="multiDefault"><str>muLti-Default</str></arr><int name="intDefault">42</int><date name="timestamp">2014-11-03T03:11:52.387Z</date></doc><doc><int name="id">15</int><arr name="range_facet_l"><long>15</long></arr><arr name="lowerfilt"><str>The fat red fox jumped over the lazy brown dogs.</str></arr><arr name="lowerfilt1and2"><str>The fat red fox jumped over the lazy brown dogs.</str></arr><long name="_version_">1483718590375919616</long><arr name="multiDefault"><str>muLti-Default</str></arr><int name="intDefault">42</int><date name="timestamp">2014-11-03T03:11:52.415Z</date></doc></result>
   [junit4]    > </response>
   [junit4]    > 	request was:q=%7B%21mlt+qf%3Dlowerfilt%7D17
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:716)
   [junit4]    > 	... 42 more
   [junit4]   2> 2170927 T3971 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 2170927 T3971 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1504592265
   [junit4]   2> 2170928 T3971 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@2f77b967
   [junit4]   2> 2170946 T3971 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=24,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0,transaction_logs_total_size=1864,transaction_logs_total_number=1}
   [junit4]   2> 2170946 T3971 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 2170946 T3971 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 2170946 T3971 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> ASYNC  NEW_CORE C1733 name=collection1 org.apache.solr.core.SolrCore@2f77b967
   [junit4]   2> 2170948 T3971 C1733 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(NRTCachingDirectory(RAMDirectory(id=lucene-798c44a0); maxCacheMB=0.9677734375 maxMergeSizeMB=0.14453125))),segFN=segments_2,generation=2}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(NRTCachingDirectory(RAMDirectory(id=lucene-798c44a0); maxCacheMB=0.9677734375 maxMergeSizeMB=0.14453125))),segFN=segments_3,generation=3}
   [junit4]   2> 2170948 T3971 C1733 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
   [junit4]   2> 2170949 T3971 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 2170950 T3971 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 2170950 T3971 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.search.mlt.SimpleMLTQParserTest-80B8668EE9622AC1-001/init-core-data-001/index [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.search.mlt.SimpleMLTQParserTest-80B8668EE9622AC1-001/init-core-data-001/index;done=false>>]
   [junit4]   2> 2170950 T3971 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.search.mlt.SimpleMLTQParserTest-80B8668EE9622AC1-001/init-core-data-001/index
   [junit4]   2> 2170950 T3971 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.search.mlt.SimpleMLTQParserTest-80B8668EE9622AC1-001/init-core-data-001 [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.search.mlt.SimpleMLTQParserTest-80B8668EE9622AC1-001/init-core-data-001;done=false>>]
   [junit4]   2> 2170951 T3971 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.search.mlt.SimpleMLTQParserTest-80B8668EE9622AC1-001/init-core-data-001
   [junit4]   2> NOTE: leaving temporary files on disk at: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.search.mlt.SimpleMLTQParserTest-80B8668EE9622AC1-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50): {lowerfilt1and2=FSTOrd50, _version_=PostingsFormat(name=Asserting), range_facet_l=FSTOrd50, timestamp=FSTOrd50, id=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), intDefault=PostingsFormat(name=Asserting), lowerfilt=FSTOrd50, multiDefault=FSTOrd50}, docValues:{timestamp=DocValuesFormat(name=Memory)}, sim=DefaultSimilarity, locale=mk, timezone=PST8PDT
   [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_65 (64-bit)/cpus=16,threads=1,free=257392704,total=335020032
   [junit4]   2> NOTE: All tests run in this JVM: [TestShortCircuitedRequests, TestUpdate, ZkNodePropsTest, TestPerFieldSimilarity, TestManagedSchemaDynamicFieldResource, DistributedFacetPivotLargeTest, TestRemoteStreaming, TestChildDocTransformer, HttpPartitionTest, TestShardHandlerFactory, TestOverriddenPrefixQueryForCustomFieldType, TestDFRSimilarityFactory, SolrTestCaseJ4Test, QueryElevationComponentTest, HighlighterConfigTest, DistributedQueueTest, TestStressRecovery, TestQueryUtils, TestCharFilters, BasicDistributedZk2Test, RAMDirectoryFactoryTest, DefaultValueUpdateProcessorTest, TestSchemaNameResource, TestRequestStatusCollectionAPI, TestSolr4Spatial, NumericFieldsTest, BinaryUpdateRequestHandlerTest, TestElisionMultitermQuery, TestCloudInspectUtil, CopyFieldTest, LegacyHTMLStripCharFilterTest, TestRandomDVFaceting, DistribDocExpirationUpdateProcessorTest, TestSolrCoreProperties, CurrencyFieldXmlFileTest, TestMultiCoreConfBootstrap, TestDocBasedVersionConstraints, AnalyticsQueryTest, BasicFunctionalityTest, TestMaxScoreQueryParser, TestSolrQueryParser, LeaderFailoverAfterPartitionTest, TestReversedWildcardFilterFactory, InfoHandlerTest, CoreMergeIndexesAdminHandlerTest, TestCloudManagedSchemaConcurrent, UUIDUpdateProcessorFallbackTest, PingRequestHandlerTest, TestQuerySenderListener, TestSurroundQueryParser, TestDynamicFieldResource, DocValuesTest, SimpleMLTQParserTest]
   [junit4] Completed on J2 in 2.09s, 1 test, 1 error <<< FAILURES!

[...truncated 535 lines...]
   [junit4] Suite: org.apache.solr.cloud.CollectionsAPIAsyncDistributedZkTest
   [junit4]   2> Creating dataDir: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest-80B8668EE9622AC1-001/init-core-data-001
   [junit4]   2> 3280331 T12933 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (true)
   [junit4]   2> 3280331 T12933 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 3280337 T12933 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 3280347 T12933 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 3280348 T12934 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 3280448 T12933 oasc.ZkTestServer.run start zk server on port:40912
   [junit4]   2> 3280449 T12933 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 3280450 T12933 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3280455 T12940 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@34d2cc2d name:ZooKeeperConnection Watcher:127.0.0.1:40912 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3280456 T12933 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3280456 T12933 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 3280457 T12933 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 3280459 T12933 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 3280461 T12933 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3280462 T12942 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4f9241b5 name:ZooKeeperConnection Watcher:127.0.0.1:40912/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3280462 T12933 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3280463 T12933 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 3280463 T12933 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 3280465 T12933 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 3280468 T12933 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 3280469 T12933 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 3280472 T12933 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 3280472 T12933 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 3280476 T12933 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 3280477 T12933 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 3280583 T12933 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 3280584 T12933 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 3280586 T12933 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 3280587 T12933 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 3280589 T12933 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 3280590 T12933 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 3280593 T12933 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 3280593 T12933 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 3280595 T12933 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 3280596 T12933 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 3280598 T12933 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 3280599 T12933 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 3280605 T12933 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 3280605 T12933 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 3280608 T12933 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 3280608 T12933 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 3280611 T12933 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 3280611 T12933 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 3280614 T12933 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 3280615 T12933 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3280616 T12944 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@625a349b name:ZooKeeperConnection Watcher:127.0.0.1:40912/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3280617 T12933 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3280617 T12933 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 3283025 T12933 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 3283030 T12933 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 3283034 T12933 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:40916
   [junit4]   2> 3283037 T12933 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 3283037 T12933 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 3283037 T12933 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest-80B8668EE9622AC1-001/tempDir-002
   [junit4]   2> 3283038 T12933 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest-80B8668EE9622AC1-001/tempDir-002/'
   [junit4]   2> 3283101 T12933 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest-80B8668EE9622AC1-001/tempDir-002/solr.xml
   [junit4]   2> 3283234 T12933 oasc.CoreContainer.<init> New CoreContainer 659255442
   [junit4]   2> 3283234 T12933 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest-80B8668EE9622AC1-001/tempDir-002/]
   [junit4]   2> 3283236 T12933 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 3283236 T12933 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 3283236 T12933 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 3283237 T12933 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 3283237 T12933 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 3283237 T12933 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 3283238 T12933 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 3283238 T12933 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 3283238 T12933 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 3283239 T12933 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 3283241 T12933 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 3283244 T12933 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 3283244 T12933 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 3283244 T12933 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 3283245 T12933 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:40912/solr
   [junit4]   2> 3283245 T12933 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 3283246 T12933 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 3283247 T12933 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3283249 T12955 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4aa8df28 name:ZooKeeperConnection Watcher:127.0.0.1:40912 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3283249 T12933 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3283250 T12933 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 3283252 T12933 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3283253 T12957 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@44dcdf92 name:ZooKeeperConnection Watcher:127.0.0.1:40912/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3283253 T12933 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3283254 T12933 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 3283257 T12933 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 3283259 T12933 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 3283262 T12933 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 3283264 T12933 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 3283267 T12933 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 3283268 T12933 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40916_
   [junit4]   2> 3283269 T12933 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:40916_
   [junit4]   2> 3283271 T12933 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 3283273 T12933 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 3283275 T12933 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 3283276 T12933 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:40916_
   [junit4]   2> 3283277 T12933 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 3283279 T12933 oasc.Overseer.start Overseer (id=92732484566908932-127.0.0.1:40916_-n_0000000000) starting
   [junit4]   2> 3283281 T12933 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 3283289 T12933 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 3283289 T12959 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 3283290 T12933 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 3283292 T12933 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 3283294 T12933 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 3283297 T12958 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 3283302 T12961 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 3283302 T12961 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 3283303 T12961 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 3283303 T12962 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 3283306 T12958 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:40916",
   [junit4]   2> 	  "node_name":"127.0.0.1:40916_",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"down"}
   [junit4]   2> 3283306 T12958 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 3283306 T12958 oasc.Overseer$ClusterStateUpdater.createCollection state version control_collection 1
   [junit4]   2> 3283306 T12958 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 3283309 T12962 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 3284303 T12961 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 3284304 T12961 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 3284305 T12961 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 3284305 T12961 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 3284306 T12961 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 3284306 T12961 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest-80B8668EE9622AC1-001/tempDir-002/collection1/'
   [junit4]   2> 3284307 T12961 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest-80B8668EE9622AC1-001/tempDir-002/collection1/lib/.svn/' to classloader
   [junit4]   2> 3284308 T12961 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest-80B8668EE9622AC1-001/tempDir-002/collection1/lib/README' to classloader
   [junit4]   2> 3284309 T12961 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest-80B8668EE9622AC1-001/tempDir-002/collection1/lib/classes/' to classloader
   [junit4]   2> 3284405 T12961 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 3284504 T12961 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 3284605 T12961 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 3284615 T12961 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 3285193 T12961 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 3285267 T12961 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 3285270 T12961 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 3285281 T12961 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 3285307 T12961 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 3285311 T12961 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 3285312 T12961 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 3285313 T12961 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 3285313 T12961 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 3285314 T12961 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 3285314 T12961 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 3285314 T12961 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 3285315 T12961 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 3285315 T12961 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest-80B8668EE9622AC1-001/tempDir-002/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest-80B8668EE9622AC1-001/tempDir-002/collection1/data/
   [junit4]   2> 3285315 T12961 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7cdf0e95
   [junit4]   2> 3285316 T12961 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest-80B8668EE9622AC1-001/tempDir-002/collection1/data
   [junit4]   2> 3285316 T12961 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest-80B8668EE9622AC1-001/tempDir-002/collection1/data/index/
   [junit4]   2> 3285317 T12961 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest-80B8668EE9622AC1-001/tempDir-002/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 3285317 T12961 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest-80B8668EE9622AC1-001/tempDir-002/collection1/data/index
   [junit4]   2> 3285318 T12961 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=728475722, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 3285318 T12961 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-3edf547a)),segFN=segments_1,generation=1}
   [junit4]   2> 3285319 T12961 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 3285323 T12961 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 3285324 T12961 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 3285324 T12961 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 3285325 T12961 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 3285325 T12961 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 3285325 T12961 oasup.UpdateR

[...truncated too long message...]

terCloser
   [junit4]   2> 3498109 T12933 oasc.SolrCore.closeSearcher [testasynccollectioncreation_shard1_replica2] Closing main searcher on request.
   [junit4]   2> 3498110 T12933 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 3498110 T12933 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest-80B8668EE9622AC1-001/tempDir-003/testasynccollectioncreation_shard1_replica2/data [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest-80B8668EE9622AC1-001/tempDir-003/testasynccollectioncreation_shard1_replica2/data;done=false>>]
   [junit4]   2> 3498111 T12933 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest-80B8668EE9622AC1-001/tempDir-003/testasynccollectioncreation_shard1_replica2/data
   [junit4]   2> 3498111 T12933 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest-80B8668EE9622AC1-001/tempDir-003/testasynccollectioncreation_shard1_replica2/data/index [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest-80B8668EE9622AC1-001/tempDir-003/testasynccollectioncreation_shard1_replica2/data/index;done=false>>]
   [junit4]   2> 3498112 T12933 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest-80B8668EE9622AC1-001/tempDir-003/testasynccollectioncreation_shard1_replica2/data/index
   [junit4]   2> 3498112 T12933 oasc.Overseer.close Overseer (id=92732484566908935-127.0.0.1:40920_-n_0000000001) closing
   [junit4]   2> 3498113 T13305 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:40920_
   [junit4]   2> 3499615 T13304 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 3499632 T12933 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
   [junit4]   2> 3504634 T12933 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
   [junit4]   2> 3504637 T12933 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:40912 40912
   [junit4]   2> 3504996 T12934 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:40912 40912
   [junit4]   2> 3504998 T12934 oasc.ZkTestServer$ZKServerMain.runFromConfig WARN Watch limit violations: 
   [junit4]   2> 	Maximum concurrent create/delete watches above limit:
   [junit4]   2> 	
   [junit4]   2> 		3	/solr/clusterstate.json
   [junit4]   2> 		3	/solr/aliases.json
   [junit4]   2> 	
   [junit4]   2> 	Maximum concurrent children watches above limit:
   [junit4]   2> 	
   [junit4]   2> 		3	/solr/live_nodes
   [junit4]   2> 		2	/solr/overseer/queue
   [junit4]   2> 	
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=CollectionsAPIAsyncDistributedZkTest -Dtests.method=testDistribSearch -Dtests.seed=80B8668EE9622AC1 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=da -Dtests.timezone=Africa/Gaborone -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] FAILURE  225s J3 | CollectionsAPIAsyncDistributedZkTest.testDistribSearch <<<
   [junit4]    > Throwable #1: org.junit.ComparisonFailure: Shard split did not complete. Last recorded state: running expected:<[completed]> but was:<[running]>
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([80B8668EE9622AC1:15EE8969E3D4AFD]:0)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIAsyncDistributedZkTest.testSolrJAPICalls(CollectionsAPIAsyncDistributedZkTest.java:114)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIAsyncDistributedZkTest.doTest(CollectionsAPIAsyncDistributedZkTest.java:64)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 3505034 T12933 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest-80B8668EE9622AC1-001
   [junit4]   2> 224709 T12932 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50): {}, docValues:{}, sim=DefaultSimilarity, locale=da, timezone=Africa/Gaborone
   [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_65 (64-bit)/cpus=16,threads=1,free=108130400,total=415236096
   [junit4]   2> NOTE: All tests run in this JVM: [TestPHPSerializedResponseWriter, FieldMutatingUpdateProcessorTest, TestMissingGroups, TestCSVResponseWriter, RequestHandlersTest, TestSort, TestSchemaSimilarityResource, TestIntervalFaceting, TestFreeTextSuggestions, TestRecoveryHdfs, TestDocSet, TestFieldResource, TestManagedSchema, CoreContainerCoreInitFailuresTest, ResponseLogComponentTest, LoggingHandlerTest, WordBreakSolrSpellCheckerTest, ZkCLITest, BlockDirectoryTest, TestSearchPerf, EchoParamsTest, TestIndexSearcher, OutOfBoxZkACLAndCredentialsProvidersTest, TestSimpleQParserPlugin, RollingRestartTest, TestInitQParser, SolrIndexSplitterTest, SliceStateTest, MultiThreadedOCPTest, TriLevelCompositeIdRoutingTest, DocumentAnalysisRequestHandlerTest, QueryResultKeyTest, ConnectionManagerTest, AsyncMigrateRouteKeyTest, SpellCheckComponentTest, TestSolrXml, UUIDFieldTest, TestJoin, TestManagedStopFilterFactory, TestCustomSort, BlockCacheTest, ClusterStateUpdateTest, TestDistribDocBasedVersion, TestManagedSchemaFieldResource, TestLMJelinekMercerSimilarityFactory, TermVectorComponentDistributedTest, UniqFieldsUpdateProcessorFactoryTest, TestFastLRUCache, IndexBasedSpellCheckerTest, FieldAnalysisRequestHandlerTest, TestSolrXmlPersistence, TestSolr4Spatial2, LeaderElectionTest, TestDistributedSearch, SuggesterTest, TestStressLucene, SuggesterWFSTTest, SchemaVersionSpecificBehaviorTest, XsltUpdateRequestHandlerTest, StandardRequestHandlerTest, XmlUpdateRequestHandlerTest, RequiredFieldsTest, SolrPluginUtilsTest, ReturnFieldsTest, JsonLoaderTest, TestBinaryResponseWriter, SolrIndexConfigTest, TestSolrDeletionPolicy2, TestBinaryField, TestDefaultSimilarityFactory, TimeZoneUtilsTest, OpenExchangeRatesOrgProviderTest, TestSuggestSpellingConverter, SpellingQueryConverterTest, TestLRUCache, TestDocumentBuilder, CursorPagingTest, TestCursorMarkWithoutUniqueKey, TestHighlightDedupGrouping, TestTolerantSearch, AssignTest, CollectionsAPIAsyncDistributedZkTest]
   [junit4] Completed on J3 in 225.00s, 1 test, 1 failure <<< FAILURES!

[...truncated 281 lines...]
BUILD FAILED
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:532: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:473: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:61: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/extra-targets.xml:39: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build.xml:189: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/common-build.xml:502: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1358: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:965: There were test failures: 441 suites, 1801 tests, 1 error, 1 failure, 54 ignored (9 assumptions)

Total time: 196 minutes 3 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-NightlyTests-trunk #664
Archived 5 artifacts
Archive block size is 32768
Received 0 blocks and 461342789 bytes
Compression is 0.0%
Took 1 min 35 sec
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 673 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/673/

1 tests failed.
REGRESSION:  org.apache.solr.cloud.DistribDocExpirationUpdateProcessorTest.testDistribSearch

Error Message:
There are still nodes recoverying - waited for 30 seconds

Stack Trace:
java.lang.AssertionError: There are still nodes recoverying - waited for 30 seconds
	at __randomizedtesting.SeedInfo.seed([9F12A04C8A34833A:1EF42E54FD6BE306]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:178)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:840)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForThingsToLevelOut(AbstractFullDistribZkTestBase.java:1459)
	at org.apache.solr.cloud.DistribDocExpirationUpdateProcessorTest.doTest(DistribDocExpirationUpdateProcessorTest.java:79)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 12214 lines...]
   [junit4] Suite: org.apache.solr.cloud.DistribDocExpirationUpdateProcessorTest
   [junit4]   2> Creating dataDir: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-9F12A04C8A34833A-001/init-core-data-001
   [junit4]   2> 38001 T26 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (true)
   [junit4]   2> 38001 T26 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /lsvv/
   [junit4]   2> 38006 T26 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 38051 T26 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 38064 T27 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 38466 T26 oasc.ZkTestServer.run start zk server on port:44037
   [junit4]   2> 38479 T26 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 38800 T26 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 39020 T33 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@334385bc name:ZooKeeperConnection Watcher:127.0.0.1:44037 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 39021 T26 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 39022 T26 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 39038 T26 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 39201 T26 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 39202 T26 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 39203 T35 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@136a9bdc name:ZooKeeperConnection Watcher:127.0.0.1:44037/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 39204 T26 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 39204 T26 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 39213 T26 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 39217 T26 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 39220 T26 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 39222 T26 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 39226 T26 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig-doc-expire-update-processor.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 39275 T26 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 39281 T26 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 39287 T26 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 39291 T26 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 39291 T26 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 39295 T26 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 39296 T26 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 39299 T26 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 39300 T26 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 39303 T26 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 39304 T26 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 39308 T26 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 39312 T26 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 39316 T26 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 39317 T26 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 39320 T26 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 39320 T26 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 39324 T26 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 39325 T26 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 39328 T26 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 39329 T26 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 39358 T26 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 39359 T26 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 39361 T37 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@344788f3 name:ZooKeeperConnection Watcher:127.0.0.1:44037/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 39361 T26 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 39361 T26 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 45221 T26 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 45653 T26 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 45687 T26 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:65204
   [junit4]   2> 45812 T26 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 45813 T26 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 45813 T26 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-9F12A04C8A34833A-001/tempDir-002
   [junit4]   2> 45814 T26 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-9F12A04C8A34833A-001/tempDir-002/'
   [junit4]   2> 45846 T26 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-9F12A04C8A34833A-001/tempDir-002/solr.xml
   [junit4]   2> 45910 T26 oasc.CoreContainer.<init> New CoreContainer 1957815480
   [junit4]   2> 45910 T26 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-9F12A04C8A34833A-001/tempDir-002/]
   [junit4]   2> 45964 T26 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 45965 T26 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 45965 T26 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 45965 T26 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 45966 T26 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 45966 T26 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 45967 T26 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 45967 T26 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 45967 T26 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 45968 T26 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 45971 T26 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 45974 T26 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 45975 T26 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 45975 T26 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 45975 T26 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:44037/solr
   [junit4]   2> 46004 T26 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 46005 T26 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 46006 T26 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 46008 T49 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@78b898d8 name:ZooKeeperConnection Watcher:127.0.0.1:44037 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 46009 T26 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 46009 T26 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 46022 T26 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 46024 T51 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3e1d2ca0 name:ZooKeeperConnection Watcher:127.0.0.1:44037/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 46024 T26 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 46038 T26 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 46055 T26 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 46063 T26 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 46068 T26 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 46073 T26 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 46092 T26 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 46095 T26 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:65204_lsvv
   [junit4]   2> 46100 T26 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:65204_lsvv
   [junit4]   2> 46105 T26 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 46108 T26 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 46112 T26 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 46127 T26 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:65204_lsvv
   [junit4]   2> 46128 T26 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 46131 T26 oasc.Overseer.start Overseer (id=92724239186264068-127.0.0.1:65204_lsvv-n_0000000000) starting
   [junit4]   2> 46179 T26 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 46222 T26 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 46244 T53 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 46245 T26 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 46248 T26 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 46250 T26 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 46282 T52 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 46305 T55 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 46306 T55 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 46317 T55 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 46321 T56 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 46344 T52 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:65204_lsvv",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:65204/lsvv"}
   [junit4]   2> 46345 T52 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 46364 T52 oasc.Overseer$ClusterStateUpdater.createCollection state version control_collection 1
   [junit4]   2> 46374 T52 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 46391 T56 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 47319 T55 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 47319 T55 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 47321 T55 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 47321 T55 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 47323 T55 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 47324 T55 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-9F12A04C8A34833A-001/tempDir-002/collection1/'
   [junit4]   2> 47327 T55 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-9F12A04C8A34833A-001/tempDir-002/collection1/lib/README' to classloader
   [junit4]   2> 47328 T55 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-9F12A04C8A34833A-001/tempDir-002/collection1/lib/classes/' to classloader
   [junit4]   2> 47329 T55 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-9F12A04C8A34833A-001/tempDir-002/collection1/lib/.svn/' to classloader
   [junit4]   2> 47477 T55 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 47533 T55 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 47535 T55 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 47566 T55 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 48159 T55 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 48163 T55 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 48166 T55 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 48173 T55 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 48244 T55 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 48245 T55 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 48245 T55 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-9F12A04C8A34833A-001/tempDir-002/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-9F12A04C8A34833A-001/tempDir-001/control/data/
   [junit4]   2> 48246 T55 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 48247 T55 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-9F12A04C8A34833A-001/tempDir-001/control/data
   [junit4]   2> 48247 T55 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-9F12A04C8A34833A-001/tempDir-001/control/data/index/
   [junit4]   2> 48247 T55 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-9F12A04C8A34833A-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 48248 T55 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-9F12A04C8A34833A-001/tempDir-001/control/data/index
   [junit4]   2> 48249 T55 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@eab96ab
   [junit4]   2> 48250 T55 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-1b780c4a)),segFN=segments_1,generation=1}
   [junit4]   2> 48251 T55 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 48252 T55 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "convert-ttl-defaults"
   [junit4]   2> 48411 T55 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "convert-ttl-field"
   [junit4]   2> 48571 T55 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "convert-ttl-param"
   [junit4]   2> 48571 T55 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "convert-ttl-field-with-param-default"
   [junit4]   2> 48572 T55 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "scheduled-delete" (default)
   [junit4]   2> 48834 T55 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "scheduled-delete" (default)
   [junit4]   2> 48835 T55 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 48836 T55 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 48836 T55 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 48837 T55 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 48837 T55 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 48837 T55 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 48838 T55 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
   [junit4]   2> 48839 T55 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 48902 T55 oasc.RequestHandlers.initHandlersFromConfig created /admin/: solr.admin.AdminHandlers
   [junit4]   2> 48935 T55 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 48938 T55 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 48941 T55 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 48944 T55 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 48960 T55 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 48961 T55 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 48962 T55 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@67fdc673
   [junit4]   2> 48963 T55 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-1b780c4a)),segFN=segments_1,generation=1}
   [junit4]   2> 48963 T55 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 48964 T55 oass.SolrIndexSearcher.<init> Opening Searcher@1f10787[collection1] main
   [junit4]   2> 48965 T55 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 48967 T55 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 48967 T55 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 48983 T55 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 48983 T55 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 48984 T55 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 48985 T55 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 48986 T55 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 48988 T55 oascc.SolrZkClient.makePath makePath: /configs/conf1/_rest_managed.json
   [junit4]   2> 48993 T55 oasr.ManagedResourceStorage$ZooKeeperStorageIO$1.close Wrote 38 bytes to new znode /configs/conf1/_rest_managed.json
   [junit4]   2> 48994 T55 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 48994 T55 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 49020 T57 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1f10787[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 49020 T55 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 49022 T61 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:65204/lsvv collection:control_collection shard:shard1
   [junit4]   2> 49023 T26 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2
   [junit4]   2> 49024 T26 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 49061 T61 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 49086 T61 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 49089 T56 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 49090 T61 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 49091 T61 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C5 name=collection1 org.apache.solr.core.SolrCore@39503ac1 url=https://127.0.0.1:65204/lsvv/collection1 node=127.0.0.1:65204_lsvv C5_STATE=coll:control_collection core:collection1 props:{node_name=127.0.0.1:65204_lsvv, state=down, core=collection1, base_url=https://127.0.0.1:65204/lsvv}
   [junit4]   2> 49103 T61 C5 P65204 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:65204/lsvv/collection1/
   [junit4]   2> 49104 T61 C5 P65204 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 49105 T61 C5 P65204 oasc.SyncStrategy.syncToMe https://127.0.0.1:65204/lsvv/collection1/ has no replicas
   [junit4]   2> 49105 T61 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:65204/lsvv/collection1/ shard1
   [junit4]   2> 49108 T56 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 49124 T61 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 49134 T56 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 49156 T26 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 49157 T26 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 49160 T63 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@25c7f5b0 name:ZooKeeperConnection Watcher:127.0.0.1:44037/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 49161 T26 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 49162 T26 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 49165 T26 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 49175 T26 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 49176 T26 oasc.AbstractFullDistribZkTestBase.createJettys Creating collection1 with stateFormat=2
   [junit4]   2> 49176 T26 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 49178 T26 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 49180 T65 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5aa1a3a3 name:ZooKeeperConnection Watcher:127.0.0.1:44037/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 49181 T26 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 49182 T26 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 49186 T56 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 49189 T52 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: collection1
   [junit4]   2> 49190 T52 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 49190 T52 oasc.Overseer$ClusterStateUpdater.createCollection state version collection1 2
   [junit4]   2> 49196 T52 oasc.Overseer$ClusterStateUpdater.updateZkStates going to create_collection /collections/collection1/state.json
   [junit4]   2> 49199 T56 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 49199 T66 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 49239 T61 oasc.ZkController.register We are https://127.0.0.1:65204/lsvv/collection1/ and leader is https://127.0.0.1:65204/lsvv/collection1/
   [junit4]   2> 49239 T61 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:65204/lsvv
   [junit4]   2> 49240 T61 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 49240 T61 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 49240 T61 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 49244 T56 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 49253 T52 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "node_name":"127.0.0.1:65204_lsvv",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:65204/lsvv"}
   [junit4]   2> 49360 T66 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 49360 T56 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 50499 T26 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 50500 T26 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 50507 T26 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 50510 T26 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:29777
   [junit4]   2> 50514 T26 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 50514 T26 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 50515 T26 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-9F12A04C8A34833A-001/tempDir-003
   [junit4]   2> 50515 T26 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-9F12A04C8A34833A-001/tempDir-003/'
   [junit4]   2> 50575 T26 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-9F12A04C8A34833A-001/tempDir-003/solr.xml
   [junit4]   2> 50693 T26 oasc.CoreContainer.<init> New CoreContainer 107045056
   [junit4]   2> 50694 T26 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-9F12A04C8A34833A-001/tempDir-003/]
   [junit4]   2> 50695 T26 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 50696 T26 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 50696 T26 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 50696 T26 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 50697 T26 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 50697 T26 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 50698 T26 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 50698 T26 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 50698 T26 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 50699 T26 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 50703 T26 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 50706 T26 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 50706 T26 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 50707 T26 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 50707 T26 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:44037/solr
   [junit4]   2> 50708 T26 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 50708 T26 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 50709 T26 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 50711 T77 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@61c29752 name:ZooKeeperConnection Watcher:127.0.0.1:44037 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 50712 T26 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 50713 T26 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 50716 T26 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 50717 T79 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3392aed7 name:ZooKeeperConnection Watcher:127.0.0.1:44037/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 50718 T26 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 50728 T26 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 51738 T26 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:29777_lsvv
   [junit4]   2> 51740 T26 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:29777_lsvv
   [junit4]   2> 51744 T26 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 51766 T81 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 51766 T81 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 51769 T56 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 51771 T81 oasc.ZkController.preRegister Registering watch for external collection collection1
   [junit4]   2> 51771 T81 oascc.ZkStateReader.addZkWatch addZkWatch collection1
   [junit4]   2> 51772 T81 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/collection1/state.json from ZooKeeper... 
   [junit4]   2> 51782 T52 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:29777_lsvv",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:29777/lsvv"}
   [junit4]   2> 51785 T81 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 0 
   [junit4]   2> 51785 T81 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 51788 T52 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 51789 T52 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 51797 T56 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 51798 T52 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
   [junit4]   2> 51799 T80 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 51800 T80 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 1 
   [junit4]   2>  C5_STATE=coll:control_collection core:collection1 props:{node_name=127.0.0.1:65204_lsvv, state=active, core=collection1, base_url=https://127.0.0.1:65204/lsvv, leader=true}
   [junit4]   2> 52020 T58 C5 P65204 oasup.DocExpirationUpdateProcessorFactory$DeleteExpiredDocsRunnable.run Begining periodic deletion of expired docs
   [junit4]   2> 52085 T58 C5 P65204 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=true,prepareCommit=false}
   [junit4]   2> 52085 T58 C5 P65204 oasc.SolrCore.openNewSearcher SolrIndexSearcher has not changed - not re-opening: org.apache.solr.search.SolrIndexSearcher
   [junit4]   2> 52086 T58 C5 P65204 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 52087 T58 C5 P65204 oasup.LogUpdateProcessor.finish [collection1] {deleteByQuery={!cache=false}eXpField_tdt:[* TO 2014-11-01T16:33:40.282Z] (-1483587841229324288),commit=} 0 67
   [junit4]   2> 52087 T58 C5 P65204 oasup.DocExpirationUpdateProcessorFactory$DeleteExpiredDocsRunnable.run Finished periodic deletion of expired docs
   [junit4]   2> 52785 T81 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 52786 T81 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 52788 T81 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 52788 T81 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 52790 T81 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 52790 T81 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-9F12A04C8A34833A-001/tempDir-003/collection1/'
   [junit4]   2> 52793 T81 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-9F12A04C8A34833A-001/tempDir-003/collection1/lib/classes/' to classloader
   [junit4]   2> 52795 T81 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-9F12A04C8A34833A-001/tempDir-003/collection1/lib/README' to classloader
   [junit4]   2> 52796 T81 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-9F12A04C8A34833A-001/tempDir-003/collection1/lib/.svn/' to classloader
   [junit4]   2> 52884 T81 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 52940 T81 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 52943 T81 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 52979 T81 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 53591 T81 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 53597 T81 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 53602 T81 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 53611 T81 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 53759 T81 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 53760 T81 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 53761 T81 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-9F12A04C8A34833A-001/tempDir-003/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-9F12A04C8A34833A-001/tempDir-001/jetty1/
   [junit4]   2> 53762 T81 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 53763 T81 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-9F12A04C8A34833A-001/tempDir-001/jetty1
   [junit4]   2> 53764 T81 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-9F12A04C8A34833A-001/tempDir-001/jetty1/index/
   [junit4]   2> 53764 T81 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-9F12A04C8A34833A-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 53766 T81 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-9F12A04C8A34833A-001/tempDir-001/jetty1/index
   [junit4]   2> 53766 T81 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@699dad28
   [junit4]   2> 53768 T81 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-15a075f9)),segFN=segments_1,generation=1}
   [junit4]   2> 53769 T81 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 53772 T81 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "convert-ttl-defaults"
   [junit4]   2> 53772 T81 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "convert-ttl-field"
   [junit4]   2> 53773 T81 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "convert-ttl-param"
   [junit4]   2> 53774 T81 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "convert-ttl-field-with-param-default"
   [junit4]   2> 53775 T81 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "scheduled-delete" (default)
   [junit4]   2> 53776 T81 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "scheduled-delete" (default)
   [junit4]   2> 53777 T81 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 53778 T81 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 53779 T81 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 53779 T81 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 53780 T81 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 53781 T81 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 53781 T81 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
   [junit4]   2> 53782 T81 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 53783 T81 oasc.RequestHandlers.initHandlersFromConfig created /admin/: solr.admin.AdminHandlers
   [junit4]   2> 53836 T81 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 53841 T81 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 53844 T81 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 53848 T81 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 53857 T81 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 53857 T81 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 53858 T81 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@631df109
   [junit4]   2> 53860 T81 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-15a075f9)),segFN=segments_1,generation=1}
   [junit4]   2> 53860 T81 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 53861 T81 oass.SolrIndexSearcher.<init> Opening Searcher@4d08ba74[collection1] main
   [junit4]   2> 53862 T81 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 53865 T81 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 53866 T81 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 53867 T81 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 53868 T81 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 53869 T81 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 53871 T81 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
   [junit4]   2> 53873 T81 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 53873 T81 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
   [junit4]   2> 53874 T81 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 53877 T82 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4d08ba74[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 53877 T81 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 53880 T86 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:29777/lsvv collection:collection1 shard:shard2
   [junit4]   2> 53883 T26 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2
   [junit4]   2> 53884 T26 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 53885 T86 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 53905 T86 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 53911 T56 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 53912 T86 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 53913 T86 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C6 name=collection1 org.apache.solr.core.SolrCore@6c03cac5 url=https://127.0.0.1:29777/lsvv/collection1 node=127.0.0.1:29777_lsvv C6_STATE=coll:collection1 core:collection1 props:{node_name=127.0.0.1:29777_lsvv, state=down, core=collection1, base_url=https://127.0.0.1:29777/lsvv}
   [junit4]   2> 53914 T86 C6 P29777 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:29777/lsvv/collection1/
   [junit4]   2> 53915 T86 C6 P29777 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 53915 T86 C6 P29777 oasc.SyncStrategy.syncToMe https://127.0.0.1:29777/lsvv/collection1/ has no replicas
   [junit4]   2> 53916 T86 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:29777/lsvv/collection1/ shard2
   [junit4]   2> 53917 T86 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 53944 T52 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
   [junit4]   2> 53946 T80 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 53949 T80 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 2 
   [junit4]   2> 53981 T56 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 53983 T52 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
   [junit4]   2> 53985 T80 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 53987 T80 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 3 
   [junit4]   2> 53990 T86 oasc.ZkController.register We are https://127.0.0.1:29777/lsvv/collection1/ and leader is https://127.0.0.1:29777/lsvv/collection1/
   [junit4]   2> 53991 T86 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:29777/lsvv
   [junit4]   2> 53992 T86 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 53992 T86 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 53993 T86 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 53996 T56 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 54007 T86 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 3 
   [junit4]   2> 54008 T52 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "node_name":"127.0.0.1:29777_lsvv",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:29777/lsvv"}
   [junit4]   2> 54026 T52 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
   [junit4]   2> 54028 T80 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 54030 T80 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 4 
   [junit4]   2> ASYNC  NEW_CORE C7 name=collection1 org.apache.solr.core.SolrCore@39503ac1 url=https://127.0.0.1:65204/lsvv/collection1 node=127.0.0.1:65204_lsvv C7_STATE=coll:control_collection core:collection1 props:{node_name=127.0.0.1:65204_lsvv, state=active, core=collection1, base_url=https://127.0.0.1:65204/lsvv, leader=true}
   [junit4]   2> 55020 T58 C7 P65204 oasup.DocExpirationUpdateProcessorFactory$DeleteExpiredDocsRunnable.run Begining periodic deletion of expired docs
   [junit4]   2> 55022 T58 C7 P65204 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=true,prepareCommit=false}
   [junit4]   2> 55022 T58 C7 P65204 oasc.SolrCore.openNewSearcher SolrIndexSearcher has not changed - not re-opening: org.apache.solr.search.SolrIndexSearcher
   [junit4]   2> 55023 T58 C7 P65204 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 55023 T58 C7 P65204 oasup.LogUpdateProcessor.finish [collection1] {deleteByQuery={!cache=false}eXpField_tdt:[* TO 2014-11-01T16:33:43.281Z] (-1483587844343595008),commit=} 0 4
   [junit4]   2> 55024 T58 C7 P65204 oasup.DocExpirationUpdateProcessorFactory$DeleteExpiredDocsRunnable.run Finished periodic deletion of expired docs
   [junit4]   2> 56307 T26 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 56309 T26 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 56320 T26 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 56325 T26 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:57250
   [junit4]   2> 56331 T26 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 56331 T26 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 56332 T26 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-9F12A04C8A34833A-001/tempDir-004
   [junit4]   2> 56332 T26 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-9F12A04C8A34833A-001/tempDir-004/'
   [junit4]   2> 56391 T26 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-9F12A04C8A34833A-001/tempDir-004/solr.xml
   [junit4]   2> 56573 T26 oasc.CoreContainer.<init> New CoreContainer 1536282437
   [junit4]   2> 56574 T26 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-9F12A04C8A34833A-001/tempDir-004/]
   [junit4]   2> 56576 T26 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 56576 T26 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 56577 T26 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 56577 T26 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 56578 T26 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 56579 T26 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 56579 T26 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 56580 T26 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 56580 T26 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 56581 T26 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 56586 T26 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 56590 T26 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 56591 T26 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 56591 T26 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 56592 T26 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:44037/solr
   [junit4]   2> 56592 T26 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 56593 T26 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 56594 T26 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 56597 T97 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@52a6f7d1 name:ZooKeeperConnection Watcher:127.0.0.1:44037 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 56598 T26 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 56598 T26 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 56602 T26 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 56605 T99 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@29af94e2 name:ZooKeeperConnection Watcher:127.0.0.1:44037/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 56605 T26 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 56620 T26 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> ASYNC  NEW_CORE C8 name=collection1 org.apache.solr.core.SolrCore@6c03cac5 url=https://127.0.0.1:29777/lsvv/collection1 node=127.0.0.1:29777_lsvv C8_STATE=coll:collection1 core:collection1 props:{node_name=127.0.0.1:29777_lsvv, state=active, core=collection1, base_url=https://127.0.0.1:29777/lsvv, leader=true}
   [junit4]   2> 56876 T83 C8 P29777 oasup.DocExpirationUpdateProcessorFactory.iAmInChargeOfPeriodicDeletes WARN Slice in charge of periodic deletes for collection1 does not currently have a leader
   [junit4]   2> 57633 T26 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57250_lsvv
   [junit4]   2> 57635 T26 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:57250_lsvv
   [junit4]   2> 57641 T26 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 57654 T101 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 57654 T101 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 57657 T56 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 57659 T101 oasc.ZkController.preRegister Registering watch for external collection collection1
   [junit4]   2> 57659 T101 oascc.ZkStateReader.addZkWatch addZkWatch collection1
   [junit4]   2> 57660 T101 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/collection1/state.json from ZooKeeper... 
   [junit4]   2> 57662 T101 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 4 
   [junit4]   2> 57662 T101 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 57664 T52 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:57250_lsvv",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:57250/lsvv"}
   [junit4]   2> 57669 T52 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 57671 T52 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 57680 T52 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
   [junit4]   2> 57681 T100 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 57681 T80 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 57683 T100 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 5 
   [junit4]   2> 57683 T80 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 5 
   [junit4]   2> ASYNC  NEW_CORE C9 name=collection1 org.apache.solr.core.SolrCore@39503ac1 url=https://127.0.0.1:65204/lsvv/collection1 node=127.0.0.1:65204_lsvv C9_STATE=coll:control_collection core:collection1 props:{node_name=127.0.0.1:65204_lsvv, state=active, core=collection1, base_url=https://127.0.0.1:65204/lsvv, leader=true}
   [junit4]   2> 58020 T58 C9 P65204 oasup.DocExpirationUpdateProcessorFactory$DeleteExpiredDocsRunnable.run Begining periodic deletion of expired docs
   [junit4]   2> 58022 T58 C9 P65204 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=true,prepareCommit=false}
   [junit4]   2> 58022 T58 C9 P65204 oasc.SolrCore.openNewSearcher SolrIndexSearcher has not changed - not re-opening: org.apache.solr.search.SolrIndexSearcher
   [junit4]   2> 58022 T58 C9 P65204 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 58023 T58 C9 P65204 oasup.LogUpdateProcessor.finish [collection1] {deleteByQuery={!cache=false}eXpField_tdt:[* TO 2014-11-01T16:33:46.281Z] (-1483587847489323008),commit=} 0 4
   [junit4]   2> 58023 T58 C9 P65204 oasup.DocExpirationUpdateProcessorFactory$DeleteExpiredDocsRunnable.run Finished periodic deletion of expired docs
   [junit4]   2> 58663 T101 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 58664 T101 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 58665 T101 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 58665 T101 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 58667 T101 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 58667 T101 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-9F12A04C8A34833A-001/tempDir-004/collection1/'
   [junit4]   2> 58668 T101 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-9F12A04C8A34833A-001/tempDir-004/collection1/lib/.svn/' to classloader
   [junit4]   2> 58669 T101 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-9F12A04C8A34833A-001/tempDir-004/collection1/lib/README' to classloader
   [junit4]   2> 58669 T101 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-9F12A04C8A34833A-001/tempDir-004/collection1/lib/classes/' to classloader
   [junit4]   2> 58722 T101 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 58758 T101 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 58760 T101 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 58774 T101 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 59057 T101 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 59059 T101 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 59061 T101 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 59065 T101 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 59112 T101 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 59113 T101 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 59113 T101 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-9F12A04C8A34833A-001/tempDir-004/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-9F12A04C8A34833A-001/tempDir-001/jetty2/
   [junit4]   2> 59114 T101 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 59114 T101 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-9F12A04C8A34833A-001/tempDir-001/jetty2
   [junit4]   2> 59115 T101 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-9F12A04C8A34833A-001/tempDir-001/jetty2/index/
   [junit4]   2> 59115 T101 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-9F12A04C8A34833A-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 59116 T101 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-9F12A04C8A34833A-001/tempDir-001/jetty2/index
   [junit4]   2> 59116 T101 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@4db991a8
   [junit4]   2> 59117 T101 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-13cdebf)),segFN=segments_1,generation=1}
   [junit4]   2> 59118 T101 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 59119 T101 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "convert-ttl-defaults"
   [junit4]   2> 59119 T101 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "convert-ttl-field"
   [junit4]   2> 59120 T101 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "convert-ttl-param"
   [junit4]   2> 59120 T101 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "convert-ttl-field-with-param-default"
   [junit4]   2> 59120 T101 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "scheduled-delete" (default)
   [junit4]   2> 59121 T101 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "scheduled-delete" (default)
   [junit4]   2> 59121 T101 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 59122 T101 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 59122 T101 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 59122 T101 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 59123 T101 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 59123 T101 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 59123 T101 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
   [junit4]   2> 59124 T101 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 59124 T101 oasc.RequestHandlers.initHandlersFromConfig created /admin/: solr.admin.AdminHandlers
   [junit4]   2> 59138 T101 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 59139 T101 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 59141 T101 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 59142 T101 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 59146 T101 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 59146 T101 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 59147 T101 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@4ec77ce8
   [junit4]   2> 59148 T101 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-13cdebf)),segFN=segments_1,generation=1}
   [junit4]   2> 59148 T101 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 59149 T101 oass.SolrIndexSearcher.<init> Opening Searcher@4abf303c[collection1] main
   [junit4]   2> 59149 T101 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 59151 T101 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 59151 T101 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 59152 T101 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 59152 T101 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 59153 T101 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 59154 T101 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
   [junit4]   2> 59155 T101 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 59155 T101 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
   [junit4]   2> 59155 T101 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 59157 T102 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4abf303c[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 59157 T101 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 59158 T106 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:57250/lsvv collection:collection1 shard:shard1
   [junit4]   2> 59159 T26 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2
   [junit4]   2> 59159 T26 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 59160 T106 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 59168 T106 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 59171 T56 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 59172 T106 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 59172 T106 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C10 name=collection1 org.apache.solr.core.SolrCore@305f12b0 url=https://127.0.0.1:57250/lsvv/collection1 node=127.0.0.1:57250_lsvv C10_STATE=coll:collection1 core:collection1 props:{node_name=127.0.0.1:57250_lsvv, state=down, core=collection1, base_url=https://127.0.0.1:57250/lsvv}
   [junit4]   2> 59172 T106 C10 P57250 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:57250/lsvv/collection1/
   [junit4]   2> 59173 T106 C10 P57250 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 59173 T106 C10 P57250 oasc.SyncStrategy.syncToMe https://127.0.0.1:57250/lsvv/collection1/ has no replicas
   [junit4]   2> 59174 T106 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:57250/lsvv/collection1/ shard1
   [junit4]   2> 59174 T106 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 59182 T52 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
   [junit4]   2> 59183 T100 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 59183 T80 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 59184 T100 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 6 
   [junit4]   2> 59184 T80 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 6 
   [junit4]   2> 59195 T56 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 59196 T52 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
   [junit4]   2> 59197 T100 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 59197 T80 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 59198 T100 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 7 
   [junit4]   2> 59199 T80 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 7 
   [junit4]   2> 59231 T106 oasc.ZkController.register We are https://127.0.0.1:57250/lsvv/collection1/ and leader is https://127.0.0.1:57250/lsvv/collection1/
   [junit4]   2> 59232 T106 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:57250/lsvv
   [junit4]   2> 59232 T106 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 59233 T106 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 59233 T106 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 59236 T56 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 59241 T106 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 7 
   [junit4]   2> 59242 T52 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "core_node_name":"core_node2",
   [junit4]   2> 	  "node_name":"127.0.0.1:57250_lsvv",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:57250/lsvv"}
   [junit4]   2> 59251 T52 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
   [junit4]   2> 59253 T100 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 59253 T80 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 59254 T100 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 8 
   [junit4]   2> 59254 T80 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 8 
   [junit4]   2> ASYNC  NEW_CORE C11 name=collection1 org.apache.solr.core.SolrCore@6c03cac5 url=https://127.0.0.1:29777/lsvv/collection1 node=127.0.0.1:29777_lsvv C11_STATE=coll:collection1 core:collection1 props:{node_name=127.0.0.1:29777_lsvv, state=active, core=collection1, base_url=https://127.0.0.1:29777/lsvv, leader=true}
   [junit4]   2> 59876 T83 C11 P29777 oasup.DocExpirationUpdateProcessorFactory.iAmInChargeOfPeriodicDeletes Not currently in charge of periodic deletes for this collection, will not trigger delete or log again until this changes
   [junit4]   2> 60247 T26 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 60248 T26 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 60256 T26 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 60259 T26 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:20308
   [junit4]   2> 60262 T26 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 60263 T26 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 60263 T26 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-9F12A04C8A34833A-001/tempDir-005
   [junit4]   2> 60263 T26 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-9F12A04C8A34833A-001/tempDir-005/'
   [junit4]   2> 60294 T26 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-9F12A04C8A34833A-001/tempDir-005/solr.xml
   [junit4]   2> 60355 T26 oasc.CoreContainer.<init> New CoreContainer 258848646
   [junit4]   2> 60355 T26 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-9F12A04C8A34833A-001/tempDir-005/]
   [junit4]   2> 60356 T26 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 60357 T26 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 60357 T26 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to

[...truncated too long message...]

xpirationUpdateProcessorTest-9F12A04C8A34833A-001/tempDir-001/jetty4/index
   [junit4]   2> 126708 T1118 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 1 directories currently being tracked
   [junit4]   2> 126708 T1118 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-9F12A04C8A34833A-001/tempDir-001/jetty4 [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-9F12A04C8A34833A-001/tempDir-001/jetty4;done=false>>]
   [junit4]   2> 126708 T1118 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-9F12A04C8A34833A-001/tempDir-001/jetty4
   [junit4]   2> 126709 T1118 oasc.LeaderElector$ElectionWatcher.process WARN  org.apache.solr.common.SolrException: Fatal Error, SolrCore not found:collection1 in []
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:222)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:162)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:124)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:313)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:220)
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:466)
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:229)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:162)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:124)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:313)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:220)
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:466)
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:229)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:162)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:124)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:313)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:220)
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:466)
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:229)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:162)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:124)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:313)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:220)
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:466)
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:229)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:162)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:124)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:357)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3$1.run(SolrZkClient.java:266)
   [junit4]   2> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
   [junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 126711 T26 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/lsvv,null}
   [junit4]   2> 128921 T26 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
   [junit4]   2> 128925 T26 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:44037 44037
   [junit4]   2> 129047 T27 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:44037 44037
   [junit4]   2> 129131 T27 oasc.ZkTestServer$ZKServerMain.runFromConfig WARN Watch limit violations: 
   [junit4]   2> 	Maximum concurrent create/delete watches above limit:
   [junit4]   2> 	
   [junit4]   2> 		6	/solr/aliases.json
   [junit4]   2> 		6	/solr/clusterstate.json
   [junit4]   2> 		4	/solr/collections/collection1/state.json
   [junit4]   2> 	
   [junit4]   2> 	Maximum concurrent children watches above limit:
   [junit4]   2> 	
   [junit4]   2> 		6	/solr/live_nodes
   [junit4]   2> 		5	/solr/overseer/collection-queue-work
   [junit4]   2> 		5	/solr/overseer/queue
   [junit4]   2> 	
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=DistribDocExpirationUpdateProcessorTest -Dtests.method=testDistribSearch -Dtests.seed=9F12A04C8A34833A -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=it_IT -Dtests.timezone=Europe/Andorra -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 91.2s J2 | DistribDocExpirationUpdateProcessorTest.testDistribSearch <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: There are still nodes recoverying - waited for 30 seconds
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([9F12A04C8A34833A:1EF42E54FD6BE306]:0)
   [junit4]    > 	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:178)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:840)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForThingsToLevelOut(AbstractFullDistribZkTestBase.java:1459)
   [junit4]    > 	at org.apache.solr.cloud.DistribDocExpirationUpdateProcessorTest.doTest(DistribDocExpirationUpdateProcessorTest.java:79)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 129213 T26 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-9F12A04C8A34833A-001
   [junit4]   2> 91219 T25 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50): {id=PostingsFormat(name=Memory doPackFST= true), _version_=Lucene50(blocksize=128)}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=it_IT, timezone=Europe/Andorra
   [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_65 (64-bit)/cpus=16,threads=1,free=97865560,total=268959744
   [junit4]   2> NOTE: All tests run in this JVM: [HighlighterTest, TestOmitPositions, CoreMergeIndexesAdminHandlerTest, DistribDocExpirationUpdateProcessorTest]
   [junit4] Completed on J2 in 92.01s, 1 test, 1 failure <<< FAILURES!

[...truncated 1392 lines...]
BUILD FAILED
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:532: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:473: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:61: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/extra-targets.xml:39: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build.xml:189: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/common-build.xml:502: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1358: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:965: There were test failures: 441 suites, 1801 tests, 1 failure, 54 ignored (9 assumptions)

Total time: 226 minutes 32 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-NightlyTests-trunk #664
Archived 5 artifacts
Archive block size is 32768
Received 0 blocks and 693736566 bytes
Compression is 0.0%
Took 2 min 5 sec
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 672 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/672/

2 tests failed.
FAILED:  org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload

Error Message:
expected:<[{indexVersion=1414758079019,generation=2,filelist=[_5.cfe, _5.cfs, _5.si, _7.fdt, _7.fdx, _7.fnm, _7.nvd, _7.nvm, _7.si, _7_MockRandom_0.doc, _7_MockRandom_0.sd, _7_MockRandom_0.tio, _7_MockRandom_0.tipo, _8.cfe, _8.cfs, _8.si, segments_2]}]> but was:<[{indexVersion=1414758079019,generation=3,filelist=[_7.fdt, _7.fdx, _7.fnm, _7.nvd, _7.nvm, _7.si, _7_MockRandom_0.doc, _7_MockRandom_0.sd, _7_MockRandom_0.tio, _7_MockRandom_0.tipo, _9.fdt, _9.fdx, _9.fnm, _9.nvd, _9.nvm, _9.si, _9_MockRandom_0.doc, _9_MockRandom_0.sd, _9_MockRandom_0.tib, _9_MockRandom_0.tii, segments_3]}, {indexVersion=1414758079019,generation=2,filelist=[_5.cfe, _5.cfs, _5.si, _7.fdt, _7.fdx, _7.fnm, _7.nvd, _7.nvm, _7.si, _7_MockRandom_0.doc, _7_MockRandom_0.sd, _7_MockRandom_0.tio, _7_MockRandom_0.tipo, _8.cfe, _8.cfs, _8.si, segments_2]}]>

Stack Trace:
java.lang.AssertionError: expected:<[{indexVersion=1414758079019,generation=2,filelist=[_5.cfe, _5.cfs, _5.si, _7.fdt, _7.fdx, _7.fnm, _7.nvd, _7.nvm, _7.si, _7_MockRandom_0.doc, _7_MockRandom_0.sd, _7_MockRandom_0.tio, _7_MockRandom_0.tipo, _8.cfe, _8.cfs, _8.si, segments_2]}]> but was:<[{indexVersion=1414758079019,generation=3,filelist=[_7.fdt, _7.fdx, _7.fnm, _7.nvd, _7.nvm, _7.si, _7_MockRandom_0.doc, _7_MockRandom_0.sd, _7_MockRandom_0.tio, _7_MockRandom_0.tipo, _9.fdt, _9.fdx, _9.fnm, _9.nvd, _9.nvm, _9.si, _9_MockRandom_0.doc, _9_MockRandom_0.sd, _9_MockRandom_0.tib, _9_MockRandom_0.tii, segments_3]}, {indexVersion=1414758079019,generation=2,filelist=[_5.cfe, _5.cfs, _5.si, _7.fdt, _7.fdx, _7.fnm, _7.nvd, _7.nvm, _7.si, _7_MockRandom_0.doc, _7_MockRandom_0.sd, _7_MockRandom_0.tio, _7_MockRandom_0.tipo, _8.cfe, _8.cfs, _8.si, segments_2]}]>
	at __randomizedtesting.SeedInfo.seed([A0D6DBA6BD68BF69:8501C096CD20B16A]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:147)
	at org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload(TestReplicationHandler.java:1175)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.search.TestSearcherReuse.test

Error Message:
expected same:<Searcher@2e2fc629[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(6.0.0):C1) Uninverting(_5(6.0.0):c1) Uninverting(_4(6.0.0):c2) Uninverting(_6(6.0.0):C1) Uninverting(_8(6.0.0):c1)))}> was not:<Searcher@3b692fcc[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(6.0.0):C1) Uninverting(_5(6.0.0):c1) Uninverting(_9(6.0.0):c3) Uninverting(_8(6.0.0):c1)))}>

Stack Trace:
java.lang.AssertionError: expected same:<Searcher@2e2fc629[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(6.0.0):C1) Uninverting(_5(6.0.0):c1) Uninverting(_4(6.0.0):c2) Uninverting(_6(6.0.0):C1) Uninverting(_8(6.0.0):c1)))}> was not:<Searcher@3b692fcc[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(6.0.0):C1) Uninverting(_5(6.0.0):c1) Uninverting(_9(6.0.0):c3) Uninverting(_8(6.0.0):c1)))}>
	at __randomizedtesting.SeedInfo.seed([A0D6DBA6BD68BF69:2882E47C1394D291]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotSame(Assert.java:641)
	at org.junit.Assert.assertSame(Assert.java:580)
	at org.junit.Assert.assertSame(Assert.java:593)
	at org.apache.solr.search.TestSearcherReuse.assertSearcherHasNotChanged(TestSearcherReuse.java:247)
	at org.apache.solr.search.TestSearcherReuse.test(TestSearcherReuse.java:104)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 12530 lines...]
   [junit4] Suite: org.apache.solr.handler.TestReplicationHandler
   [junit4]   2> Creating dataDir: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-A0D6DBA6BD68BF69-001/init-core-data-001
   [junit4]   2> 1767851 T8901 oas.SolrTestCaseJ4.setUp ###Starting doTestReplicateAfterCoreReload
   [junit4]   2> 1767868 T8901 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1767874 T8901 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:50242
   [junit4]   2> 1767875 T8901 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1767876 T8901 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1767876 T8901 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-A0D6DBA6BD68BF69-001/solr-instance-001
   [junit4]   2> 1767876 T8901 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-A0D6DBA6BD68BF69-001/solr-instance-001/'
   [junit4]   2> 1767925 T8901 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-A0D6DBA6BD68BF69-001/solr-instance-001/solr.xml
   [junit4]   2> 1768011 T8901 oasc.CoreContainer.<init> New CoreContainer 1840354923
   [junit4]   2> 1768012 T8901 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-A0D6DBA6BD68BF69-001/solr-instance-001/]
   [junit4]   2> 1768013 T8901 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1768014 T8901 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1768014 T8901 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1768015 T8901 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1768015 T8901 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 1768016 T8901 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1768016 T8901 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1768017 T8901 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1768017 T8901 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1768018 T8901 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1768019 T8901 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 1768019 T8901 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1768020 T8901 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1768020 T8901 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1768027 T8911 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-A0D6DBA6BD68BF69-001/solr-instance-001/collection1/'
   [junit4]   2> 1768063 T8911 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 1768149 T8911 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1768150 T8911 oass.IndexSchema.readSchema Reading Solr Schema from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-A0D6DBA6BD68BF69-001/solr-instance-001/collection1/conf/schema.xml
   [junit4]   2> 1768157 T8911 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1768174 T8911 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1768175 T8911 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from instancedir /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-A0D6DBA6BD68BF69-001/solr-instance-001/collection1/
   [junit4]   2> 1768176 T8911 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1768176 T8911 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-A0D6DBA6BD68BF69-001/solr-instance-001/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-A0D6DBA6BD68BF69-001/solr-instance-001/collection1/data/
   [junit4]   2> 1768176 T8911 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 1768178 T8911 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-A0D6DBA6BD68BF69-001/solr-instance-001/collection1/data
   [junit4]   2> 1768179 T8911 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-A0D6DBA6BD68BF69-001/solr-instance-001/collection1/data/index/
   [junit4]   2> 1768180 T8911 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-A0D6DBA6BD68BF69-001/solr-instance-001/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1768181 T8911 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-A0D6DBA6BD68BF69-001/solr-instance-001/collection1/data/index
   [junit4]   2> 1768182 T8911 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@79cf2868
   [junit4]   2> 1768182 T8911 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-759965f4)),segFN=segments_1,generation=1}
   [junit4]   2> 1768183 T8911 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1768183 T8911 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1768184 T8911 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1768184 T8911 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1768184 T8911 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1768184 T8911 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1768185 T8911 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1768185 T8911 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
   [junit4]   2> 1768185 T8911 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
   [junit4]   2> 1768185 T8911 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
   [junit4]   2> 1768185 T8911 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1768187 T8911 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1768189 T8911 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1768190 T8911 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1768192 T8911 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1768194 T8911 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1768194 T8911 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1768195 T8911 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=2, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.10907706589224242]
   [junit4]   2> 1768195 T8911 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-759965f4)),segFN=segments_1,generation=1}
   [junit4]   2> 1768195 T8911 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1768195 T8911 oass.SolrIndexSearcher.<init> Opening Searcher@682e0ced[collection1] main
   [junit4]   2> 1768196 T8911 oasr.ManagedResourceStorage$FileStorageIO.configure File-based storage initialized to use dir: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-A0D6DBA6BD68BF69-001/solr-instance-001/collection1/conf
   [junit4]   2> 1768196 T8911 oasr.RestManager.init Initializing RestManager with initArgs: {storageDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-A0D6DBA6BD68BF69-001/solr-instance-001/collection1/conf}
   [junit4]   2> 1768196 T8911 oasr.ManagedResourceStorage.load Reading _rest_managed.json using file:dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-A0D6DBA6BD68BF69-001/solr-instance-001/collection1/conf
   [junit4]   2> 1768197 T8911 oasr.ManagedResource.reloadFromStorage WARN No stored data found for /rest/managed
   [junit4]   2> 1768197 T8911 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using file:dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-A0D6DBA6BD68BF69-001/solr-instance-001/collection1/conf
   [junit4]   2> 1768197 T8911 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 1768198 T8911 oash.ReplicationHandler.inform Replication enabled for following config files: schema.xml,xslt/dummy.xsl
   [junit4]   2> 1768198 T8911 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 1768198 T8912 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@682e0ced[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1768198 T8911 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1768199 T8901 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2
   [junit4]   2> 1768199 T8901 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1768207 T8901 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1768210 T8901 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:50247
   [junit4]   2> 1768211 T8901 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1768211 T8901 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1768211 T8901 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-A0D6DBA6BD68BF69-001/solr-instance-002
   [junit4]   2> 1768212 T8901 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-A0D6DBA6BD68BF69-001/solr-instance-002/'
   [junit4]   2> 1768226 T8901 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-A0D6DBA6BD68BF69-001/solr-instance-002/solr.xml
   [junit4]   2> 1768259 T8901 oasc.CoreContainer.<init> New CoreContainer 198017426
   [junit4]   2> 1768260 T8901 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-A0D6DBA6BD68BF69-001/solr-instance-002/]
   [junit4]   2> 1768261 T8901 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1768261 T8901 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1768262 T8901 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1768262 T8901 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1768262 T8901 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 1768262 T8901 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1768262 T8901 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1768263 T8901 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1768263 T8901 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1768263 T8901 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1768264 T8901 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 1768264 T8901 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1768264 T8901 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1768265 T8901 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1768268 T8923 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-A0D6DBA6BD68BF69-001/solr-instance-002/collection1/'
   [junit4]   2> 1768305 T8923 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 1768336 T8923 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1768337 T8923 oass.IndexSchema.readSchema Reading Solr Schema from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-A0D6DBA6BD68BF69-001/solr-instance-002/collection1/conf/schema.xml
   [junit4]   2> 1768342 T8923 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1768350 T8923 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1768351 T8923 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from instancedir /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-A0D6DBA6BD68BF69-001/solr-instance-002/collection1/
   [junit4]   2> 1768351 T8923 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1768351 T8923 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-A0D6DBA6BD68BF69-001/solr-instance-002/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-A0D6DBA6BD68BF69-001/solr-instance-002/collection1/data/
   [junit4]   2> 1768352 T8923 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 1768353 T8923 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-A0D6DBA6BD68BF69-001/solr-instance-002/collection1/data
   [junit4]   2> 1768354 T8923 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-A0D6DBA6BD68BF69-001/solr-instance-002/collection1/data/index/
   [junit4]   2> 1768354 T8923 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-A0D6DBA6BD68BF69-001/solr-instance-002/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1768355 T8923 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-A0D6DBA6BD68BF69-001/solr-instance-002/collection1/data/index
   [junit4]   2> 1768356 T8923 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@2ea315aa
   [junit4]   2> 1768356 T8923 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-14fa3f69)),segFN=segments_1,generation=1}
   [junit4]   2> 1768356 T8923 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1768357 T8923 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1768357 T8923 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1768358 T8923 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1768358 T8923 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1768358 T8923 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1768358 T8923 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
   [junit4]   2> 1768358 T8923 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
   [junit4]   2> 1768358 T8923 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
   [junit4]   2> 1768359 T8923 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1768359 T8923 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1768372 T8923 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1768374 T8923 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1768376 T8923 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1768377 T8923 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1768379 T8923 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1768379 T8923 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1768380 T8923 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=2, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.10907706589224242]
   [junit4]   2> 1768380 T8923 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-14fa3f69)),segFN=segments_1,generation=1}
   [junit4]   2> 1768381 T8923 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1768381 T8923 oass.SolrIndexSearcher.<init> Opening Searcher@2a30b211[collection1] main
   [junit4]   2> 1768381 T8923 oasr.ManagedResourceStorage$FileStorageIO.configure File-based storage initialized to use dir: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-A0D6DBA6BD68BF69-001/solr-instance-002/collection1/conf
   [junit4]   2> 1768381 T8923 oasr.RestManager.init Initializing RestManager with initArgs: {storageDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-A0D6DBA6BD68BF69-001/solr-instance-002/collection1/conf}
   [junit4]   2> 1768382 T8923 oasr.ManagedResourceStorage.load Reading _rest_managed.json using file:dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-A0D6DBA6BD68BF69-001/solr-instance-002/collection1/conf
   [junit4]   2> 1768382 T8923 oasr.ManagedResource.reloadFromStorage WARN No stored data found for /rest/managed
   [junit4]   2> 1768383 T8923 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using file:dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-A0D6DBA6BD68BF69-001/solr-instance-002/collection1/conf
   [junit4]   2> 1768383 T8923 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 1768387 T8923 oash.SnapPuller.startExecutorService Poll Scheduled at an interval of 1000ms
   [junit4]   2> 1768387 T8924 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2a30b211[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1768387 T8923 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1768388 T8901 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2
   [junit4]   2> 1768388 T8901 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1768389 T8901 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=198017426
   [junit4]   2> 1768391 T8901 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@42bea9da
   [junit4]   2> 1768392 T8901 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
   [junit4]   2> 1768392 T8901 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 1768392 T8901 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 1768393 T8901 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 1768396 T8901 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 1768397 T8901 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 1768397 T8901 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-A0D6DBA6BD68BF69-001/solr-instance-002/collection1/data [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-A0D6DBA6BD68BF69-001/solr-instance-002/collection1/data;done=false>>]
   [junit4]   2> 1768398 T8901 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-A0D6DBA6BD68BF69-001/solr-instance-002/collection1/data
   [junit4]   2> 1768398 T8901 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-A0D6DBA6BD68BF69-001/solr-instance-002/collection1/data/index [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-A0D6DBA6BD68BF69-001/solr-instance-002/collection1/data/index;done=false>>]
   [junit4]   2> 1768399 T8901 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-A0D6DBA6BD68BF69-001/solr-instance-002/collection1/data/index
   [junit4]   2> 1768400 T8901 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
   [junit4]   2> 1773215 T8901 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1840354923
   [junit4]   2> 1773218 T8901 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@1c5a62c8
   [junit4]   2> 1773218 T8901 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
   [junit4]   2> 1773218 T8901 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 1773219 T8901 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 1773219 T8901 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 1773221 T8901 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 1773221 T8901 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 1773222 T8901 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-A0D6DBA6BD68BF69-001/solr-instance-001/collection1/data/index [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-A0D6DBA6BD68BF69-001/solr-instance-001/collection1/data/index;done=false>>]
   [junit4]   2> 1773222 T8901 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-A0D6DBA6BD68BF69-001/solr-instance-001/collection1/data/index
   [junit4]   2> 1773223 T8901 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-A0D6DBA6BD68BF69-001/solr-instance-001/collection1/data [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-A0D6DBA6BD68BF69-001/solr-instance-001/collection1/data;done=false>>]
   [junit4]   2> 1773223 T8901 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-A0D6DBA6BD68BF69-001/solr-instance-001/collection1/data
   [junit4]   2> 1773224 T8901 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
   [junit4]   2> 1777875 T8901 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1777879 T8901 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:50269
   [junit4]   2> 1777880 T8901 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1777880 T8901 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1777880 T8901 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-A0D6DBA6BD68BF69-001/solr-instance-001
   [junit4]   2> 1777881 T8901 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-A0D6DBA6BD68BF69-001/solr-instance-001/'
   [junit4]   2> 1777897 T8901 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-A0D6DBA6BD68BF69-001/solr-instance-001/solr.xml
   [junit4]   2> 1777942 T8901 oasc.CoreContainer.<init> New CoreContainer 1326993072
   [junit4]   2> 1777943 T8901 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-A0D6DBA6BD68BF69-001/solr-instance-001/]
   [junit4]   2> 1777944 T8901 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1777945 T8901 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1777945 T8901 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1777945 T8901 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1777945 T8901 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 1777946 T8901 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1777946 T8901 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1777946 T8901 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1777947 T8901 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1777947 T8901 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1777948 T8901 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 1777948 T8901 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1777948 T8901 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1777949 T8901 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1777953 T8936 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-A0D6DBA6BD68BF69-001/solr-instance-001/collection1/'
   [junit4]   2> 1777999 T8936 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 1778049 T8936 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1778050 T8936 oass.IndexSchema.readSchema Reading Solr Schema from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-A0D6DBA6BD68BF69-001/solr-instance-001/collection1/conf/schema.xml
   [junit4]   2> 1778057 T8936 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1778068 T8936 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1778069 T8936 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from instancedir /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-A0D6DBA6BD68BF69-001/solr-instance-001/collection1/
   [junit4]   2> 1778070 T8936 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1778070 T8936 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-A0D6DBA6BD68BF69-001/solr-instance-001/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-A0D6DBA6BD68BF69-001/solr-instance-001/collection1/data/
   [junit4]   2> 1778070 T8936 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 1778073 T8936 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-A0D6DBA6BD68BF69-001/solr-instance-001/collection1/data
   [junit4]   2> 1778074 T8936 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-A0D6DBA6BD68BF69-001/solr-instance-001/collection1/data/index/
   [junit4]   2> 1778074 T8936 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-A0D6DBA6BD68BF69-001/solr-instance-001/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1778076 T8936 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-A0D6DBA6BD68BF69-001/solr-instance-001/collection1/data/index
   [junit4]   2> 1778076 T8936 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@6fc6ffaf
   [junit4]   2> 1778077 T8936 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-1e287d7d)),segFN=segments_1,generation=1}
   [junit4]   2> 1778077 T8936 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1778078 T8936 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1778078 T8936 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1778079 T8936 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1778079 T8936 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1778079 T8936 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1778079 T8936 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1778080 T8936 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
   [junit4]   2> 1778080 T8936 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
   [junit4]   2> 1778080 T8936 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
   [junit4]   2> 1778080 T8936 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1778082 T8936 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1778084 T8936 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1778086 T8936 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1778088 T8936 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1778091 T8936 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1778091 T8936 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1778091 T8936 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=2, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.10907706589224242]
   [junit4]   2> 1778092 T8936 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-1e287d7d)),segFN=segments_1,generation=1}
   [junit4]   2> 1778092 T8936 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1778093 T8936 oass.SolrIndexSearcher.<init> Opening Searcher@4ff6a067[collection1] main
   [junit4]   2> 1778093 T8936 oasr.ManagedResourceStorage$FileStorageIO.configure File-based storage initialized to use dir: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-A0D6DBA6BD68BF69-001/solr-instance-001/collection1/conf
   [junit4]   2> 1778093 T8936 oasr.RestManager.init Initializing RestManager with initArgs: {storageDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-A0D6DBA6BD68BF69-001/solr-instance-001/collection1/conf}
   [junit4]   2> 1778094 T8936 oasr.ManagedResourceStorage.load Reading _rest_managed.json using file:dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-A0D6DBA6BD68BF69-001/solr-instance-001/collection1/conf
   [junit4]   2> 1778094 T8936 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using file:dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-A0D6DBA6BD68BF69-001/solr-instance-001/collection1/conf
   [junit4]   2> 1778094 T8936 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
   [junit4]   2> 1778095 T8936 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 1778095 T8936 oash.ReplicationHandler.inform Replication enabled for following config files: schema.xml
   [junit4]   2> 1778095 T8936 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 1778095 T8937 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4ff6a067[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1778095 T8936 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1778096 T8901 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2
   [junit4]   2> 1778097 T8901 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> ASYNC  NEW_CORE C949 name=collection1 org.apache.solr.core.SolrCore@830ec47
   [junit4]   2> 1778099 T8929 C949 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 1778100 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {deleteByQuery=*:*} 0 1
   [junit4]   2> 1778105 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[0]} 0 1
   [junit4]   2> 1778107 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[1]} 0 0
   [junit4]   2> 1778111 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[2]} 0 0
   [junit4]   2> 1778113 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[3]} 0 0
   [junit4]   2> 1778115 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[4]} 0 0
   [junit4]   2> 1778118 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[5]} 0 0
   [junit4]   2> 1778121 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[6]} 0 0
   [junit4]   2> 1778124 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[7]} 0 0
   [junit4]   2> 1778128 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[8]} 0 0
   [junit4]   2> 1778131 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[9]} 0 0
   [junit4]   2> 1778134 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[10]} 0 0
   [junit4]   2> 1778137 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[11]} 0 0
   [junit4]   2> 1778139 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[12]} 0 0
   [junit4]   2> 1778141 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[13]} 0 0
   [junit4]   2> 1778144 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[14]} 0 0
   [junit4]   2> 1778147 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[15]} 0 0
   [junit4]   2> 1778149 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[16]} 0 0
   [junit4]   2> 1778152 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[17]} 0 0
   [junit4]   2> 1778155 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[18]} 0 0
   [junit4]   2> 1778158 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[19]} 0 0
   [junit4]   2> 1778161 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[20]} 0 0
   [junit4]   2> 1778165 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[21]} 0 0
   [junit4]   2> 1778167 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[22]} 0 0
   [junit4]   2> 1778170 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[23]} 0 0
   [junit4]   2> 1778173 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[24]} 0 0
   [junit4]   2> 1778175 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[25]} 0 0
   [junit4]   2> 1778178 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[26]} 0 0
   [junit4]   2> 1778181 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[27]} 0 0
   [junit4]   2> 1778184 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[28]} 0 0
   [junit4]   2> 1778186 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[29]} 0 0
   [junit4]   2> 1778188 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[30]} 0 0
   [junit4]   2> 1778190 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[31]} 0 0
   [junit4]   2> 1778192 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[32]} 0 0
   [junit4]   2> 1778195 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[33]} 0 0
   [junit4]   2> 1778197 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[34]} 0 0
   [junit4]   2> 1778199 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[35]} 0 0
   [junit4]   2> 1778201 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[36]} 0 0
   [junit4]   2> 1778203 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[37]} 0 0
   [junit4]   2> 1778205 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[38]} 0 0
   [junit4]   2> 1778207 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[39]} 0 0
   [junit4]   2> 1778210 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[40]} 0 0
   [junit4]   2> 1778212 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[41]} 0 0
   [junit4]   2> 1778214 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[42]} 0 0
   [junit4]   2> 1778217 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[43]} 0 0
   [junit4]   2> 1778219 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[44]} 0 0
   [junit4]   2> 1778221 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[45]} 0 0
   [junit4]   2> 1778223 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[46]} 0 0
   [junit4]   2> 1778225 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[47]} 0 0
   [junit4]   2> 1778227 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[48]} 0 0
   [junit4]   2> 1778229 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[49]} 0 0
   [junit4]   2> 1778233 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[50]} 0 0
   [junit4]   2> 1778236 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[51]} 0 0
   [junit4]   2> 1778238 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[52]} 0 0
   [junit4]   2> 1778241 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[53]} 0 0
   [junit4]   2> 1778243 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[54]} 0 0
   [junit4]   2> 1778245 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[55]} 0 0
   [junit4]   2> 1778248 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[56]} 0 0
   [junit4]   2> 1778250 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[57]} 0 0
   [junit4]   2> 1778252 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[58]} 0 0
   [junit4]   2> 1778254 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[59]} 0 0
   [junit4]   2> 1778256 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[60]} 0 0
   [junit4]   2> 1778258 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[61]} 0 0
   [junit4]   2> 1778260 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[62]} 0 0
   [junit4]   2> 1778262 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[63]} 0 0
   [junit4]   2> 1778265 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[64]} 0 0
   [junit4]   2> 1778267 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[65]} 0 0
   [junit4]   2> 1778270 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[66]} 0 0
   [junit4]   2> 1778272 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[67]} 0 0
   [junit4]   2> 1778274 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[68]} 0 0
   [junit4]   2> 1778277 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[69]} 0 0
   [junit4]   2> 1778279 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[70]} 0 0
   [junit4]   2> 1778281 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[71]} 0 0
   [junit4]   2> 1778283 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[72]} 0 0
   [junit4]   2> 1778285 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[73]} 0 0
   [junit4]   2> 1778287 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[74]} 0 0
   [junit4]   2> 1778289 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[75]} 0 0
   [junit4]   2> 1778292 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[76]} 0 0
   [junit4]   2> 1778295 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[77]} 0 0
   [junit4]   2> 1778297 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[78]} 0 0
   [junit4]   2> 1778300 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[79]} 0 0
   [junit4]   2> 1778302 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[80]} 0 0
   [junit4]   2> 1778305 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[81]} 0 0
   [junit4]   2> 1778308 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[82]} 0 0
   [junit4]   2> 1778311 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[83]} 0 0
   [junit4]   2> 1778313 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[84]} 0 0
   [junit4]   2> 1778316 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[85]} 0 0
   [junit4]   2> 1778319 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[86]} 0 0
   [junit4]   2> 1778321 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[87]} 0 0
   [junit4]   2> 1778324 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[88]} 0 0
   [junit4]   2> 1778327 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[89]} 0 0
   [junit4]   2> 1778330 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[90]} 0 0
   [junit4]   2> 1778333 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[91]} 0 0
   [junit4]   2> 1778336 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[92]} 0 0
   [junit4]   2> 1778338 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[93]} 0 0
   [junit4]   2> 1778340 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[94]} 0 0
   [junit4]   2> 1778342 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[95]} 0 0
   [junit4]   2> 1778344 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[96]} 0 0
   [junit4]   2> 1778347 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[97]} 0 0
   [junit4]   2> 1778349 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[98]} 0 0
   [junit4]   2> 1778351 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[99]} 0 0
   [junit4]   2> 1778353 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[100]} 0 0
   [junit4]   2> 1778357 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[101]} 0 0
   [junit4]   2> 1778359 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[102]} 0 0
   [junit4]   2> 1778362 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[103]} 0 0
   [junit4]   2> 1778366 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[104]} 0 0
   [junit4]   2> 1778368 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[105]} 0 0
   [junit4]   2> 1778371 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[106]} 0 0
   [junit4]   2> 1778374 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[107]} 0 0
   [junit4]   2> 1778376 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[108]} 0 0
   [junit4]   2> 1778378 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[109]} 0 0
   [junit4]   2> 1778380 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[110]} 0 0
   [junit4]   2> 1778382 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[111]} 0 0
   [junit4]   2> 1778385 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[112]} 0 0
   [junit4]   2> 1778387 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[113]} 0 0
   [junit4]   2> 1778389 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[114]} 0 0
   [junit4]   2> 1778392 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[115]} 0 0
   [junit4]   2> 1778394 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[116]} 0 0
   [junit4]   2> 1778397 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[117]} 0 0
   [junit4]   2> 1778399 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[118]} 0 0
   [junit4]   2> 1778401 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[119]} 0 0
   [junit4]   2> 1778404 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[120]} 0 0
   [junit4]   2> 1778407 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[121]} 0 0
   [junit4]   2> 1778409 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[122]} 0 0
   [junit4]   2> 1778412 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[123]} 0 0
   [junit4]   2> 1778414 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[124]} 0 0
   [junit4]   2> 1778416 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[125]} 0 0
   [junit4]   2> 1778419 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[126]} 0 0
   [junit4]   2> 1778422 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[127]} 0 0
   [junit4]   2> 1778425 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[128]} 0 0
   [junit4]   2> 1778427 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[129]} 0 0
   [junit4]   2> 1778429 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[130]} 0 0
   [junit4]   2> 1778432 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[131]} 0 0
   [junit4]   2> 1778434 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[132]} 0 0
   [junit4]   2> 1778436 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[133]} 0 0
   [junit4]   2> 1778438 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[134]} 0 0
   [junit4]   2> 1778442 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[135]} 0 0
   [junit4]   2> 1778444 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[136]} 0 0
   [junit4]   2> 1778446 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[137]} 0 0
   [junit4]   2> 1778448 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[138]} 0 0
   [junit4]   2> 1778452 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[139]} 0 0
   [junit4]   2> 1778454 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[140]} 0 0
   [junit4]   2> 1778456 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[141]} 0 0
   [junit4]   2> 1778458 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[142]} 0 0
   [junit4]   2> 1778461 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[143]} 0 0
   [junit4]   2> 1778463 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[144]} 0 0
   [junit4]   2> 1778466 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[145]} 0 0
   [junit4]   2> 1778469 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[146]} 0 0
   [junit4]   2> 1778471 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[147]} 0 0
   [junit4]   2> 1778474 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[148]} 0 0
   [junit4]   2> 1778476 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[149]} 0 0
   [junit4]   2> 1778478 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[150]} 0 0
   [junit4]   2> 1778480 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[151]} 0 0
   [junit4]   2> 1778483 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[152]} 0 0
   [junit4]   2> 1778485 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[153]} 0 0
   [junit4]   2> 1778488 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[154]} 0 0
   [junit4]   2> 1778491 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[155]} 0 0
   [junit4]   2> 1778494 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[156]} 0 0
   [junit4]   2> 1778496 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[157]} 0 0
   [junit4]   2> 1778498 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[158]} 0 0
   [junit4]   2> 1778501 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[159]} 0 0
   [junit4]   2> 1778504 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[160]} 0 0
   [junit4]   2> 1778506 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[161]} 0 0
   [junit4]   2> 1778508 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[162]} 0 0
   [junit4]   2> 1778510 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[163]} 0 0
   [junit4]   2> 1778513 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[164]} 0 0
   [junit4]   2> 1778516 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[165]} 0 0
   [junit4]   2> 1778518 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[166]} 0 0
   [junit4]   2> 1778520 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[167]} 0 0
   [junit4]   2> 1778523 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[168]} 0 0
   [junit4]   2> 1778526 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[169]} 0 0
   [junit4]   2> 1778529 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[170]} 0 0
   [junit4]   2> 1778531 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[171]} 0 0
   [junit4]   2> 1778533 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[172]} 0 0
   [junit4]   2> 1778535 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[173]} 0 0
   [junit4]   2> 1778537 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[174]} 0 0
   [junit4]   2> 1778540 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[175]} 0 0
   [junit4]   2> 1778542 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[176]} 0 0
   [junit4]   2> 1778545 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[177]} 0 0
   [junit4]   2> 1778547 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[178]} 0 0
   [junit4]   2> 1778550 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[179]} 0 0
   [junit4]   2> 1778552 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[180]} 0 0
   [junit4]   2> 1778554 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[181]} 0 0
   [junit4]   2> 1778557 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[182]} 0 0
   [junit4]   2> 1778559 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[183]} 0 0
   [junit4]   2> 1778561 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[184]} 0 0
   [junit4]   2> 1778563 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[185]} 0 0
   [junit4]   2> 1778565 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[186]} 0 0
   [junit4]   2> 1778567 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[187]} 0 0
   [junit4]   2> 1778570 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[188]} 0 0
   [junit4]   2> 1778573 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[189]} 0 0
   [junit4]   2> 1778575 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[190]} 0 0
   [junit4]   2> 1778578 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[191]} 0 0
   [junit4]   2> 1778580 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[192]} 0 0
   [junit4]   2> 1778582 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[193]} 0 0
   [junit4]   2> 1778584 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[194]} 0 0
   [junit4]   2> 1778586 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[195]} 0 0
   [junit4]   2> 1778588 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[196]} 0 0
   [junit4]   2> 1778590 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[197]} 0 0
   [junit4]   2> 1778592 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[198]} 0 0
   [junit4]   2> 1778595 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[199]} 0 0
   [junit4]   2> 1778598 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[200]} 0 0
   [junit4]   2> 1778600 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[201]} 0 0
   [junit4]   2> 1778602 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[202]} 0 0
   [junit4]   2> 1778604 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[203]} 0 0
   [junit4]   2> 1778606 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[204]} 0 0
   [junit4]   2> 1778608 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[205]} 0 0
   [junit4]   2> 1778610 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[206]} 0 0
   [junit4]   2> 1778612 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[207]} 0 0
   [junit4]   2> 1778615 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[208]} 0 0
   [junit4]   2> 1778618 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[209]} 0 0
   [junit4]   2> 1778621 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[210]} 0 0
   [junit4]   2> 1778624 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[211]} 0 0
   [junit4]   2> 1778627 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[212]} 0 0
   [junit4]   2> 1778629 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[213]} 0 0
   [junit4]   2> 1778632 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[214]} 0 0
   [junit4]   2> 1778636 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[215]} 0 0
   [junit4]   2> 1778639 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[216]} 0 0
   [junit4]   2> 1778641 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[217]} 0 0
   [junit4]   2> 1778645 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[218]} 0 0
   [junit4]   2> 1778647 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[219]} 0 0
   [junit4]   2> 1778649 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[220]} 0 0
   [junit4]   2> 1778651 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[221]} 0 0
   [junit4]   2> 1778653 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[222]} 0 0
   [junit4]   2> 1778655 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[223]} 0 0
   [junit4]   2> 1778657 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[224]} 0 0
   [junit4]   2> 1778659 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[225]} 0 0
   [junit4]   2> 1778661 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[226]} 0 0
   [junit4]   2> 1778664 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[227]} 0 0
   [junit4]   2> 1778667 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[228]} 0 0
   [junit4]   2> 1778670 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[229]} 0 0
   [junit4]   2> 1778672 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[230]} 0 0
   [junit4]   2> 1778674 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[231]} 0 0
   [junit4]   2> 1778676 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[232]} 0 0
   [junit4]   2> 1778679 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[233]} 0 0
   [junit4]   2> 1778681 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[234]} 0 0
   [junit4]   2> 1778684 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[235]} 0 0
   [junit4]   2> 1778686 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[236]} 0 0
   [junit4]   2> 1778688 T8929 C949 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[237]} 0 0
   [junit4]   2> 1778690 T8929 C949 oasup.LogUpdateProcessor.finish [

[...truncated too long message...]

 [junit4]   2> 		commit{dir=NRTCachingDirectory(MMapDirectory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.search.TestSearcherReuse-A0D6DBA6BD68BF69-001/init-core-data-001/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@70b2ddb3; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_9,generation=9}
   [junit4]   2> 3485192 T12388 C1526 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 9
   [junit4]   2> 3485196 T12388 C1526 oass.SolrIndexSearcher.<init> Opening Searcher@34640dd5[collection1] main
   [junit4]   2> 3485197 T12388 C1526 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 3485197 T12390 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@34640dd5[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3485202 T12388 C1526 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {commit=} 0 18
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestSearcherReuse -Dtests.method=test -Dtests.seed=A0D6DBA6BD68BF69 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=es_SV -Dtests.timezone=America/North_Dakota/New_Salem -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] FAILURE 0.43s J2 | TestSearcherReuse.test <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: expected same:<Searcher@2e2fc629[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(6.0.0):C1) Uninverting(_5(6.0.0):c1) Uninverting(_4(6.0.0):c2) Uninverting(_6(6.0.0):C1) Uninverting(_8(6.0.0):c1)))}> was not:<Searcher@3b692fcc[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(6.0.0):C1) Uninverting(_5(6.0.0):c1) Uninverting(_9(6.0.0):c3) Uninverting(_8(6.0.0):c1)))}>
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([A0D6DBA6BD68BF69:2882E47C1394D291]:0)
   [junit4]    > 	at org.apache.solr.search.TestSearcherReuse.assertSearcherHasNotChanged(TestSearcherReuse.java:247)
   [junit4]    > 	at org.apache.solr.search.TestSearcherReuse.test(TestSearcherReuse.java:104)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 3485217 T12388 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 3485218 T12388 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1816460082
   [junit4]   2> 3485218 T12388 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@5a68cccb
   [junit4]   2> 3485220 T12388 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=8,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=6,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0,transaction_logs_total_size=643,transaction_logs_total_number=7}
   [junit4]   2> 3485220 T12388 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 3485221 T12388 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 3485221 T12388 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 3485222 T12388 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 3485223 T12388 oasc.CachingDirectoryFactory.close Closing NRTCachingDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 3485223 T12388 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.search.TestSearcherReuse-A0D6DBA6BD68BF69-001/init-core-data-001 [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.search.TestSearcherReuse-A0D6DBA6BD68BF69-001/init-core-data-001;done=false>>]
   [junit4]   2> 3485224 T12388 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.search.TestSearcherReuse-A0D6DBA6BD68BF69-001/init-core-data-001
   [junit4]   2> 3485224 T12388 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.search.TestSearcherReuse-A0D6DBA6BD68BF69-001/init-core-data-001/index [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.search.TestSearcherReuse-A0D6DBA6BD68BF69-001/init-core-data-001/index;done=false>>]
   [junit4]   2> 3485225 T12388 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.search.TestSearcherReuse-A0D6DBA6BD68BF69-001/init-core-data-001/index
   [junit4]   2> NOTE: leaving temporary files on disk at: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J2/temp/solr.search.TestSearcherReuse-A0D6DBA6BD68BF69-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50): {}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=false,coord=crazy): {}, locale=es_SV, timezone=America/North_Dakota/New_Salem
   [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_65 (64-bit)/cpus=16,threads=1,free=227616056,total=344457216
   [junit4]   2> NOTE: All tests run in this JVM: [RAMDirectoryFactoryTest, OverseerCollectionProcessorTest, SimpleMLTQParserTest, TestRandomDVFaceting, TestFieldSortValues, HttpPartitionTest, TestQuerySenderListener, DefaultValueUpdateProcessorTest, TestSchemaNameResource, TestCollationFieldDocValues, IndexSchemaTest, HdfsChaosMonkeySafeLeaderTest, CustomCollectionTest, PingRequestHandlerTest, MultiTermTest, ZkNodePropsTest, HardAutoCommitTest, BasicDistributedZk2Test, CopyFieldTest, TestDistributedSearch, TestMaxScoreQueryParser, BinaryUpdateRequestHandlerTest, TestCloudInspectUtil, TestIndexingPerformance, TestStressLucene, TestPseudoReturnFields, DistributedQueryComponentCustomSortTest, TestStressUserVersions, AnalyticsQueryTest, TestRequestStatusCollectionAPI, TestCharFilters, TestRealTimeGet, TestHashPartitioner, HdfsUnloadDistributedZkTest, DistributedDebugComponentTest, DistributedIntervalFacetingTest, DistributedQueryComponentOptimizationTest, LegacyHTMLStripCharFilterTest, HighlighterTest, TestDFRSimilarityFactory, TestRemoteStreaming, TestSolrCoreProperties, InfoHandlerTest, DistributedFacetPivotLargeTest, TestReversedWildcardFilterFactory, QueryParsingTest, HdfsRecoveryZkTest, TestSolrQueryParser, DistributedQueueTest, HdfsWriteToMultipleCollectionsTest, TestPerFieldSimilarity, TestReplicationHandler, UpdateRequestProcessorFactoryTest, DistribDocExpirationUpdateProcessorTest, VMParamsZkACLAndCredentialsProvidersTest, HdfsBasicDistributedZk2Test, HdfsBasicDistributedZkTest, HdfsCollectionsAPIDistributedZkTest, HdfsSyncSliceTest, CachingDirectoryFactoryTest, TestInitParams, TestNonNRTOpen, TestShardHandlerFactory, TestSolrXml, TestSolrXmlPersistence, CoreAdminRequestStatusTest, CoreMergeIndexesAdminHandlerTest, DistributedExpandComponentTest, DistributedFacetPivotLongTailTest, DistributedFacetPivotSmallTest, DistributedSuggestComponentTest, TestIntervalFaceting, TestManagedSchemaFieldTypeResource, TestSchemaResource, TestSchemaVersionResource, BadCopyFieldTest, ChangedSchemaMergeTest, DateRangeFieldTest, EnumFieldTest, ModifyConfFileTest, TestBulkSchemaConcurrent, TestSchemaManager, AnalyticsMergeStrategyTest, CursorMarkTest, MergeStrategyTest, RankQueryTest, TestAddFieldRealTimeGet, TestComplexPhraseQParserPlugin, TestInitQParser, TestNoOpRegenerator, TestOverriddenPrefixQueryForCustomFieldType, TestReRankQParserPlugin, TestRecoveryHdfs, TestSearcherReuse]
   [junit4] Completed on J2 in 1.14s, 1 test, 1 failure <<< FAILURES!

[...truncated 83 lines...]
BUILD FAILED
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/build.xml:532: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/build.xml:473: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/build.xml:61: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/extra-targets.xml:39: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build.xml:189: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/common-build.xml:502: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/lucene/common-build.xml:1358: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/lucene/common-build.xml:965: There were test failures: 441 suites, 1801 tests, 2 failures, 54 ignored (9 assumptions)

Total time: 205 minutes 1 second
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-NightlyTests-trunk #664
Archived 5 artifacts
Archive block size is 32768
Received 0 blocks and 456546303 bytes
Compression is 0.0%
Took 1 min 21 sec
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Re: [JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 671 - Still Failing

Posted by Chris Hostetter <ho...@fucit.org>.

Pretty sure this is a test only issue ... I filed LUCENE-6036 for this, 
but don't really have time to try and fix it at the moment...

https://issues.apache.org/jira/browse/LUCENE-6036



: Date: Thu, 30 Oct 2014 12:02:25 +0000 (UTC)
: From: Apache Jenkins Server <je...@builds.apache.org>
: Reply-To: dev@lucene.apache.org
: To: dev@lucene.apache.org
: Subject: [JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 671 - Still
:     Failing
: 
: Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/671/
: 
: 1 tests failed.
: REGRESSION:  org.apache.lucene.index.TestIndexWriterOnJRECrash.testNRTThreads
: 
: Error Message:
: posix_spawn is not a supported process launch mechanism on this platform.
: 
: Stack Trace:
: java.lang.Error: posix_spawn is not a supported process launch mechanism on this platform.
: 	at __randomizedtesting.SeedInfo.seed([B2D360EA192CA242:290A74F158D7B429]:0)
: 	at java.lang.UNIXProcess$1.run(UNIXProcess.java:111)
: 	at java.lang.UNIXProcess$1.run(UNIXProcess.java:93)
: 	at java.security.AccessController.doPrivileged(Native Method)
: 	at java.lang.UNIXProcess.<clinit>(UNIXProcess.java:91)
: 	at java.lang.ProcessImpl.start(ProcessImpl.java:130)
: 	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1028)
: 	at org.apache.lucene.index.TestIndexWriterOnJRECrash.forkTest(TestIndexWriterOnJRECrash.java:113)
: 	at org.apache.lucene.index.TestIndexWriterOnJRECrash.testNRTThreads(TestIndexWriterOnJRECrash.java:59)
: 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
: 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
: 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
: 	at java.lang.reflect.Method.invoke(Method.java:606)
: 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
: 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
: 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
: 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
: 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
: 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
: 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
: 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
: 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
: 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
: 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
: 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
: 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
: 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
: 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
: 	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
: 	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
: 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
: 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
: 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
: 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
: 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
: 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
: 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
: 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
: 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
: 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
: 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
: 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
: 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
: 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
: 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
: 	at java.lang.Thread.run(Thread.java:745)
: 
: 
: 
: 
: Build Log:
: [...truncated 550 lines...]
:    [junit4] Suite: org.apache.lucene.index.TestIndexWriterOnJRECrash
:    [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
:    [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestIndexWriterOnJRECrash -Dtests.method=testNRTThreads -Dtests.seed=B2D360EA192CA242 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=tr -Dtests.timezone=Africa/Monrovia -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
:    [junit4] ERROR   0.04s J0 | TestIndexWriterOnJRECrash.testNRTThreads <<<
:    [junit4]    > Throwable #1: java.lang.Error: posix_spawn is not a supported process launch mechanism on this platform.
:    [junit4]    > 	at __randomizedtesting.SeedInfo.seed([B2D360EA192CA242:290A74F158D7B429]:0)
:    [junit4]    > 	at java.lang.UNIXProcess$1.run(UNIXProcess.java:111)
:    [junit4]    > 	at java.lang.UNIXProcess$1.run(UNIXProcess.java:93)
:    [junit4]    > 	at java.security.AccessController.doPrivileged(Native Method)
:    [junit4]    > 	at java.lang.UNIXProcess.<clinit>(UNIXProcess.java:91)
:    [junit4]    > 	at java.lang.ProcessImpl.start(ProcessImpl.java:130)
:    [junit4]    > 	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1028)
:    [junit4]    > 	at org.apache.lucene.index.TestIndexWriterOnJRECrash.forkTest(TestIndexWriterOnJRECrash.java:113)
:    [junit4]    > 	at org.apache.lucene.index.TestIndexWriterOnJRECrash.testNRTThreads(TestIndexWriterOnJRECrash.java:59)
:    [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
:    [junit4]   2> NOTE: leaving temporary files on disk at: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/lucene/build/core/test/J0/temp/lucene.index.TestIndexWriterOnJRECrash-B2D360EA192CA242-001
:    [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50): {}, docValues:{}, sim=DefaultSimilarity, locale=tr, timezone=Africa/Monrovia
:    [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_65 (64-bit)/cpus=16,threads=1,free=173363664,total=209715200
:    [junit4]   2> NOTE: All tests run in this JVM: [TestIsCurrent, TestCodecUtil, Test4GBStoredFields, TestLucene50NormsFormat, TestPagedBytes, TestStressIndexing, TestMixedCodecs, TestPayloadExplanations, TestIndexInput, TestFilterSpans, TestBytesRef, TestAssertingDocValuesFormat, TestFileSwitchDirectory, TestMultiValuedNumericRangeQuery, TestIndexWriterOnJRECrash]
:    [junit4] Completed on J0 in 0.08s, 1 test, 1 error <<< FAILURES!
: 
: [...truncated 1132 lines...]
: BUILD FAILED
: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/build.xml:532: The following error occurred while executing this line:
: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/build.xml:473: The following error occurred while executing this line:
: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/build.xml:61: The following error occurred while executing this line:
: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/extra-targets.xml:39: The following error occurred while executing this line:
: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/lucene/build.xml:49: The following error occurred while executing this line:
: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/lucene/common-build.xml:1358: The following error occurred while executing this line:
: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/lucene/common-build.xml:965: There were test failures: 404 suites, 3309 tests, 1 error, 42 ignored (32 assumptions)
: 
: Total time: 90 minutes 39 seconds
: Build step 'Invoke Ant' marked build as failure
: Archiving artifacts
: Sending artifact delta relative to Lucene-Solr-NightlyTests-trunk #664
: Archived 5 artifacts
: Archive block size is 32768
: Received 0 blocks and 3634458 bytes
: Compression is 0.0%
: Took 0.88 sec
: Recording test results
: Email was triggered for: Failure
: Sending email for trigger: Failure
: 
: 
: 

-Hoss
http://www.lucidworks.com/

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 671 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/671/

1 tests failed.
REGRESSION:  org.apache.lucene.index.TestIndexWriterOnJRECrash.testNRTThreads

Error Message:
posix_spawn is not a supported process launch mechanism on this platform.

Stack Trace:
java.lang.Error: posix_spawn is not a supported process launch mechanism on this platform.
	at __randomizedtesting.SeedInfo.seed([B2D360EA192CA242:290A74F158D7B429]:0)
	at java.lang.UNIXProcess$1.run(UNIXProcess.java:111)
	at java.lang.UNIXProcess$1.run(UNIXProcess.java:93)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.lang.UNIXProcess.<clinit>(UNIXProcess.java:91)
	at java.lang.ProcessImpl.start(ProcessImpl.java:130)
	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1028)
	at org.apache.lucene.index.TestIndexWriterOnJRECrash.forkTest(TestIndexWriterOnJRECrash.java:113)
	at org.apache.lucene.index.TestIndexWriterOnJRECrash.testNRTThreads(TestIndexWriterOnJRECrash.java:59)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 550 lines...]
   [junit4] Suite: org.apache.lucene.index.TestIndexWriterOnJRECrash
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestIndexWriterOnJRECrash -Dtests.method=testNRTThreads -Dtests.seed=B2D360EA192CA242 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=tr -Dtests.timezone=Africa/Monrovia -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   0.04s J0 | TestIndexWriterOnJRECrash.testNRTThreads <<<
   [junit4]    > Throwable #1: java.lang.Error: posix_spawn is not a supported process launch mechanism on this platform.
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([B2D360EA192CA242:290A74F158D7B429]:0)
   [junit4]    > 	at java.lang.UNIXProcess$1.run(UNIXProcess.java:111)
   [junit4]    > 	at java.lang.UNIXProcess$1.run(UNIXProcess.java:93)
   [junit4]    > 	at java.security.AccessController.doPrivileged(Native Method)
   [junit4]    > 	at java.lang.UNIXProcess.<clinit>(UNIXProcess.java:91)
   [junit4]    > 	at java.lang.ProcessImpl.start(ProcessImpl.java:130)
   [junit4]    > 	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1028)
   [junit4]    > 	at org.apache.lucene.index.TestIndexWriterOnJRECrash.forkTest(TestIndexWriterOnJRECrash.java:113)
   [junit4]    > 	at org.apache.lucene.index.TestIndexWriterOnJRECrash.testNRTThreads(TestIndexWriterOnJRECrash.java:59)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> NOTE: leaving temporary files on disk at: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/lucene/build/core/test/J0/temp/lucene.index.TestIndexWriterOnJRECrash-B2D360EA192CA242-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50): {}, docValues:{}, sim=DefaultSimilarity, locale=tr, timezone=Africa/Monrovia
   [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_65 (64-bit)/cpus=16,threads=1,free=173363664,total=209715200
   [junit4]   2> NOTE: All tests run in this JVM: [TestIsCurrent, TestCodecUtil, Test4GBStoredFields, TestLucene50NormsFormat, TestPagedBytes, TestStressIndexing, TestMixedCodecs, TestPayloadExplanations, TestIndexInput, TestFilterSpans, TestBytesRef, TestAssertingDocValuesFormat, TestFileSwitchDirectory, TestMultiValuedNumericRangeQuery, TestIndexWriterOnJRECrash]
   [junit4] Completed on J0 in 0.08s, 1 test, 1 error <<< FAILURES!

[...truncated 1132 lines...]
BUILD FAILED
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/build.xml:532: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/build.xml:473: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/build.xml:61: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/extra-targets.xml:39: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/lucene/build.xml:49: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/lucene/common-build.xml:1358: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/lucene/common-build.xml:965: There were test failures: 404 suites, 3309 tests, 1 error, 42 ignored (32 assumptions)

Total time: 90 minutes 39 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-NightlyTests-trunk #664
Archived 5 artifacts
Archive block size is 32768
Received 0 blocks and 3634458 bytes
Compression is 0.0%
Took 0.88 sec
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 670 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/670/

1 tests failed.
FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testDistribSearch

Error Message:
Captured an uncaught exception in thread: Thread[id=3850, name=Thread-1223, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=3850, name=Thread-1223, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
Caused by: java.lang.NullPointerException
	at __randomizedtesting.SeedInfo.seed([BAC342B198BF0FEB]:0)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:1044)




Build Log:
[...truncated 11726 lines...]
   [junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
   [junit4]   2> Creating dataDir: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BAC342B198BF0FEB-001/init-core-data-001
   [junit4]   2> 671916 T3019 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
   [junit4]   2> 671916 T3019 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 671922 T3019 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 671923 T3019 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 671924 T3020 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 672024 T3019 oasc.ZkTestServer.run start zk server on port:57521
   [junit4]   2> 672025 T3019 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 672026 T3019 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 672031 T3026 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@654fdfb9 name:ZooKeeperConnection Watcher:127.0.0.1:57521 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 672031 T3019 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 672031 T3019 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 672032 T3019 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 672034 T3019 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 672036 T3019 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 672037 T3028 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4c192e0b name:ZooKeeperConnection Watcher:127.0.0.1:57521/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 672037 T3019 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 672037 T3019 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 672038 T3019 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 672040 T3019 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 672042 T3019 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 672044 T3019 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 672046 T3019 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 672047 T3019 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 672050 T3019 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 672051 T3019 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 672053 T3019 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 672054 T3019 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 672056 T3019 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 672057 T3019 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 672059 T3019 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 672060 T3019 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 672062 T3019 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 672062 T3019 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 672065 T3019 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 672065 T3019 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 672067 T3019 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 672068 T3019 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 672070 T3019 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 672071 T3019 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 672073 T3019 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 672073 T3019 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 672076 T3019 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 672076 T3019 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 672079 T3019 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 672079 T3019 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 672081 T3030 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@242d90b7 name:ZooKeeperConnection Watcher:127.0.0.1:57521/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 672081 T3019 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 672081 T3019 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> Using old style solr.xml
   [junit4]   2> 672083 T3019 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 672084 T3019 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 672085 T3032 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@58e5e25 name:ZooKeeperConnection Watcher:127.0.0.1:57521 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 672085 T3019 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 672085 T3019 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 672086 T3019 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 672087 T3019 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 672088 T3019 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 672089 T3034 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2d6f4e1a name:ZooKeeperConnection Watcher:127.0.0.1:57521/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 672090 T3019 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 672090 T3019 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 672092 T3019 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf2/solrconfig.xml
   [junit4]   2> 672092 T3019 oascc.SolrZkClient.makePath makePath: /configs/conf2/solrconfig.xml
   [junit4]   2> 672095 T3019 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf2/schema.xml
   [junit4]   2> 672095 T3019 oascc.SolrZkClient.makePath makePath: /configs/conf2/schema.xml
   [junit4]   2> 672098 T3019 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf2/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 672098 T3019 oascc.SolrZkClient.makePath makePath: /configs/conf2/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 672100 T3019 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf2/stopwords.txt
   [junit4]   2> 672101 T3019 oascc.SolrZkClient.makePath makePath: /configs/conf2/stopwords.txt
   [junit4]   2> 672103 T3019 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf2/protwords.txt
   [junit4]   2> 672103 T3019 oascc.SolrZkClient.makePath makePath: /configs/conf2/protwords.txt
   [junit4]   2> 672105 T3019 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf2/currency.xml
   [junit4]   2> 672106 T3019 oascc.SolrZkClient.makePath makePath: /configs/conf2/currency.xml
   [junit4]   2> 672108 T3019 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf2/open-exchange-rates.json
   [junit4]   2> 672109 T3019 oascc.SolrZkClient.makePath makePath: /configs/conf2/open-exchange-rates.json
   [junit4]   2> 672111 T3019 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf2/mapping-ISOLatin1Accent.txt
   [junit4]   2> 672111 T3019 oascc.SolrZkClient.makePath makePath: /configs/conf2/mapping-ISOLatin1Accent.txt
   [junit4]   2> 672113 T3019 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf2/old_synonyms.txt
   [junit4]   2> 672114 T3019 oascc.SolrZkClient.makePath makePath: /configs/conf2/old_synonyms.txt
   [junit4]   2> 672116 T3019 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf2/synonyms.txt
   [junit4]   2> 672116 T3019 oascc.SolrZkClient.makePath makePath: /configs/conf2/synonyms.txt
   [junit4]   2> 672119 T3019 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/core/src/test-files/solr/collection1/conf/elevate.xml to /configs/conf2/elevate.xml
   [junit4]   2> 672119 T3019 oascc.SolrZkClient.makePath makePath: /configs/conf2/elevate.xml
   [junit4]   2> 673273 T3019 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 673287 T3019 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 673291 T3019 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:33847
   [junit4]   2> 673293 T3019 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 673294 T3019 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 673294 T3019 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BAC342B198BF0FEB-001/tempDir-002
   [junit4]   2> 673295 T3019 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BAC342B198BF0FEB-001/tempDir-002/'
   [junit4]   2> 673338 T3019 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BAC342B198BF0FEB-001/tempDir-002/solr.xml
   [junit4]   2> 673417 T3019 oasc.CoreContainer.<init> New CoreContainer 999010979
   [junit4]   2> 673417 T3019 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BAC342B198BF0FEB-001/tempDir-002/]
   [junit4]   2> 673418 T3019 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 673419 T3019 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 673419 T3019 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 673420 T3019 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 673420 T3019 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 673420 T3019 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 673421 T3019 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 673421 T3019 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 673421 T3019 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 673422 T3019 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 673425 T3019 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=3400000&connTimeout=45000&retry=false
   [junit4]   2> 673427 T3019 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 673427 T3019 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 673428 T3019 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 673428 T3019 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:57521/solr
   [junit4]   2> 673429 T3019 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 673429 T3019 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 673430 T3019 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 673432 T3045 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@368f9385 name:ZooKeeperConnection Watcher:127.0.0.1:57521 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 673432 T3019 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 673433 T3019 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 673434 T3019 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 673435 T3047 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@77a72333 name:ZooKeeperConnection Watcher:127.0.0.1:57521/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 673435 T3019 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 673437 T3019 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 673439 T3019 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 673441 T3019 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 673443 T3019 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 673445 T3019 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 673448 T3019 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 673449 T3019 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33847_
   [junit4]   2> 673450 T3019 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:33847_
   [junit4]   2> 673452 T3019 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 673453 T3019 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 673455 T3019 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 673457 T3019 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:33847_
   [junit4]   2> 673457 T3019 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 673459 T3019 oasc.Overseer.start Overseer (id=92705533017784326-127.0.0.1:33847_-n_0000000000) starting
   [junit4]   2> 673461 T3019 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 673468 T3019 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 673468 T3049 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 673469 T3019 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 673471 T3019 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 673472 T3019 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 673475 T3048 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 673478 T3019 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J3
   [junit4]   2> 673478 T3019 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 673483 T3019 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 673484 T3019 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 673485 T3053 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7a34fbbb name:ZooKeeperConnection Watcher:127.0.0.1:57521/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 673485 T3019 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 673486 T3019 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 673487 T3019 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 673489 T3019 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 673490 T3019 oasc.AbstractFullDistribZkTestBase.createJettys Creating collection1 with stateFormat=2
   [junit4]   2> 673490 T3019 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 673491 T3019 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 673492 T3055 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@20dc5a31 name:ZooKeeperConnection Watcher:127.0.0.1:57521/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 673492 T3019 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 673492 T3019 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 673494 T3056 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 673496 T3048 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: collection1
   [junit4]   2> 673496 T3048 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 673496 T3048 oasc.Overseer$ClusterStateUpdater.createCollection state version collection1 2
   [junit4]   2> 673498 T3048 oasc.Overseer$ClusterStateUpdater.updateZkStates going to create_collection /collections/collection1/state.json
   [junit4]   2> 673499 T3056 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 673500 T3057 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 674688 T3019 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 674689 T3019 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 674692 T3019 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 674694 T3019 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:10701
   [junit4]   2> 674697 T3019 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 674697 T3019 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 674697 T3019 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BAC342B198BF0FEB-001/tempDir-003
   [junit4]   2> 674697 T3019 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BAC342B198BF0FEB-001/tempDir-003/'
   [junit4]   2> 674726 T3019 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BAC342B198BF0FEB-001/tempDir-003/solr.xml
   [junit4]   2> 674779 T3019 oasc.CoreContainer.<init> New CoreContainer 563729374
   [junit4]   2> 674779 T3019 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BAC342B198BF0FEB-001/tempDir-003/]
   [junit4]   2> 674780 T3019 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 674780 T3019 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 674780 T3019 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 674780 T3019 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 674781 T3019 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 674781 T3019 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 674781 T3019 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 674781 T3019 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 674782 T3019 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 674782 T3019 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 674784 T3019 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=3400000&connTimeout=45000&retry=false
   [junit4]   2> 674786 T3019 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 674786 T3019 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 674786 T3019 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 674786 T3019 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:57521/solr
   [junit4]   2> 674787 T3019 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 674787 T3019 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 674788 T3019 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 674789 T3068 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1df60e59 name:ZooKeeperConnection Watcher:127.0.0.1:57521 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 674789 T3019 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 674789 T3019 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 674791 T3019 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 674792 T3070 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@30974b7e name:ZooKeeperConnection Watcher:127.0.0.1:57521/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 674792 T3019 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 674797 T3019 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 675802 T3019 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:10701_
   [junit4]   2> 675803 T3019 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:10701_
   [junit4]   2> 675806 T3019 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 675812 T3019 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J3
   [junit4]   2> 675812 T3019 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 677393 T3019 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 677394 T3019 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 677398 T3019 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 677400 T3019 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:13927
   [junit4]   2> 677402 T3019 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 677402 T3019 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 677402 T3019 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BAC342B198BF0FEB-001/tempDir-004
   [junit4]   2> 677402 T3019 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BAC342B198BF0FEB-001/tempDir-004/'
   [junit4]   2> 677448 T3019 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BAC342B198BF0FEB-001/tempDir-004/solr.xml
   [junit4]   2> 677527 T3019 oasc.CoreContainer.<init> New CoreContainer 1291249394
   [junit4]   2> 677527 T3019 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BAC342B198BF0FEB-001/tempDir-004/]
   [junit4]   2> 677528 T3019 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 677528 T3019 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 677529 T3019 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 677529 T3019 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 677529 T3019 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 677529 T3019 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 677529 T3019 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 677530 T3019 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 677530 T3019 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 677530 T3019 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 677532 T3019 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=3400000&connTimeout=45000&retry=false
   [junit4]   2> 677534 T3019 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 677534 T3019 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 677534 T3019 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 677534 T3019 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:57521/solr
   [junit4]   2> 677535 T3019 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 677535 T3019 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 677536 T3019 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 677537 T3083 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@39cb5470 name:ZooKeeperConnection Watcher:127.0.0.1:57521 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 677537 T3019 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 677538 T3019 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 677539 T3019 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 677540 T3085 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4d21be23 name:ZooKeeperConnection Watcher:127.0.0.1:57521/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 677540 T3019 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 677544 T3019 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 678549 T3019 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:13927_
   [junit4]   2> 678550 T3019 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:13927_
   [junit4]   2> 678553 T3019 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 678556 T3019 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J3
   [junit4]   2> 678557 T3019 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 679966 T3019 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 679966 T3019 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 679970 T3019 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 679972 T3019 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:38126
   [junit4]   2> 679974 T3019 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 679974 T3019 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 679974 T3019 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BAC342B198BF0FEB-001/tempDir-005
   [junit4]   2> 679975 T3019 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BAC342B198BF0FEB-001/tempDir-005/'
   [junit4]   2> 680003 T3019 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BAC342B198BF0FEB-001/tempDir-005/solr.xml
   [junit4]   2> 680055 T3019 oasc.CoreContainer.<init> New CoreContainer 483982703
   [junit4]   2> 680055 T3019 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BAC342B198BF0FEB-001/tempDir-005/]
   [junit4]   2> 680056 T3019 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 680056 T3019 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 680057 T3019 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 680057 T3019 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 680057 T3019 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 680057 T3019 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 680057 T3019 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 680058 T3019 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 680058 T3019 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 680058 T3019 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 680060 T3019 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=3400000&connTimeout=45000&retry=false
   [junit4]   2> 680062 T3019 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 680062 T3019 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 680062 T3019 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 680062 T3019 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:57521/solr
   [junit4]   2> 680063 T3019 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 680063 T3019 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 680064 T3019 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 680078 T3098 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@50f8866d name:ZooKeeperConnection Watcher:127.0.0.1:57521 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 680079 T3019 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 680079 T3019 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 680080 T3019 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 680081 T3100 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6cbeb7b8 name:ZooKeeperConnection Watcher:127.0.0.1:57521/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 680081 T3019 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 680086 T3019 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 681090 T3019 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38126_
   [junit4]   2> 681091 T3019 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:38126_
   [junit4]   2> 681095 T3019 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 681101 T3019 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J3
   [junit4]   2> 681101 T3019 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 682228 T3019 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 4
   [junit4]   2> 682229 T3019 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 682233 T3019 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 682235 T3019 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:19331
   [junit4]   2> 682237 T3019 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 682237 T3019 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 682237 T3019 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BAC342B198BF0FEB-001/tempDir-006
   [junit4]   2> 682237 T3019 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BAC342B198BF0FEB-001/tempDir-006/'
   [junit4]   2> 682270 T3019 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BAC342B198BF0FEB-001/tempDir-006/solr.xml
   [junit4]   2> 682327 T3019 oasc.CoreContainer.<init> New CoreContainer 462815002
   [junit4]   2> 682327 T3019 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BAC342B198BF0FEB-001/tempDir-006/]
   [junit4]   2> 682328 T3019 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 682328 T3019 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 682329 T3019 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 682329 T3019 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 682329 T3019 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 682329 T3019 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 682329 T3019 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 682330 T3019 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 682330 T3019 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 682330 T3019 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 682332 T3019 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=3400000&connTimeout=45000&retry=false
   [junit4]   2> 682334 T3019 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 682334 T3019 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 682334 T3019 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 682334 T3019 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:57521/solr
   [junit4]   2> 682335 T3019 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 682335 T3019 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 682336 T3019 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 682337 T3113 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@63387b2 name:ZooKeeperConnection Watcher:127.0.0.1:57521 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 682337 T3019 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 682338 T3019 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 682339 T3019 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 682340 T3115 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@29559c74 name:ZooKeeperConnection Watcher:127.0.0.1:57521/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 682340 T3019 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 682345 T3019 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 683350 T3019 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:19331_
   [junit4]   2> 683351 T3019 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:19331_
   [junit4]   2> 683355 T3019 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 683359 T3019 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J3
   [junit4]   2> 683360 T3019 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 683514 T3061 oasha.CollectionsHandler.handleCreateAction Creating Collection : name=solrj_collection&replicationFactor=2&action=CREATE&collection.configName=conf1&numShards=2&wt=javabin&router.field=myOwnField&version=2
   [junit4]   2> 683516 T3056 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 683517 T3049 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"create",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"solrj_collection",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "collection.configName":"conf1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "router.field":"myOwnField"}
   [junit4]   2> 683524 T3119 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : create , {
   [junit4]   2> 	  "operation":"create",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"solrj_collection",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "collection.configName":"conf1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "router.field":"myOwnField"}
   [junit4]   2> 683529 T3119 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/solrj_collection 
   [junit4]   2> 683530 T3119 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection
   [junit4]   2> 683533 T3056 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 683536 T3048 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: solrj_collection
   [junit4]   2> 683536 T3048 oasc.Overseer$ClusterStateUpdater.createCollection Create collection solrj_collection with shards [shard1, shard2]
   [junit4]   2> 683536 T3048 oasc.Overseer$ClusterStateUpdater.createCollection state version solrj_collection 1
   [junit4]   2> 683538 T3086 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 683538 T3116 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 683538 T3101 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 683538 T3056 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 683538 T3057 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 683538 T3071 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 683633 T3119 oasc.OverseerCollectionProcessor.createCollection Creating SolrCores for new collection solrj_collection, shardNames [shard1, shard2] , replicationFactor : 2
   [junit4]   2> 683634 T3119 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard1_replica1 as part of slice shard1 of collection solrj_collection on 127.0.0.1:13927_
   [junit4]   2> 683635 T3119 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard1_replica2 as part of slice shard1 of collection solrj_collection on 127.0.0.1:38126_
   [junit4]   2> 683637 T3119 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard2_replica1 as part of slice shard2 of collection solrj_collection on 127.0.0.1:10701_
   [junit4]   2> 683639 T3119 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard2_replica2 as part of slice shard2 of collection solrj_collection on 127.0.0.1:33847_
   [junit4]   2> 683762 T3040 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&shard=shard2&name=solrj_collection_shard2_replica2&action=CREATE&collection.configName=conf1&numShards=2&wt=javabin&collection=solrj_collection&version=2
   [junit4]   2> 683765 T3075 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&shard=shard1&name=solrj_collection_shard1_replica1&action=CREATE&collection.configName=conf1&numShards=2&wt=javabin&collection=solrj_collection&version=2
   [junit4]   2> 683766 T3065 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&shard=shard2&name=solrj_collection_shard2_replica1&action=CREATE&collection.configName=conf1&numShards=2&wt=javabin&collection=solrj_collection&version=2
   [junit4]   2> 683767 T3090 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&shard=shard1&name=solrj_collection_shard1_replica2&action=CREATE&collection.configName=conf1&numShards=2&wt=javabin&collection=solrj_collection&version=2
   [junit4]   2> 683773 T3065 oasc.ZkController.publish publishing core=solrj_collection_shard2_replica1 state=down collection=solrj_collection
   [junit4]   2> 683773 T3090 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica2 state=down collection=solrj_collection
   [junit4]   2> 683773 T3040 oasc.ZkController.publish publishing core=solrj_collection_shard2_replica2 state=down collection=solrj_collection
   [junit4]   2> 683773 T3075 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica1 state=down collection=solrj_collection
   [junit4]   2> 683774 T3056 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 683774 T3090 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 683774 T3065 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 683774 T3075 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 683774 T3040 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 683777 T3048 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:10701_",
   [junit4]   2> 	  "core":"solrj_collection_shard2_replica1",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:10701"}
   [junit4]   2> 683779 T3048 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:38126_",
   [junit4]   2> 	  "core":"solrj_collection_shard1_replica2",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:38126"}
   [junit4]   2> 683781 T3056 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 683782 T3048 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:33847_",
   [junit4]   2> 	  "core":"solrj_collection_shard2_replica2",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:33847"}
   [junit4]   2> 683784 T3048 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:13927_",
   [junit4]   2> 	  "core":"solrj_collection_shard1_replica1",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:13927"}
   [junit4]   2> 683786 T3056 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 683786 T3056 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 683787 T3071 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 683787 T3057 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 683787 T3086 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 683787 T3116 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 683787 T3101 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 684774 T3090 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard1_replica2
   [junit4]   2> 684775 T3090 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 684775 T3065 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard2_replica1
   [junit4]   2> 684776 T3090 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 684775 T3040 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard2_replica2
   [junit4]   2> 684776 T3090 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 684776 T3075 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard1_replica1
   [junit4]   2> 684776 T3065 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 684777 T3075 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 684776 T3040 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 684778 T3065 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 684778 T3090 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 684778 T3040 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 684778 T3065 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 684778 T3075 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 684779 T3040 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 684780 T3065 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 684779 T3090 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BAC342B198BF0FEB-001/tempDir-005/solrj_collection_shard1_replica2/'
   [junit4]   2> 684780 T3065 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BAC342B198BF0FEB-001/tempDir-003/solrj_collection_shard2_replica1/'
   [junit4]   2> 684780 T3075 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 684781 T3040 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 684782 T3040 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BAC342B198BF0FEB-001/tempDir-002/solrj_collection_shard2_replica2/'
   [junit4]   2> 684782 T3075 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 684782 T3075 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BAC342B198BF0FEB-001/tempDir-004/solrj_collection_shard1_replica1/'
   [junit4]   2> 684838 T3090 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 684845 T3075 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 684847 T3065 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 684865 T3040 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 684910 T3090 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 684912 T3090 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 684916 T3065 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 684916 T3075 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 684917 T3065 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 684917 T3075 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 684925 T3090 oass.IndexSchema.readSchema [solrj_collection_shard1_replica2] Schema name=test
   [junit4]   2> 684926 T3040 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 684927 T3040 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 684928 T3075 oass.IndexSchema.readSchema [solrj_collection_shard1_replica1] Schema name=test
   [junit4]   2> 684929 T3065 oass.IndexSchema.readSchema [solrj_collection_shard2_replica1] Schema name=test
   [junit4]   2> 684940 T3040 oass.IndexSchema.readSchema [solrj_collection_shard2_replica2] Schema name=test
   [junit4]   2> 685397 T3065 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 685402 T3075 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 685402 T3090 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 685408 T3040 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 685408 T3065 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 685409 T3075 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 685411 T3090 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 685412 T3065 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 685413 T3075 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 685414 T3090 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 685415 T3040 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 685418 T3040 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 685427 T3065 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 685428 T3075 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 685429 T3090 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 685432 T3040 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 685433 T3065 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 685433 T3075 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 685434 T3090 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 685437 T3040 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 685438 T3065 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 685440 T3065 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 685440 T3090 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 685440 T3075 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 685440 T3065 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 685441 T3090 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 685442 T3075 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 685441 T3065 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 685442 T3075 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 685442 T3040 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 685443 T3065 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 685442 T3090 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 685444 T3040 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 685444 T3090 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 685443 T3065 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 685443 T3075 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 685445 T3090 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 685445 T3065 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard2_replica1' using configuration from collection solrj_collection
   [junit4]   2> 685444 T3040 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 685446 T3065 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 685446 T3075 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 685446 T3090 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 685447 T3075 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 685447 T3065 oasc.SolrCore.<init> [solrj_collection_shard2_replica1] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BAC342B198BF0FEB-001/tempDir-003/solrj_collection_shard2_replica1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BAC342B198BF0FEB-001/tempDir-003/solrj_collection_shard2_replica1/data/
   [junit4]   2> 685447 T3040 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 685449 T3065 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@244e4f54
   [junit4]   2> 685448 T3075 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard1_replica1' using configuration from collection solrj_collection
   [junit4]   2> 685448 T3090 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard1_replica2' using configuration from collection solrj_collection
   [junit4]   2> 685451 T3075 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 685450 T3040 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 685452 T3075 oasc.SolrCore.<init> [solrj_collection_shard1_replica1] Opening new SolrCore at ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BAC342B198BF0FEB-001/tempDir-004/solrj_collection_shard1_replica1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BAC342B198BF0FEB-001/tempDir-004/solrj_collection_shard1_replica1/data/
   [junit4]   2> 685451 T3065 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BAC342B198BF0FEB-001/tempDir-003/solrj_collection_shard2_replica1/data
   [junit4]   2> 685451 T3090 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 685453 T3065 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BAC342B198BF0FEB-001/tempDir-003/solrj_collection_shard2_replica1/data/index/
   [junit4]   2> 685452 T3075 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@244e4f54
   [junit4]   2> 685452 T3040 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 685453 T3065 oasc.SolrCore.initIndex WARN [solrj_collection_shard2_replica1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BAC342B198BF0FEB-001/tempDir-003/solrj_collection_shard2_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 685453 T3090 oasc.SolrCore.<init> [solrj_collection_shard1_replica2] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BAC342B198BF0FEB-001/tempDir-005/solrj_collection_shard1_replica2/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BAC342B198BF0FEB-001/tempDir-005/solrj_collection_shard1_replica2/data/
   [junit4]   2> 685455 T3065 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BAC342B198BF0FEB-001/tempDir-003/solrj_collection_shard2_replica1/data/index
   [junit4]   2> 685455 T3090 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@244e4f54
   [junit4]   2> 685454 T3075 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BAC342B198BF0FEB-001/tempDir-004/solrj_collection_shard1_replica1/data
   [junit4]   2> 685454 T3040 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard2_replica2' using configuration from collection solrj_collection
   [junit4]   2> 685456 T3075 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BAC342B198BF0FEB-001/tempDir-004/solrj_collection_shard1_replica1/data/index/
   [junit4]   2> 685455 T3065 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=29, maxMergeAtOnceExplicit=17, maxMergedSegmentMB=24.2255859375, floorSegmentMB=2.037109375, forceMergeDeletesPctAllowed=7.414876246236076, segmentsPerTier=39.0, maxCFSSegmentSizeMB=2.01171875, noCFSRatio=0.6164819115401692
   [junit4]   2> 685457 T3075 oasc.SolrCore.initIndex WARN [solrj_collection_shard1_replica1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BAC342B198BF0FEB-001/tempDir-004/solrj_collection_shard1_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 685456 T3090 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BAC342B198BF0FEB-001/tempDir-005/solrj_collection_shard1_replica2/data
   [junit4]   2> 685458 T3075 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BAC342B198BF0FEB-001/tempDir-004/solrj_collection_shard1_replica1/data/index
   [junit4]   2> 685456 T3040 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 685458 T3075 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=29, maxMergeAtOnceExplicit=17, maxMergedSegmentMB=24.2255859375, floorSegmentMB=2.037109375, forceMergeDeletesPctAllowed=7.414876246236076, segmentsPerTier=39.0, maxCFSSegmentSizeMB=2.01171875, noCFSRatio=0.6164819115401692
   [junit4]   2> 685458 T3090 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BAC342B198BF0FEB-001/tempDir-005/solrj_collection_shard1_replica2/data/index/
   [junit4]   2> 685458 T3065 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(NRTCachingDirectory(RAMDirectory(id=lucene-29aba046); maxCacheMB=0.8798828125 maxMergeSizeMB=0.60546875)),segFN=segments_1,generation=1}
   [junit4]   2> 685460 T3075 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(NRTCachingDirectory(RAMDirectory(id=lucene-2d184736); maxCacheMB=0.8798828125 maxMergeSizeMB=0.60546875)),segFN=segments_1,generation=1}
   [junit4]   2> 685460 T3065 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 685459 T3090 oasc.SolrCore.initIndex WARN [solrj_collection_shard1_replica2] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BAC342B198BF0FEB-001/tempDir-005/solrj_collection_shard1_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 685459 T3040 oasc.SolrCore.<init> [solrj_collection_shard2_replica2] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BAC342B198BF0FEB-001/tempDir-002/solrj_collection_shard2_replica2/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BAC342B198BF0FEB-001/tempDir-002/solrj_collection_shard2_replica2/data/
   [junit4]   2> 685461 T3090 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BAC342B198BF0FEB-001/tempDir-005/solrj_collection_shard1_replica2/data/index
   [junit4]   2> 685460 T3075 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 685462 T3090 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=29, maxMergeAtOnceExplicit=17, maxMergedSegmentMB=24.2255859375, floorSegmentMB=2.037109375, forceMergeDeletesPctAllowed=7.414876246236076, segmentsPerTier=39.0, maxCFSSegmentSizeMB=2.01171875, noCFSRatio=0.6164819115401692
   [junit4]   2> 685462 T3040 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@244e4f54
   [junit4]   2> 685464 T3090 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(NRTCachingDirectory(RAMDirectory(id=lucene-4bf57fb); maxCacheMB=0.8798828125 maxMergeSizeMB=0.60546875)),segFN=segments_1,generation=1}
   [junit4]   2> 685464 T3090 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 685464 T3040 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BAC342B198BF0FEB-001/tempDir-002/solrj_collection_shard2_replica2/data
   [junit4]   2> 685465 T3040 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BAC342B198BF0FEB-001/tempDir-002/solrj_collection_shard2_replica2/data/index/
   [junit4]   2> 685466 T3065 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 685466 T3040 oasc.SolrCore.initIndex WARN [solrj_collection_shard2_replica2] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BAC342B198BF0FEB-001/tempDir-002/solrj_collection_shard2_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 685466 T3065 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 685467 T3040 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BAC342B198BF0FEB-001/tempDir-002/solrj_collection_shard2_replica2/data/index
   [junit4]   2> 685467 T3065 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 685468 T3040 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=29, maxMergeAtOnceExplicit=17, maxMergedSegmentMB=24.2255859375, floorSegmentMB=2.037109375, forceMergeDeletesPctAllowed=7.414876246236076, segmentsPerTier=39.0, maxCFSSegmentSizeMB=2.01171875, noCFSRatio=0.6164819115401692
   [junit4]   2> 685468 T3065 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 685469 T3075 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 685469 T3065 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 685469 T3075 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 685469 T3040 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(NRTCachingDirectory(RAMDirectory(id=lucene-79295b9b); maxCacheMB=0.8798828125 maxMergeSizeMB=0.60546875)),segFN=segments_1,generation=1}
   [junit4]   2> 685470 T3075 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 685470 T3090 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 685470 T3065 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 685471 T3090 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 685472 T3065 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 685471 T3075 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 685470 T3040 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 685472 T3075 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 685472 T3065 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 685472 T3090 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 685473 T3065 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 685473 T3075 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 685474 T3090 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 685475 T3075 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 685475 T3065 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 685476 T3075 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 685475 T3090 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 685476 T3075 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 685476 T3065 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 685477 T3090 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 685478 T3065 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 685478 T3075 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 685478 T3065 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 685478 T3090 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 685479 T3065 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 685478 T3040 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 685479 T3065 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 685479 T3090 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 685479 T3075 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 685480 T3090 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 685480 T3065 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 685480 T3040 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 685481 T3065 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 685481 T3090 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 685481 T3075 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 685482 T3065 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 685482 T3040 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 685483 T3065 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 685483 T3075 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 685484 T3065 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 685483 T3090 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 685484 T3075 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 685483 T3040 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 685485 T3075 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 685485 T3090 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 685486 T3075 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 685485 T3040 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 685486 T3075 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 685486 T3090 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 685487 T3075 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 685486 T3040 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 685487 T3090 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 685488 T3075 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 685488 T3040 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 685488 T3090 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 685489 T3040 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 685489 T3075 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 685489 T3040 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 685489 T3090 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 685490 T3040 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 685491 T3090 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 685491 T3040 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 685491 T3090 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 685492 T3040 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 685492 T3090 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 685492 T3040 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 685493 T3090 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 685493 T3040 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 685494 T3040 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 685494 T3040 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 685495 T3040 oasc.RequestHand

[...truncated too long message...]

ard5/election/92705533017784336-core_node15-n_0000000000
   [junit4]   2> 		2	/solr/collections/awholynewstresscollection_collection1_2/leader_elect/shard6/election/92705533017784330-core_node16-n_0000000000
   [junit4]   2> 		2	/solr/collections/awholynewstresscollection_collection2_2/leader_elect/shard2/election/92705533017784341-core_node5-n_0000000000
   [junit4]   2> 		2	/solr/collections/awholynewstresscollection_collection4_2/leader_elect/shard3/election/92705533017784334-core_node9-n_0000000000
   [junit4]   2> 		2	/solr/collections/awholynewstresscollection_collection3_2/leader_elect/shard4/election/92705533017784334-core_node10-n_0000000000
   [junit4]   2> 		2	/solr/collections/halfdeletedcollection2/leader_elect/shard3/election/92705533017784334-core_node8-n_0000000000
   [junit4]   2> 		2	/solr/collections/awholynewstresscollection_collection2_2/leader_elect/shard3/election/92705533017784341-core_node10-n_0000000000
   [junit4]   2> 		2	/solr/collections/awholynewstresscollection_collection0_2/leader_elect/shard5/election/92705533017784336-core_node9-n_0000000000
   [junit4]   2> 		2	/solr/collections/awholynewstresscollection_collection2_2/leader_elect/shard1/election/92705533017784336-core_node1-n_0000000000
   [junit4]   2> 	
   [junit4]   2> 	Maximum concurrent children watches above limit:
   [junit4]   2> 	
   [junit4]   2> 		16	/solr/live_nodes
   [junit4]   2> 		4	/solr/overseer/collection-queue-work
   [junit4]   2> 		3	/solr/overseer/queue
   [junit4]   2> 	
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=CollectionsAPIDistributedZkTest -Dtests.method=testDistribSearch -Dtests.seed=BAC342B198BF0FEB -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=ar_JO -Dtests.timezone=America/Danmarkshavn -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR    588s J3 | CollectionsAPIDistributedZkTest.testDistribSearch <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: There are still nodes recoverying - waited for 330 seconds
   [junit4]    > 	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:178)
   [junit4]    > 	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:137)
   [junit4]    > 	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:132)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:834)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.deleteCollectionWithDownNodes(CollectionsAPIDistributedZkTest.java:430)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:212)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)Throwable #2: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=3850, name=Thread-1223, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    > Caused by: java.lang.NullPointerException
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([BAC342B198BF0FEB]:0)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:1044)Throwable #3: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=3852, name=Thread-1225, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    > Caused by: java.lang.NullPointerException
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([BAC342B198BF0FEB]:0)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:1044)Throwable #4: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=3851, name=Thread-1224, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    > Caused by: java.lang.NullPointerException
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([BAC342B198BF0FEB]:0)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:1044)Throwable #5: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=3848, name=Thread-1221, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    > Caused by: java.lang.NullPointerException
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([BAC342B198BF0FEB]:0)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:1044)Throwable #6: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=3847, name=Thread-1220, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    > Caused by: java.lang.NullPointerException
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([BAC342B198BF0FEB]:0)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:1044)Throwable #7: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=3849, name=Thread-1222, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    > Caused by: java.lang.NullPointerException
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([BAC342B198BF0FEB]:0)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:1044)
   [junit4]   2> 1259691 T3019 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BAC342B198BF0FEB-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50): {timestamp=PostingsFormat(name=Memory doPackFST= false), intDefault=PostingsFormat(name=Memory doPackFST= false), text=PostingsFormat(name=LuceneVarGapDocFreqInterval), range_facet_l=PostingsFormat(name=Memory doPackFST= false), _version_=PostingsFormat(name=Memory doPackFST= false), a_t=PostingsFormat(name=Memory doPackFST= false), a_i1=PostingsFormat(name=Memory doPackFST= false), other_tl1=PostingsFormat(name=Memory doPackFST= false), id=PostingsFormat(name=Memory doPackFST= true), multiDefault=PostingsFormat(name=LuceneFixedGap)}, docValues:{timestamp=DocValuesFormat(name=SimpleText)}, sim=DefaultSimilarity, locale=ar_JO, timezone=America/Danmarkshavn
   [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_65 (64-bit)/cpus=16,threads=1,free=149281680,total=508559360
   [junit4]   2> NOTE: All tests run in this JVM: [UniqFieldsUpdateProcessorFactoryTest, StatelessScriptUpdateProcessorFactoryTest, LeaderElectionTest, StandardRequestHandlerTest, TestRawResponseWriter, SpellPossibilityIteratorTest, BadCopyFieldTest, DocValuesMultiTest, DocumentAnalysisRequestHandlerTest, SearchHandlerTest, ZkCLITest, EchoParamsTest, TestSchemaManager, TestMissingGroups, TestLMJelinekMercerSimilarityFactory, TestIndexSearcher, TestJoin, DocValuesMissingTest, FieldMutatingUpdateProcessorTest, TestNRTOpen, TestReRankQParserPlugin, TestManagedSchemaFieldResource, TestSearchPerf, TriLevelCompositeIdRoutingTest, CoreContainerCoreInitFailuresTest, SliceStateTest, SuggesterTSTTest, TestCloudSchemaless, LoggingHandlerTest, TestManagedSchema, FieldAnalysisRequestHandlerTest, TestCSVLoader, TestOrdValues, SuggesterTest, SimpleFacetsTest, UUIDFieldTest, TestMergePolicyConfig, TestPivotHelperCode, CollectionsAPIDistributedZkTest]
   [junit4] Completed on J3 in 587.82s, 1 test, 1 error <<< FAILURES!

[...truncated 929 lines...]
BUILD FAILED
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/build.xml:532: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/build.xml:473: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/build.xml:61: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/extra-targets.xml:39: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/build.xml:189: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/solr/common-build.xml:496: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/lucene/common-build.xml:1358: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk@2/lucene/common-build.xml:965: There were test failures: 440 suites, 1800 tests, 1 error, 54 ignored (9 assumptions)

Total time: 215 minutes 25 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-NightlyTests-trunk #664
Archived 5 artifacts
Archive block size is 32768
Received 0 blocks and 433414723 bytes
Compression is 0.0%
Took 1 min 15 sec
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure