You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2014/09/24 02:59:54 UTC

[JENKINS] Lucene-Solr-NightlyTests-5.x - Build # 630 - Still Failing

Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-5.x/630/

3 tests failed.
REGRESSION:  org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch

Error Message:
The Monkey ran for over 20 seconds and no jetties were stopped - this is worth investigating!

Stack Trace:
java.lang.AssertionError: The Monkey ran for over 20 seconds and no jetties were stopped - this is worth investigating!
	at __randomizedtesting.SeedInfo.seed([D2E2470F5513A1F0:5304C917224CC1CC]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.ChaosMonkey.stopTheMonkey(ChaosMonkey.java:535)
	at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:140)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderTest

Error Message:
2 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest:     1) Thread[id=138, name=StopableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.StopableIndexingThread.run(StopableIndexingThread.java:121)    2) Thread[id=137, name=StopableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.StopableIndexingThread.run(StopableIndexingThread.java:121)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest: 
   1) Thread[id=138, name=StopableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.solr.cloud.StopableIndexingThread.run(StopableIndexingThread.java:121)
   2) Thread[id=137, name=StopableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.solr.cloud.StopableIndexingThread.run(StopableIndexingThread.java:121)
	at __randomizedtesting.SeedInfo.seed([D2E2470F5513A1F0]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=137, name=StopableIndexingThread, state=BLOCKED, group=TGRP-ChaosMonkeySafeLeaderTest]         at java.io.PrintStream.println(PrintStream.java:806)         at org.apache.solr.cloud.StopableIndexingThread.run(StopableIndexingThread.java:107)    2) Thread[id=138, name=StopableIndexingThread, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]         at java.lang.Class.getEnclosingMethod0(Native Method)         at java.lang.Class.getEnclosingMethodInfo(Class.java:959)         at java.lang.Class.getEnclosingClass(Class.java:1132)         at java.lang.Class.getSimpleBinaryName(Class.java:1277)         at java.lang.Class.getSimpleName(Class.java:1169)         at java.lang.Class.isAnonymousClass(Class.java:1245)         at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.AnonymousAndLocalClassExclusionStrategy.isAnonymousOrLocal(AnonymousAndLocalClassExclusionStrategy.java:35)         at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.AnonymousAndLocalClassExclusionStrategy.shouldSkipClass(AnonymousAndLocalClassExclusionStrategy.java:31)         at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.DisjunctionExclusionStrategy.shouldSkipClass(DisjunctionExclusionStrategy.java:47)         at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.internal.bind.ExcludedTypeAdapterFactory.create(ExcludedTypeAdapterFactory.java:41)         at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.internal.bind.MiniGson.getAdapter(MiniGson.java:92)         at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.Gson.toJson(Gson.java:504)         at com.carrotsearch.ant.tasks.junit4.events.Serializer.serialize(Serializer.java:87)         at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain$4.write(SlaveMain.java:410)         at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)         at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)         at java.io.PrintStream.flush(PrintStream.java:338)         at java.io.FilterOutputStream.flush(FilterOutputStream.java:140)         at java.io.PrintStream.write(PrintStream.java:482)         at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)         at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)         at sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:104)         at java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:185)         at java.io.PrintStream.newLine(PrintStream.java:546)         at java.io.PrintStream.println(PrintStream.java:824)         at java.lang.Throwable$WrappedPrintStream.println(Throwable.java:748)         at java.lang.Throwable.printStackTrace(Throwable.java:658)         at java.lang.Throwable.printStackTrace(Throwable.java:643)         at java.lang.Throwable.printStackTrace(Throwable.java:634)         at org.apache.solr.cloud.StopableIndexingThread.run(StopableIndexingThread.java:108)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=137, name=StopableIndexingThread, state=BLOCKED, group=TGRP-ChaosMonkeySafeLeaderTest]
        at java.io.PrintStream.println(PrintStream.java:806)
        at org.apache.solr.cloud.StopableIndexingThread.run(StopableIndexingThread.java:107)
   2) Thread[id=138, name=StopableIndexingThread, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
        at java.lang.Class.getEnclosingMethod0(Native Method)
        at java.lang.Class.getEnclosingMethodInfo(Class.java:959)
        at java.lang.Class.getEnclosingClass(Class.java:1132)
        at java.lang.Class.getSimpleBinaryName(Class.java:1277)
        at java.lang.Class.getSimpleName(Class.java:1169)
        at java.lang.Class.isAnonymousClass(Class.java:1245)
        at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.AnonymousAndLocalClassExclusionStrategy.isAnonymousOrLocal(AnonymousAndLocalClassExclusionStrategy.java:35)
        at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.AnonymousAndLocalClassExclusionStrategy.shouldSkipClass(AnonymousAndLocalClassExclusionStrategy.java:31)
        at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.DisjunctionExclusionStrategy.shouldSkipClass(DisjunctionExclusionStrategy.java:47)
        at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.internal.bind.ExcludedTypeAdapterFactory.create(ExcludedTypeAdapterFactory.java:41)
        at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.internal.bind.MiniGson.getAdapter(MiniGson.java:92)
        at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.Gson.toJson(Gson.java:504)
        at com.carrotsearch.ant.tasks.junit4.events.Serializer.serialize(Serializer.java:87)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain$4.write(SlaveMain.java:410)
        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
        at java.io.PrintStream.flush(PrintStream.java:338)
        at java.io.FilterOutputStream.flush(FilterOutputStream.java:140)
        at java.io.PrintStream.write(PrintStream.java:482)
        at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
        at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)
        at sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:104)
        at java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:185)
        at java.io.PrintStream.newLine(PrintStream.java:546)
        at java.io.PrintStream.println(PrintStream.java:824)
        at java.lang.Throwable$WrappedPrintStream.println(Throwable.java:748)
        at java.lang.Throwable.printStackTrace(Throwable.java:658)
        at java.lang.Throwable.printStackTrace(Throwable.java:643)
        at java.lang.Throwable.printStackTrace(Throwable.java:634)
        at org.apache.solr.cloud.StopableIndexingThread.run(StopableIndexingThread.java:108)
	at __randomizedtesting.SeedInfo.seed([D2E2470F5513A1F0]:0)




Build Log:
[...truncated 11404 lines...]
   [junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
   [junit4]   2> Creating dataDir: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-D2E2470F5513A1F0-001/init-core-data-001
   [junit4]   2> 25078 T18 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (true)
   [junit4]   2> 25079 T18 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 25124 T18 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 25132 T18 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 25141 T19 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 25441 T18 oasc.ZkTestServer.run start zk server on port:19165
   [junit4]   2> 25452 T18 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 25767 T18 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 25825 T20 oazs.NIOServerCnxn.doIO WARN Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
   [junit4]   2> 27660 T25 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4eb3a43d name:ZooKeeperConnection Watcher:127.0.0.1:19165 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 27661 T18 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 27662 T18 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 27677 T18 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 27827 T18 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 27829 T18 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 27830 T27 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7079f82c name:ZooKeeperConnection Watcher:127.0.0.1:19165/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 27831 T18 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 27832 T18 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 27840 T18 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 27845 T18 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 27848 T18 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 27851 T18 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 27855 T18 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 27899 T18 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 27905 T18 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 27906 T18 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 27910 T18 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 27911 T18 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 27915 T18 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 27916 T18 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 27920 T18 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 27921 T18 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 27925 T18 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 27926 T18 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 27930 T18 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 27932 T18 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 27937 T18 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 27938 T18 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 27942 T18 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 27943 T18 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 27947 T18 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 27948 T18 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 27952 T18 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 27952 T18 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 31034 T18 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 31764 T18 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:18175
   [junit4]   2> 31971 T18 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 31972 T18 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 31973 T18 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-D2E2470F5513A1F0-001/tempDir-002
   [junit4]   2> 31974 T18 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-D2E2470F5513A1F0-001/tempDir-002/'
   [junit4]   2> 32032 T18 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-D2E2470F5513A1F0-001/tempDir-002/solr.xml
   [junit4]   2> 32155 T18 oasc.CoreContainer.<init> New CoreContainer 736133528
   [junit4]   2> 32156 T18 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-D2E2470F5513A1F0-001/tempDir-002/]
   [junit4]   2> 32280 T18 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 32281 T18 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 32281 T18 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 32282 T18 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 32283 T18 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 32284 T18 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 32284 T18 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 32285 T18 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 32286 T18 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 32287 T18 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 32288 T18 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 32289 T18 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 32290 T18 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 32290 T18 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:19165/solr
   [junit4]   2> 32356 T18 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 32357 T18 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 32358 T18 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 32361 T39 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@57c85d64 name:ZooKeeperConnection Watcher:127.0.0.1:19165 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 32362 T18 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 32363 T18 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 32391 T18 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 32394 T41 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b41015d name:ZooKeeperConnection Watcher:127.0.0.1:19165/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 32395 T18 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 32425 T18 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 32449 T18 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 32467 T18 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 32473 T18 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 32478 T18 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 32515 T18 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 32518 T18 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:18175_
   [junit4]   2> 32531 T18 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:18175_
   [junit4]   2> 32538 T18 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 32542 T18 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 32546 T18 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 32581 T18 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:18175_
   [junit4]   2> 32582 T18 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 32585 T18 oasc.Overseer.start Overseer (id=92505213584211971-127.0.0.1:18175_-n_0000000000) starting
   [junit4]   2> 32618 T18 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 32667 T18 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 33358 T43 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 33359 T18 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 33363 T18 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 33367 T18 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 33443 T42 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 33464 T45 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 33464 T45 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 33491 T45 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 33501 T46 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 33552 T42 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:18175",
   [junit4]   2> 	  "node_name":"127.0.0.1:18175_",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "shard":null}
   [junit4]   2> 33553 T42 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 33601 T42 oasc.Overseer$ClusterStateUpdater.createCollection state version control_collection 1
   [junit4]   2> 33628 T42 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 33659 T46 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 34492 T45 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 34492 T45 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 34493 T45 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 34494 T45 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 34495 T45 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 34496 T45 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-D2E2470F5513A1F0-001/tempDir-002/collection1/'
   [junit4]   2> 34498 T45 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-D2E2470F5513A1F0-001/tempDir-002/collection1/lib/README' to classloader
   [junit4]   2> 34499 T45 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-D2E2470F5513A1F0-001/tempDir-002/collection1/lib/.svn/' to classloader
   [junit4]   2> 34500 T45 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-D2E2470F5513A1F0-001/tempDir-002/collection1/lib/classes/' to classloader
   [junit4]   2> 34674 T45 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 34728 T45 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 34731 T45 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 34799 T45 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 35457 T45 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 35460 T45 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 35463 T45 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 35491 T45 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 35549 T45 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 35549 T45 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 35628 T45 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-D2E2470F5513A1F0-001/tempDir-002/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-D2E2470F5513A1F0-001/tempDir-001/control/data/
   [junit4]   2> 35629 T45 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2127b2ad
   [junit4]   2> 35645 T45 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-D2E2470F5513A1F0-001/tempDir-001/control/data
   [junit4]   2> 35646 T45 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-D2E2470F5513A1F0-001/tempDir-001/control/data/index/
   [junit4]   2> 35647 T45 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-D2E2470F5513A1F0-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 35648 T45 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-D2E2470F5513A1F0-001/tempDir-001/control/data/index
   [junit4]   2> 35649 T45 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=34, maxMergeAtOnceExplicit=38, maxMergedSegmentMB=34.2861328125, floorSegmentMB=1.2841796875, forceMergeDeletesPctAllowed=20.677853650720508, segmentsPerTier=4.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6436012966525844
   [junit4]   2> 35655 T45 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-D2E2470F5513A1F0-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 35656 T45 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 35666 T45 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 35801 T45 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 35802 T45 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 35803 T45 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 35803 T45 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 35804 T45 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 35804 T45 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 35805 T45 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 35805 T45 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 35813 T45 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 35814 T45 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 35816 T45 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 35817 T45 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 35819 T45 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 35819 T45 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 35820 T45 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 35821 T45 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 35822 T45 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 35823 T45 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 35825 T45 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 35840 T45 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 35842 T45 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 35844 T45 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 35846 T45 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 35853 T45 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 35854 T45 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 35857 T45 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=5, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 35859 T45 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-D2E2470F5513A1F0-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 35859 T45 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 35860 T45 oass.SolrIndexSearcher.<init> Opening Searcher@5814ef1c[collection1] main
   [junit4]   2> 35861 T45 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 35863 T45 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 35863 T45 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 35872 T45 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 35873 T45 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 35873 T45 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 35875 T45 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 35875 T45 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 35877 T45 oascc.SolrZkClient.makePath makePath: /configs/conf1/_rest_managed.json
   [junit4]   2> 35880 T45 oasr.ManagedResourceStorage$ZooKeeperStorageIO$1.close Wrote 38 bytes to new znode /configs/conf1/_rest_managed.json
   [junit4]   2> 35881 T45 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 35881 T45 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 35892 T47 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5814ef1c[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> 35905 T45 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 35907 T50 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:18175 collection:control_collection shard:shard1
   [junit4]   2> 35908 T18 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1
   [junit4]   2> 35908 T18 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 35936 T50 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 35944 T50 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 35946 T46 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 35947 T50 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 35948 T50 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C1 name=collection1 org.apache.solr.core.SolrCore@7750c73c url=http://127.0.0.1:18175/collection1 node=127.0.0.1:18175_ C1_STATE=coll:control_collection core:collection1 props:{base_url=http://127.0.0.1:18175, node_name=127.0.0.1:18175_, state=down, core=collection1}
   [junit4]   2> 35958 T50 C1 P18175 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:18175/collection1/
   [junit4]   2> 35959 T50 C1 P18175 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 35959 T50 C1 P18175 oasc.SyncStrategy.syncToMe http://127.0.0.1:18175/collection1/ has no replicas
   [junit4]   2> 35960 T50 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:18175/collection1/ shard1
   [junit4]   2> 35962 T46 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 35971 T50 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 35978 T46 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 35997 T18 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 35998 T18 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 36000 T52 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@50e5a757 name:ZooKeeperConnection Watcher:127.0.0.1:19165/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 36000 T18 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 36001 T18 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 36003 T18 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 36011 T18 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 36088 T46 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 36088 T53 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 36131 T50 oasc.ZkController.register We are http://127.0.0.1:18175/collection1/ and leader is http://127.0.0.1:18175/collection1/
   [junit4]   2> 36132 T50 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:18175
   [junit4]   2> 36132 T50 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 36133 T50 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 36133 T50 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 36136 T46 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 36143 T42 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:18175",
   [junit4]   2> 	  "node_name":"127.0.0.1:18175_",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "shard":"shard1"}
   [junit4]   2> 36250 T46 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 36250 T53 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 37275 T18 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 37282 T18 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 37285 T18 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:43926
   [junit4]   2> 37285 T18 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 37286 T18 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 37286 T18 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-D2E2470F5513A1F0-001/tempDir-003
   [junit4]   2> 37287 T18 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-D2E2470F5513A1F0-001/tempDir-003/'
   [junit4]   2> 37335 T18 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-D2E2470F5513A1F0-001/tempDir-003/solr.xml
   [junit4]   2> 37432 T18 oasc.CoreContainer.<init> New CoreContainer 1719203423
   [junit4]   2> 37432 T18 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-D2E2470F5513A1F0-001/tempDir-003/]
   [junit4]   2> 37434 T18 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 37434 T18 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 37435 T18 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 37435 T18 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 37436 T18 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 37436 T18 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 37437 T18 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 37437 T18 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 37437 T18 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 37438 T18 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 37439 T18 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 37439 T18 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 37440 T18 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 37440 T18 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:19165/solr
   [junit4]   2> 37441 T18 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 37441 T18 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 37442 T18 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 37444 T64 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f770b88 name:ZooKeeperConnection Watcher:127.0.0.1:19165 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 37445 T18 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 37446 T18 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 37449 T18 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 37451 T66 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@372318b7 name:ZooKeeperConnection Watcher:127.0.0.1:19165/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 37451 T18 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 37464 T18 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 38472 T18 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43926_
   [junit4]   2> 38474 T18 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:43926_
   [junit4]   2> 38480 T18 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 38496 T68 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 38496 T68 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 38497 T46 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 38498 T68 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 38503 T42 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:43926",
   [junit4]   2> 	  "node_name":"127.0.0.1:43926_",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "shard":null}
   [junit4]   2> 38503 T42 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3, shard4]
   [junit4]   2> 38504 T42 oasc.Overseer$ClusterStateUpdater.createCollection state version collection1 1
   [junit4]   2> 38504 T42 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 38510 T46 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 38510 T67 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 38510 T53 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 39498 T68 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 39499 T68 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 39500 T68 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 39501 T68 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 39503 T68 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 39503 T68 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-D2E2470F5513A1F0-001/tempDir-003/collection1/'
   [junit4]   2> 39506 T68 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-D2E2470F5513A1F0-001/tempDir-003/collection1/lib/classes/' to classloader
   [junit4]   2> 39507 T68 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-D2E2470F5513A1F0-001/tempDir-003/collection1/lib/README' to classloader
   [junit4]   2> 39509 T68 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-D2E2470F5513A1F0-001/tempDir-003/collection1/lib/.svn/' to classloader
   [junit4]   2> 39610 T68 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 39679 T68 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 39681 T68 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 39760 T68 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 40173 T68 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 40176 T68 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 40178 T68 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 40184 T68 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 40239 T68 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 40240 T68 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 40240 T68 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-D2E2470F5513A1F0-001/tempDir-003/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-D2E2470F5513A1F0-001/tempDir-001/jetty1/
   [junit4]   2> 40241 T68 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2127b2ad
   [junit4]   2> 40243 T68 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-D2E2470F5513A1F0-001/tempDir-001/jetty1
   [junit4]   2> 40243 T68 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-D2E2470F5513A1F0-001/tempDir-001/jetty1/index/
   [junit4]   2> 40244 T68 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-D2E2470F5513A1F0-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 40245 T68 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-D2E2470F5513A1F0-001/tempDir-001/jetty1/index
   [junit4]   2> 40246 T68 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=34, maxMergeAtOnceExplicit=38, maxMergedSegmentMB=34.2861328125, floorSegmentMB=1.2841796875, forceMergeDeletesPctAllowed=20.677853650720508, segmentsPerTier=4.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6436012966525844
   [junit4]   2> 40250 T68 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-D2E2470F5513A1F0-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 40250 T68 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 40261 T68 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 40261 T68 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 40262 T68 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 40262 T68 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 40262 T68 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 40263 T68 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 40264 T68 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 40264 T68 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 40265 T68 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 40267 T68 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 40268 T68 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 40270 T68 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 40271 T68 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 40273 T68 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 40273 T68 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 40273 T68 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 40275 T68 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 40276 T68 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 40277 T68 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 40279 T68 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 40293 T68 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 40294 T68 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 40296 T68 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 40298 T68 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 40301 T68 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 40302 T68 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 40304 T68 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=5, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 40306 T68 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-D2E2470F5513A1F0-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 40306 T68 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 40307 T68 oass.SolrIndexSearcher.<init> Opening Searcher@5b52e370[collection1] main
   [junit4]   2> 40307 T68 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 40309 T68 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 40310 T68 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 40311 T68 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 40311 T68 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 40311 T68 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 40313 T68 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
   [junit4]   2> 40314 T68 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 40314 T68 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
   [junit4]   2> 40314 T68 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 40327 T69 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5b52e370[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> 40339 T68 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 40340 T72 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:43926 collection:collection1 shard:shard2
   [junit4]   2> 40340 T18 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1
   [junit4]   2> 40341 T18 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 40342 T72 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 40350 T72 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 40353 T46 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 40354 T72 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 40354 T72 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C2 name=collection1 org.apache.solr.core.SolrCore@13f2c67b url=http://127.0.0.1:43926/collection1 node=127.0.0.1:43926_ C2_STATE=coll:collection1 core:collection1 props:{base_url=http://127.0.0.1:43926, node_name=127.0.0.1:43926_, state=down, core=collection1}
   [junit4]   2> 40355 T72 C2 P43926 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:43926/collection1/
   [junit4]   2> 40355 T72 C2 P43926 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 40356 T72 C2 P43926 oasc.SyncStrategy.syncToMe http://127.0.0.1:43926/collection1/ has no replicas
   [junit4]   2> 40356 T72 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:43926/collection1/ shard2
   [junit4]   2> 40357 T72 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 40364 T67 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 40364 T46 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 40364 T53 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 40378 T46 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 40380 T67 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 40380 T53 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 40380 T46 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 40417 T72 oasc.ZkController.register We are http://127.0.0.1:43926/collection1/ and leader is http://127.0.0.1:43926/collection1/
   [junit4]   2> 40418 T72 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:43926
   [junit4]   2> 40418 T72 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 40418 T72 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 40419 T72 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 40421 T46 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 40427 T42 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:43926",
   [junit4]   2> 	  "node_name":"127.0.0.1:43926_",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "shard":"shard2"}
   [junit4]   2> 40534 T46 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 40534 T67 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 40534 T53 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 41559 T18 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 41560 T18 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 41563 T18 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:43992
   [junit4]   2> 41564 T18 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 41564 T18 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 41565 T18 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-D2E2470F5513A1F0-001/tempDir-004
   [junit4]   2> 41565 T18 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-D2E2470F5513A1F0-001/tempDir-004/'
   [junit4]   2> 41596 T18 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-D2E2470F5513A1F0-001/tempDir-004/solr.xml
   [junit4]   2> 41660 T18 oasc.CoreContainer.<init> New CoreContainer 1507387457
   [junit4]   2> 41661 T18 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-D2E2470F5513A1F0-001/tempDir-004/]
   [junit4]   2> 41662 T18 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 41662 T18 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 41663 T18 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 41663 T18 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 41664 T18 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 41664 T18 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 41664 T18 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 41665 T18 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 41665 T18 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 41666 T18 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 41666 T18 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 41667 T18 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 41667 T18 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 41668 T18 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:19165/solr
   [junit4]   2> 41668 T18 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 41668 T18 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 41669 T18 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 41672 T83 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2f78317c name:ZooKeeperConnection Watcher:127.0.0.1:19165 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 41673 T18 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 41673 T18 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 41675 T18 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 41677 T85 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@77398a7f name:ZooKeeperConnection Watcher:127.0.0.1:19165/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 41677 T18 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 41685 T18 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 42691 T18 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43992_
   [junit4]   2> 42693 T18 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:43992_
   [junit4]   2> 42699 T18 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 42707 T87 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 42708 T87 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 42709 T46 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 42709 T87 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 42715 T42 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:43992",
   [junit4]   2> 	  "node_name":"127.0.0.1:43992_",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "shard":null}
   [junit4]   2> 42716 T42 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=4
   [junit4]   2> 42716 T42 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
   [junit4]   2> 42733 T53 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 42733 T46 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 42733 T86 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 42733 T67 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 43710 T87 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 43711 T87 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 43713 T87 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 43713 T87 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 43715 T87 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 43715 T87 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-D2E2470F5513A1F0-001/tempDir-004/collection1/'
   [junit4]   2> 43718 T87 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-D2E2470F5513A1F0-001/tempDir-004/collection1/lib/README' to classloader
   [junit4]   2> 43719 T87 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-D2E2470F5513A1F0-001/tempDir-004/collection1/lib/.svn/' to classloader
   [junit4]   2> 43720 T87 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-D2E2470F5513A1F0-001/tempDir-004/collection1/lib/classes/' to classloader
   [junit4]   2> 43819 T87 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 43894 T87 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 43897 T87 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 43973 T87 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 44478 T87 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 44482 T87 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 44486 T87 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 44494 T87 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 44562 T87 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 44563 T87 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 44563 T87 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-D2E2470F5513A1F0-001/tempDir-004/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-D2E2470F5513A1F0-001/tempDir-001/jetty2/
   [junit4]   2> 44564 T87 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2127b2ad
   [junit4]   2> 44567 T87 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-D2E2470F5513A1F0-001/tempDir-001/jetty2
   [junit4]   2> 44568 T87 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-D2E2470F5513A1F0-001/tempDir-001/jetty2/index/
   [junit4]   2> 44569 T87 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-D2E2470F5513A1F0-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 44571 T87 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-D2E2470F5513A1F0-001/tempDir-001/jetty2/index
   [junit4]   2> 44571 T87 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=34, maxMergeAtOnceExplicit=38, maxMergedSegmentMB=34.2861328125, floorSegmentMB=1.2841796875, forceMergeDeletesPctAllowed=20.677853650720508, segmentsPerTier=4.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6436012966525844
   [junit4]   2> 44586 T87 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-D2E2470F5513A1F0-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 44587 T87 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 44601 T87 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 44602 T87 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 44602 T87 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 44603 T87 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 44603 T87 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 44604 T87 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 44605 T87 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 44606 T87 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 44606 T87 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 44609 T87 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 44611 T87 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 44613 T87 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 44615 T87 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 44617 T87 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 44618 T87 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 44618 T87 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 44621 T87 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 44622 T87 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 44624 T87 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 44634 T87 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 44654 T87 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 44657 T87 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 44660 T87 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 44663 T87 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 44668 T87 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 44669 T87 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 44671 T87 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=5, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 44674 T87 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-D2E2470F5513A1F0-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 44675 T87 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 44675 T87 oass.SolrIndexSearcher.<init> Opening Searcher@19080dcc[collection1] main
   [junit4]   2> 44676 T87 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 44678 T87 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 44678 T87 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 44680 T87 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 44680 T87 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 44681 T87 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 44682 T87 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
   [junit4]   2> 44683 T87 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 44684 T87 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
   [junit4]   2> 44684 T87 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 44696 T88 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@19080dcc[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> 44709 T87 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 44710 T91 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:43992 collection:collection1 shard:shard3
   [junit4]   2> 44711 T18 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1
   [junit4]   2> 44712 T18 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 44712 T91 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
   [junit4]   2> 44720 T91 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard3
   [junit4]   2> 44723 T46 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 44724 T91 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 44724 T91 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C3 name=collection1 org.apache.solr.core.SolrCore@7d27c57d url=http://127.0.0.1:43992/collection1 node=127.0.0.1:43992_ C3_STATE=coll:collection1 core:collection1 props:{base_url=http://127.0.0.1:43992, node_name=127.0.0.1:43992_, state=down, core=collection1}
   [junit4]   2> 44725 T91 C3 P43992 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:43992/collection1/
   [junit4]   2> 44725 T91 C3 P43992 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 44725 T91 C3 P43992 oasc.SyncStrategy.syncToMe http://127.0.0.1:43992/collection1/ has no replicas
   [junit4]   2> 44726 T91 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:43992/collection1/ shard3
   [junit4]   2> 44726 T91 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
   [junit4]   2> 44734 T67 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 44734 T86 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 44734 T53 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 44734 T46 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 44748 T46 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 44750 T46 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 44750 T86 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 44750 T67 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 44750 T53 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 44785 T91 oasc.ZkController.register We are http://127.0.0.1:43992/collection1/ and leader is http://127.0.0.1:43992/collection1/
   [junit4]   2> 44786 T91 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:43992
   [junit4]   2> 44786 T91 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 44786 T91 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 44787 T91 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 44789 T46 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 44794 T42 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "core_node_name":"core_node2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:43992",
   [junit4]   2> 	  "node_name":"127.0.0.1:43992_",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "shard":"shard3"}
   [junit4]   2> 44900 T86 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (l

[...truncated too long message...]

nwiki.random.lines.txt -Dtests.locale=ja_JP_JP_#u-ca-japanese -Dtests.timezone=SystemV/HST10 -Dtests.file.encoding=US-ASCII
   [junit4]   2> REQUEST FAILED for id=1-1974
   [junit4]   2> java.lang.IllegalStateException: Connection pool shut down
   [junit4]   2> 	at org.apache.http.util.Asserts.check(Asserts.java:34)
   [junit4]   2> 	at org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:169)
   [junit4]   2> 	at org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:202)
   [junit4]   2> 	at org.apache.http.impl.conn.PoolingClientConnectionManager.requestConnection(PoolingClientConnectionManager.java:184)
   [junit4]   2> 	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:414)
   [junit4]   2> 	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
   [junit4]   2> 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
   [junit4]   2> 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
   [junit4]   2> 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:448)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
   [junit4]   2> 	at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:124)
   [junit4]   2> 	at org.apache.solr.cloud.StopableIndexingThread.indexDoc(StopableIndexingThread.java:168)
   [junit4]   2> 	at org.apache.solr.cloud.StopableIndexingThread.indexr(StopableIndexingThread.java:158)
   [junit4]   2> 	at org.apache.solr.cloud.StopableIndexingThread.run(StopableIndexingThread.java:103)
   [junit4]   2> java.lang.IllegalStateException: Connection pool shut down
   [junit4]   2> 	at org.apache.http.util.Asserts.check(Asserts.java:34)
   [junit4]   2> 	at org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:169)
   [junit4]   2> 	at org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:202)
   [junit4]   2> 	at org.apache.http.impl.conn.PoolingClientConnectionManager.requestConnection(PoolingClientConnectionManager.java:184)
   [junit4]   2> 	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:414)
   [junit4]   2> 	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
   [junit4]   2> 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
   [junit4]   2> 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
   [junit4]   2> 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:448)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
   [junit4]   2> 	at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:124)
   [junit4]   2> 	at org.apache.solr.cloud.StopableIndexingThread.indexDoc(StopableIndexingThread.java:168)
   [junit4]   2> 	at org.apache.solr.cloud.StopableIndexingThread.indexr(StopableIndexingThread.java:158)
   [junit4]   2> 	at org.apache.solr.cloud.StopableIndexingThread.run(StopableIndexingThread.java:103)
   [junit4] ERROR   0.00s J1 | ChaosMonkeySafeLeaderTest (suite) <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest: 
   [junit4]    >    1) Thread[id=138, name=StopableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.solr.cloud.StopableIndexingThread.run(StopableIndexingThread.java:121)
   [junit4]    >    2) Thread[id=137, name=StopableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.solr.cloud.StopableIndexingThread.run(StopableIndexingThread.java:121)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([D2E2470F5513A1F0]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=137, name=StopableIndexingThread, state=BLOCKED, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]    >         at java.io.PrintStream.println(PrintStream.java:806)
   [junit4]    >         at org.apache.solr.cloud.StopableIndexingThread.run(StopableIndexingThread.java:107)
   [junit4]    >    2) Thread[id=138, name=StopableIndexingThread, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]    >         at java.lang.Class.getEnclosingMethod0(Native Method)
   [junit4]    >         at java.lang.Class.getEnclosingMethodInfo(Class.java:959)
   [junit4]    >         at java.lang.Class.getEnclosingClass(Class.java:1132)
   [junit4]    >         at java.lang.Class.getSimpleBinaryName(Class.java:1277)
   [junit4]    >         at java.lang.Class.getSimpleName(Class.java:1169)
   [junit4]    >         at java.lang.Class.isAnonymousClass(Class.java:1245)
   [junit4]    >         at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.AnonymousAndLocalClassExclusionStrategy.isAnonymousOrLocal(AnonymousAndLocalClassExclusionStrategy.java:35)
   [junit4]    >         at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.AnonymousAndLocalClassExclusionStrategy.shouldSkipClass(AnonymousAndLocalClassExclusionStrategy.java:31)
   [junit4]    >         at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.DisjunctionExclusionStrategy.shouldSkipClass(DisjunctionExclusionStrategy.java:47)
   [junit4]    >         at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.internal.bind.ExcludedTypeAdapterFactory.create(ExcludedTypeAdapterFactory.java:41)
   [junit4]    >         at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.internal.bind.MiniGson.getAdapter(MiniGson.java:92)
   [junit4]    >         at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.Gson.toJson(Gson.java:504)
   [junit4]    >         at com.carrotsearch.ant.tasks.junit4.events.Serializer.serialize(Serializer.java:87)
   [junit4]    >         at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
   [junit4]    >         at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
   [junit4]    >         at java.io.PrintStream.flush(PrintStream.java:338)
   [junit4]    >         at java.io.FilterOutputStream.flush(FilterOutputStream.java:140)
   [junit4]    >         at java.io.PrintStream.write(PrintStream.java:482)
   [junit4]    >         at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
   [junit4]    >         at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)
   [junit4]    >         at sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:104)
   [junit4]    >         at java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:185)
   [junit4]    >         at java.io.PrintStream.newLine(PrintStream.java:546)
   [junit4]    >         at java.io.PrintStream.println(PrintStream.java:824)
   [junit4]    >         at java.lang.Throwable$WrappedPrintStream.println(Throwable.java:748)
   [junit4]    >         at java.lang.Throwable.printStackTrace(Throwable.java:658)
   [junit4]    >         at java.lang.Throwable.printStackTrace(Throwable.java:643)
   [junit4]    >         at java.lang.Throwable.printStackTrace(Throwable.java:634)
   [junit4]    >         at org.apache.solr.cloud.StopableIndexingThread.run(StopableIndexingThread.java:108)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([D2E2470F5513A1F0]:0)
   [junit4] Completed on J1 in 106.07s, 1 test, 1 failure, 2 errors <<< FAILURES!

[...truncated 1267 lines...]
BUILD FAILED
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:498: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:471: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:60: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/extra-targets.xml:39: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build.xml:189: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/common-build.xml:496: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:1359: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:962: There were test failures: 429 suites, 1774 tests, 2 suite-level errors, 1 failure, 1036 ignored (7 assumptions)

Total time: 190 minutes 45 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-NightlyTests-5.x #628
Archived 5 artifacts
Archive block size is 32768
Received 0 blocks and 175263721 bytes
Compression is 0.0%
Took 30 sec
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-NightlyTests-5.x - Build # 631 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-5.x/631/

1 tests failed.
REGRESSION:  org.apache.solr.handler.component.DistributedSpellCheckComponentTest.testDistribSearch

Error Message:
Captured an uncaught exception in thread: Thread[id=2263, name=Thread-1076, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=2263, name=Thread-1076, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]
	at __randomizedtesting.SeedInfo.seed([4382992141BC9E4E:C264173936E3FE72]:0)
Caused by: java.lang.RuntimeException: org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: https://127.0.0.1:44787
	at __randomizedtesting.SeedInfo.seed([4382992141BC9E4E]:0)
	at org.apache.solr.BaseDistributedSearchTestCase$5.run(BaseDistributedSearchTestCase.java:580)
Caused by: org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: https://127.0.0.1:44787
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:558)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
	at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:91)
	at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
	at org.apache.solr.BaseDistributedSearchTestCase$5.run(BaseDistributedSearchTestCase.java:575)
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(SocketInputStream.java:152)
	at java.net.SocketInputStream.read(SocketInputStream.java:122)
	at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
	at sun.security.ssl.InputRecord.read(InputRecord.java:480)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927)
	at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884)
	at sun.security.ssl.AppInputStream.read(AppInputStream.java:102)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:260)
	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:271)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:682)
	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:486)
	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:448)
	... 5 more




Build Log:
[...truncated 12088 lines...]
   [junit4] Suite: org.apache.solr.handler.component.DistributedSpellCheckComponentTest
   [junit4]   2> Creating dataDir: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.component.DistributedSpellCheckComponentTest-4382992141BC9E4E-001/init-core-data-001
   [junit4]   2> 261144 T500 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (true)
   [junit4]   2> 261144 T500 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 261148 T500 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 261151 T500 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 261158 T500 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 261163 T500 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:43442
   [junit4]   2> 261166 T500 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 261167 T500 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 261167 T500 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr
   [junit4]   2> 261167 T500 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/'
   [junit4]   2> 261183 T500 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/solr.xml
   [junit4]   2> 261307 T500 oasc.CoreContainer.<init> New CoreContainer 7563004
   [junit4]   2> 261308 T500 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/]
   [junit4]   2> 261309 T500 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 261309 T500 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: https
   [junit4]   2> 261309 T500 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 261310 T500 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 261310 T500 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 261310 T500 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 261310 T500 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 261311 T500 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 261311 T500 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 261314 T500 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 261317 T500 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 261317 T500 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 261318 T500 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 261322 T510 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/'
   [junit4]   2> 261324 T510 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 261325 T510 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/lib/.svn/' to classloader
   [junit4]   2> 261326 T510 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 261456 T510 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 261819 T510 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 261820 T510 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 261833 T510 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 262248 T510 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 262264 T510 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 262267 T510 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 262283 T510 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 262289 T510 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 262294 T510 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 262295 T510 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 262296 T510 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 262296 T510 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 262297 T510 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 262297 T510 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 262298 T510 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from instancedir /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/
   [junit4]   2> 262298 T510 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
   [junit4]   2> 262384 T510 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.component.DistributedSpellCheckComponentTest-4382992141BC9E4E-001/tempDir-001/control/data/
   [junit4]   2> 262384 T510 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@41d42a35
   [junit4]   2> 262388 T510 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.component.DistributedSpellCheckComponentTest-4382992141BC9E4E-001/tempDir-001/control/data
   [junit4]   2> 262390 T510 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.component.DistributedSpellCheckComponentTest-4382992141BC9E4E-001/tempDir-001/control/data/index/
   [junit4]   2> 262391 T510 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.component.DistributedSpellCheckComponentTest-4382992141BC9E4E-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 262392 T510 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.component.DistributedSpellCheckComponentTest-4382992141BC9E4E-001/tempDir-001/control/data/index
   [junit4]   2> 262392 T510 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=43, maxMergeAtOnceExplicit=41, maxMergedSegmentMB=14.1728515625, floorSegmentMB=1.9228515625, forceMergeDeletesPctAllowed=9.43486417838211, segmentsPerTier=39.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5610078102937456
   [junit4]   2> 262416 T510 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(NRTCachingDirectory(MMapDirectory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.component.DistributedSpellCheckComponentTest-4382992141BC9E4E-001/tempDir-001/control/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@63d07aa2; maxCacheMB=48.0 maxMergeSizeMB=4.0)),segFN=segments_1,generation=1}
   [junit4]   2> 262416 T510 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 262417 T510 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
   [junit4]   2> 262420 T510 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 262421 T510 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 262421 T510 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 262421 T510 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 262421 T510 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 262422 T510 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 262422 T510 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 262422 T510 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 262422 T510 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 262423 T510 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 262423 T510 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 262423 T510 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 262424 T510 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 262424 T510 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 262425 T510 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 262425 T510 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 262425 T510 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 262426 T510 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 262426 T510 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 262427 T510 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
   [junit4]   2> 262427 T510 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
   [junit4]   2> 262427 T510 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 262428 T510 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 262428 T510 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 262429 T510 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
   [junit4]   2> 262429 T510 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
   [junit4]   2> 262429 T510 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
   [junit4]   2> 262430 T510 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 262430 T510 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 262430 T510 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 262431 T510 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 262431 T510 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 262432 T510 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 262432 T510 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 262432 T510 oasc.RequestHandlers.initHandlersFromConfig created mltrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 262433 T510 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 262433 T510 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
   [junit4]   2> 262434 T510 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
   [junit4]   2> 262460 T510 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 262462 T510 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 262464 T510 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 262467 T510 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 262475 T510 oasc.SolrCore.initDeprecatedSupport WARN solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
   [junit4]   2> 262477 T510 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 262477 T510 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 262478 T510 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@462215f2
   [junit4]   2> 262480 T510 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(NRTCachingDirectory(MMapDirectory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.component.DistributedSpellCheckComponentTest-4382992141BC9E4E-001/tempDir-001/control/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@63d07aa2; maxCacheMB=48.0 maxMergeSizeMB=4.0)),segFN=segments_1,generation=1}
   [junit4]   2> 262481 T510 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 262481 T510 oass.SolrIndexSearcher.<init> Opening Searcher@12e39683[collection1] main
   [junit4]   2> 262481 T510 oasr.RestManager.init Initializing RestManager with initArgs: {storageIO=org.apache.solr.rest.ManagedResourceStorage$InMemoryStorageIO}
   [junit4]   2> 262482 T510 oasr.ManagedResourceStorage.load Reading _rest_managed.json using InMemoryStorage
   [junit4]   2> 262482 T510 oasr.ManagedResource.reloadFromStorage WARN No stored data found for /rest/managed
   [junit4]   2> 262482 T510 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using InMemoryStorage
   [junit4]   2> 262482 T510 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 262483 T510 oashc.SpellCheckComponent.inform Initializing spell checkers
   [junit4]   2> 262494 T510 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 262555 T511 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@12e39683[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> 262559 T510 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 262560 T500 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0
   [junit4]   2> 262560 T500 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 262567 T500 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 262575 T500 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 262579 T500 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:43448
   [junit4]   2> 262583 T500 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 262583 T500 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 262584 T500 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr
   [junit4]   2> 262584 T500 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/'
   [junit4]   2> 262641 T500 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/solr.xml
   [junit4]   2> 262738 T500 oasc.CoreContainer.<init> New CoreContainer 1637709079
   [junit4]   2> 262739 T500 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/]
   [junit4]   2> 262740 T500 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 262740 T500 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: https
   [junit4]   2> 262741 T500 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 262741 T500 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 262741 T500 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 262742 T500 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 262742 T500 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 262742 T500 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 262743 T500 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 262747 T500 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 262750 T500 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 262751 T500 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 262751 T500 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 262757 T522 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/'
   [junit4]   2> 262759 T522 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 262759 T522 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/lib/.svn/' to classloader
   [junit4]   2> 262760 T522 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 262842 T522 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 262914 T522 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 262915 T522 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 262929 T522 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 263267 T522 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 263278 T522 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 263280 T522 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 263290 T522 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 263293 T522 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 263296 T522 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 263297 T522 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 263297 T522 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 263297 T522 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 263298 T522 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 263298 T522 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 263298 T522 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from instancedir /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/
   [junit4]   2> 263298 T522 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
   [junit4]   2> 263299 T522 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.component.DistributedSpellCheckComponentTest-4382992141BC9E4E-001/tempDir-001/shard0/data/
   [junit4]   2> 263299 T522 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@41d42a35
   [junit4]   2> 263301 T522 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.component.DistributedSpellCheckComponentTest-4382992141BC9E4E-001/tempDir-001/shard0/data
   [junit4]   2> 263302 T522 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.component.DistributedSpellCheckComponentTest-4382992141BC9E4E-001/tempDir-001/shard0/data/index/
   [junit4]   2> 263302 T522 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.component.DistributedSpellCheckComponentTest-4382992141BC9E4E-001/tempDir-001/shard0/data/index' doesn't exist. Creating new index...
   [junit4]   2> 263303 T522 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.component.DistributedSpellCheckComponentTest-4382992141BC9E4E-001/tempDir-001/shard0/data/index
   [junit4]   2> 263303 T522 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=43, maxMergeAtOnceExplicit=41, maxMergedSegmentMB=14.1728515625, floorSegmentMB=1.9228515625, forceMergeDeletesPctAllowed=9.43486417838211, segmentsPerTier=39.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5610078102937456
   [junit4]   2> 263335 T522 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(NRTCachingDirectory(MMapDirectory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.component.DistributedSpellCheckComponentTest-4382992141BC9E4E-001/tempDir-001/shard0/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1fb182ac; maxCacheMB=48.0 maxMergeSizeMB=4.0)),segFN=segments_1,generation=1}
   [junit4]   2> 263335 T522 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 263336 T522 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
   [junit4]   2> 263339 T522 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 263339 T522 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 263339 T522 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 263339 T522 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 263339 T522 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 263339 T522 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 263340 T522 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 263340 T522 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 263340 T522 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 263340 T522 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 263341 T522 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 263341 T522 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 263341 T522 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 263342 T522 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 263342 T522 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 263342 T522 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 263342 T522 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 263343 T522 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 263343 T522 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 263343 T522 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
   [junit4]   2> 263344 T522 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
   [junit4]   2> 263344 T522 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 263344 T522 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 263345 T522 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 263345 T522 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
   [junit4]   2> 263345 T522 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
   [junit4]   2> 263345 T522 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
   [junit4]   2> 263346 T522 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 263346 T522 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 263346 T522 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 263347 T522 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 263347 T522 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 263347 T522 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 263348 T522 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 263348 T522 oasc.RequestHandlers.initHandlersFromConfig created mltrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 263348 T522 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 263349 T522 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
   [junit4]   2> 263349 T522 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
   [junit4]   2> 263364 T522 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 263365 T522 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 263367 T522 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 263368 T522 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 263373 T522 oasc.SolrCore.initDeprecatedSupport WARN solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
   [junit4]   2> 263374 T522 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 263374 T522 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 263375 T522 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@567d49b6
   [junit4]   2> 263376 T522 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(NRTCachingDirectory(MMapDirectory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.component.DistributedSpellCheckComponentTest-4382992141BC9E4E-001/tempDir-001/shard0/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1fb182ac; maxCacheMB=48.0 maxMergeSizeMB=4.0)),segFN=segments_1,generation=1}
   [junit4]   2> 263376 T522 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 263377 T522 oass.SolrIndexSearcher.<init> Opening Searcher@115bbaa5[collection1] main
   [junit4]   2> 263377 T522 oasr.RestManager.init Initializing RestManager with initArgs: {storageIO=org.apache.solr.rest.ManagedResourceStorage$InMemoryStorageIO}
   [junit4]   2> 263377 T522 oasr.ManagedResourceStorage.load Reading _rest_managed.json using InMemoryStorage
   [junit4]   2> 263377 T522 oasr.ManagedResource.reloadFromStorage WARN No stored data found for /rest/managed
   [junit4]   2> 263377 T522 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using InMemoryStorage
   [junit4]   2> 263378 T522 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 263378 T522 oashc.SpellCheckComponent.inform Initializing spell checkers
   [junit4]   2> 263388 T522 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 263495 T523 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@115bbaa5[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> 263500 T522 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 263500 T500 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0
   [junit4]   2> 263501 T500 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> ASYNC  NEW_CORE C164 name=collection1 org.apache.solr.core.SolrCore@767fb281
   [junit4]   2> 263654 T503 C164 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 263655 T503 C164 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {deleteByQuery=*:*} 0 2
   [junit4]   2> ASYNC  NEW_CORE C165 name=collection1 org.apache.solr.core.SolrCore@45382e46
   [junit4]   2> 263796 T515 C165 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 263797 T515 C165 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {deleteByQuery=*:*} 0 2
   [junit4]   2> 263819 T504 C164 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[1]} 0 7
   [junit4]   2> 263836 T516 C165 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[1]} 0 4
   [junit4]   2> 263848 T505 C164 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[2]} 0 2
   [junit4]   2> 263860 T517 C165 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[2]} 0 1
   [junit4]   2> 263873 T506 C164 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[3]} 0 1
   [junit4]   2> 263886 T518 C165 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[3]} 0 1
   [junit4]   2> 263899 T507 C164 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[4]} 0 2
   [junit4]   2> 263910 T519 C165 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[4]} 0 1
   [junit4]   2> 263922 T508 C164 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[5]} 0 1
   [junit4]   2> 263934 T520 C165 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[5]} 0 1
   [junit4]   2> 263944 T509 C164 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[6]} 0 1
   [junit4]   2> 263955 T521 C165 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[6]} 0 1
   [junit4]   2> 263967 T539 C164 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[7]} 0 1
   [junit4]   2> 263980 T541 C165 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[7]} 0 1
   [junit4]   2> 263990 T543 C164 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[8]} 0 0
   [junit4]   2> 264001 T545 C165 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[8]} 0 1
   [junit4]   2> 264014 T547 C164 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[9]} 0 2
   [junit4]   2> 264026 T549 C165 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[9]} 0 2
   [junit4]   2> ASYNC  NEW_CORE C166 name=collection1 org.apache.solr.core.SolrCore@767fb281
   [junit4]   2> 264043 T551 C166 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10]} 0 1
   [junit4]   2> ASYNC  NEW_CORE C167 name=collection1 org.apache.solr.core.SolrCore@45382e46
   [junit4]   2> 264055 T553 C167 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10]} 0 1
   [junit4]   2> 264067 T555 C166 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[12]} 0 1
   [junit4]   2> 264077 T557 C167 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[12]} 0 1
   [junit4]   2> 264088 T559 C166 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[13]} 0 1
   [junit4]   2> 264099 T561 C167 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[13]} 0 1
   [junit4]   2> 264112 T563 C166 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[14]} 0 1
   [junit4]   2> 264124 T565 C167 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[14]} 0 2
   [junit4]   2> 264136 T567 C166 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[15]} 0 2
   [junit4]   2> 264146 T569 C167 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[15]} 0 1
   [junit4]   2> 264177 T571 C166 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[16]} 0 22
   [junit4]   2> 264202 T573 C167 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[16]} 0 16
   [junit4]   2> 264214 T575 C166 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[17]} 0 3
   [junit4]   2> 264225 T577 C167 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[17]} 0 2
   [junit4]   2> 264236 T579 C166 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[18]} 0 1
   [junit4]   2> 264245 T581 C167 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[18]} 0 0
   [junit4]   2> 264256 T583 C166 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[19]} 0 1
   [junit4]   2> 264267 T585 C167 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[19]} 0 1
   [junit4]   2> 264279 T587 C166 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[20]} 0 2
   [junit4]   2> 264290 T589 C167 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[20]} 0 1
   [junit4]   2> 264301 T591 C166 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[21]} 0 2
   [junit4]   2> 264312 T593 C167 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[21]} 0 2
   [junit4]   2> 264323 T595 C166 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[22]} 0 2
   [junit4]   2> 264334 T597 C167 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[22]} 0 1
   [junit4]   2> 264345 T599 C166 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[23]} 0 2
   [junit4]   2> 264356 T601 C167 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[23]} 0 2
   [junit4]   2> 264366 T603 C166 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[24]} 0 2
   [junit4]   2> 264376 T605 C167 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[24]} 0 2
   [junit4]   2> 264386 T607 C166 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[25]} 0 2
   [junit4]   2> 264395 T609 C167 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[25]} 0 2
   [junit4]   2> 264403 T611 C166 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[26]} 0 1
   [junit4]   2> 264411 T613 C167 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[26]} 0 1
   [junit4]   2> 264418 T615 C166 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 264479 T615 C166 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(NRTCachingDirectory(MMapDirectory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.component.DistributedSpellCheckComponentTest-4382992141BC9E4E-001/tempDir-001/control/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@63d07aa2; maxCacheMB=48.0 maxMergeSizeMB=4.0)),segFN=segments_1,generation=1}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(NRTCachingDirectory(MMapDirectory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.component.DistributedSpellCheckComponentTest-4382992141BC9E4E-001/tempDir-001/control/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@63d07aa2; maxCacheMB=48.0 maxMergeSizeMB=4.0)),segFN=segments_2,generation=2}
   [junit4]   2> 264480 T615 C166 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
   [junit4]   2> 264482 T615 C166 oass.SolrIndexSearcher.<init> Opening Searcher@686fe37a[collection1] main
   [junit4]   2> 264483 T615 C166 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 264484 T511 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@686fe37a[collection1] main{UninvertingDirectoryReader(Uninverting(_0(5.0.0):c15) Uninverting(_1(5.0.0):c10))}
   [junit4]   2> 264485 T615 C166 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&softCommit=false&commit=true&version=2&wt=javabin} {commit=} 0 67
   [junit4]   2> 264493 T618 C167 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 264537 T618 C167 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(NRTCachingDirectory(MMapDirectory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.component.DistributedSpellCheckComponentTest-4382992141BC9E4E-001/tempDir-001/shard0/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1fb182ac; maxCacheMB=48.0 maxMergeSizeMB=4.0)),segFN=segments_1,generation=1}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(NRTCachingDirectory(MMapDirectory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.component.DistributedSpellCheckComponentTest-4382992141BC9E4E-001/tempDir-001/shard0/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1fb182ac; maxCacheMB=48.0 maxMergeSizeMB=4.0)),segFN=segments_2,generation=2}
   [junit4]   2> 264538 T618 C167 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
   [junit4]   2> 264542 T618 C167 oass.SolrIndexSearcher.<init> Opening Searcher@634d47f3[collection1] main
   [junit4]   2> 264543 T618 C167 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 264544 T523 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@634d47f3[collection1] main{UninvertingDirectoryReader(Uninverting(_0(5.0.0):c15) Uninverting(_1(5.0.0):c10))}
   [junit4]   2> 264545 T618 C167 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&softCommit=false&commit=true&version=2&wt=javabin} {commit=} 0 52
   [junit4]   2> 264733 T621 C166 oasc.SolrCore.execute [collection1] webapp= path=/select params={q=*:*&shards.qt=spellCheckCompRH&qt=spellCheckCompRH&spellcheck=true&spellcheck.build=true&fl=id,lowerfilt&version=2&wt=javabin} hits=25 status=0 QTime=179 
   [junit4]   2> 265057 T626 C167 oasc.SolrCore.execute [collection1] webapp= path=/select params={q=*:*&fsv=true&distrib=false&shards.qt=spellCheckCompRH&qt=spellCheckCompRH&shard.url=https://127.0.0.1:43448/&spellcheck.count=5&spellcheck.onlyMorePopular=false&fl=id,score&spellcheck.extendedResults=false&rows=10&version=2&wt=javabin&start=0&NOW=1411591452567&spellcheck=true&spellcheck.build=true&isShard=true} hits=25 status=0 QTime=88 
   [junit4]   2> 265079 T628 C167 oasc.SolrCore.execute [collection1] webapp= path=/select params={q=*:*&distrib=false&shards.qt=spellCheckCompRH&qt=spellCheckCompRH&shard.url=https://127.0.0.1:43448/&spellcheck.count=1&spellcheck.onlyMorePopular=false&fl=id,lowerfilt&spellcheck.extendedResults=false&version=2&wt=javabin&ids=1,7,10,2,3,5,8,4,6,9&NOW=1411591452567&spellcheck=false&spellcheck.build=true&isShard=true} status=0 QTime=1 
   [junit4]   2> 265093 T623 C167 oashc.SpellCheckComponent.finishStage https://127.0.0.1:43448/ null
   [junit4]   2> 265094 T623 C167 oashc.SpellCheckComponent.finishStage https://127.0.0.1:43448/ null
   [junit4]   2> 265094 T623 C167 oasc.SolrCore.execute [collection1] webapp= path=/select params={q=*:*&shards.qt=spellCheckCompRH&qt=spellCheckCompRH&shards=https://127.0.0.1:43448/&spellcheck=true&spellcheck.build=true&fl=id,lowerfilt&version=2&wt=javabin} hits=25 status=0 QTime=352 
   [junit4]   2> 265433 T630 C166 oasc.SolrCore.execute [collection1] webapp= path=/select params={q=*:*&distrib=false&shards.qt=spellCheckCompRH_Direct&qt=spellCheckCompRH_Direct&spellcheck=true&spellcheck.q=toyata&fl=id,lowerfilt&version=2&wt=javabin} hits=25 status=0 QTime=323 
   [junit4]   2> 265485 T634 C167 oasc.SolrCore.execute [collection1] webapp= path=/select params={fsv=true&q=*:*&distrib=false&spellcheck.dictionary=direct&shards.qt=spellCheckCompRH_Direct&qt=spellCheckCompRH_Direct&shard.url=[ff01::114]:33332/|https://127.0.0.1:43448/&spellcheck.count=5&spellcheck.q=toyata&spellcheck.onlyMorePopular=false&fl=id,score&spellcheck.extendedResults=false&rows=10&version=2&wt=javabin&start=0&NOW=1411591453277&spellcheck=true&isShard=true} hits=25 status=0 QTime=26 
   [junit4]   2> 265494 T636 C167 oasc.SolrCore.execute [collection1] webapp= path=/select params={q=*:*&distrib=false&spellcheck.dictionary=direct&shards.qt=spellCheckCompRH_Direct&qt=spellCheckCompRH_Direct&shard.url=[ff01::114]:33332/|https://127.0.0.1:43448/&spellcheck.count=1&spellcheck.q=toyata&spellcheck.onlyMorePopular=false&fl=id,lowerfilt&spellcheck.extendedResults=false&version=2&wt=javabin&ids=1,7,10,2,3,5,8,4,6,9&NOW=1411591453277&spellcheck=false&isShard=true} status=0 QTime=1 
   [junit4]   2> 265496 T632 C167 oashc.SpellCheckComponent.finishStage [ff01::114]:33332/|https://127.0.0.1:43448/ {suggestions={toyata={numFound=1,startOffset=0,endOffset=6,suggestion=[toyota]}},originalTerms=[toyata]}
   [junit4]   2> 265496 T632 C167 oashc.SpellCheckComponent.finishStage [ff01::114]:33332/|https://127.0.0.1:43448/ null
   [junit4]   2> 265497 T632 C167 oasc.SolrCore.execute [collection1] webapp= path=/select params={q=*:*&shards.qt=spellCheckCompRH_Direct&qt=spellCheckCompRH_Direct&shards=[ff01::114]:33332/|https://127.0.0.1:43448/&spellcheck=true&spellcheck.q=toyata&fl=id,lowerfilt&version=2&wt=javabin} hits=25 status=0 QTime=45 
   [junit4]   2> 265499 T500 oas.BaseDistributedSearchTestCase.query starting stress...
   [junit4]   2> 265678 T649 C167 oasc.SolrCore.execute [collection1] webapp= path=/select params={fsv=true&q=*:*&distrib=false&spellcheck.dictionary=direct&shards.qt=spellCheckCompRH_Direct&qt=spellCheckCompRH_Direct&shard.url=[ff01::114]:33332/|https://127.0.0.1:43448/&spellcheck.count=5&spellcheck.q=toyata&spellcheck.onlyMorePopular=false&fl=id,score&spellcheck.extendedResults=false&rows=10&version=2&wt=javabin&start=0&NOW=1411591453337&spellcheck=true&isShard=true} hits=25 status=0 QTime=18 
   [junit4]   2> 265687 T656 C167 oasc.SolrCore.execute [collection1] webapp= path=/select params={q=*:*&distrib=false&spellcheck.dictionary=direct&shards.qt=spellCheckCompRH_Direct&qt=spellCheckCompRH_Direct&shard.url=[ff01::114]:33332/|https://127.0.0.1:43448/&spellcheck.count=1&spellcheck.q=toyata&spellcheck.onlyMorePopular=false&fl=id,lowerfilt&spellcheck.extendedResults=false&version=2&wt=javabin&ids=1,7,10,2,3,5,8,4,6,9&NOW=1411591453337&spellcheck=false&isShard=true} status=0 QTime=1 
   [junit4]   2> 265688 T652 C167 oasc.SolrCore.execute [collection1] webapp= path=/select params={fsv=true&q=*:*&distrib=false&spellcheck.dictionary=direct&shards.qt=spellCheckCompRH_Direct&qt=spellCheckCompRH_Direct&shard.url=[ff01::114]:33332/|https://127.0.0.1:43448/&spellcheck.count=5&spellcheck.q=toyata&spellcheck.onlyMorePopular=false&fl=id,score&spellcheck.extendedResults=false&rows=10&version=2&wt=javabin&start=0&NOW=1411591453338&spellcheck=true&isShard=true} hits=25 status=0 QTime=11 
   [junit4]   2> 265688 T641 C167 oashc.SpellCheckComponent.finishStage [ff01::114]:33332/|https://127.0.0.1:43448/ {suggestions={toyata={numFound=1,startOffset=0,endOffset=6,suggestion=[toyota]}},originalTerms=[toyata]}
   [junit4]   2> 265689 T641 C167 oashc.SpellCheckComponent.finishStage [ff01::114]:33332/|https://127.0.0.1:43448/ null
   [junit4]   2> 265689 T641 C167 oasc.SolrCore.execute [collection1] webapp= path=/select params={q=*:*&shards.qt=spellCheckCompRH_Direct&qt=spellCheckCompRH_Direct&shards=[ff01::114]:33332/|https://127.0.0.1:43448/&spellcheck=true&spellcheck.q=toyata&fl=id,lowerfilt&version=2&wt=javabin} hits=25 status=0 QTime=177 
   [junit4]   2> 265693 T653 C167 oasc.SolrCore.execute [collection1] webapp= path=/select params={fsv=true&q=*:*&distrib=false&spellcheck.dictionary=direct&shards.qt=spellCheckCompRH_Direct&qt=spellCheckCompRH_Direct&shard.url=[ff01::114]:33332/|https://127.0.0.1:43448/&spellcheck.count=5&spellcheck.q=toyata&spellcheck.onlyMorePopular=false&fl=id,score&spellcheck.extendedResults=false&rows=10&version=2&wt=javabin&start=0&NOW=1411591453337&spellcheck=true&isShard=true} hits=25 status=0 QTime=15 
   [junit4]   2> 265697 T659 C167 oasc.SolrCore.execute [collection1] webapp= path=/select params={q=*:*&distrib=false&spellcheck.dictionary=direct&shards.qt=spellCheckCompRH_Direct&qt=spellCheckCompRH_Direct&shard.url=[ff01::114]:33332/|https://127.0.0.1:43448/&spellcheck.count=1&spellcheck.q=toyata&spellcheck.onlyMorePopular=false&fl=id,lowerfilt&spellcheck.extendedResults=false&version=2&wt=javabin&ids=1,7,10,2,3,5,8,4,6,9&NOW=1411591453338&spellcheck=false&isShard=true} status=0 QTime=1 
   [junit4]   2> 265698 T643 C167 oashc.SpellCheckComponent.finishStage [ff01::114]:33332/|https://127.0.0.1:43448/ {suggestions={toyata={numFound=1,startOffset=0,endOffset=6,suggestion=[toyota]}},originalTerms=[toyata]}
   [junit4]   2> 265699 T643 C167 oashc.SpellCheckComponent.finishStage [ff01::114]:33332/|https://127.0.0.1:43448/ null
   [junit4]   2> 265699 T643 C167 oasc.SolrCore.execute [collection1] webapp= path=/select params={q=*:*&shards.qt=spellCheckCompRH_Direct&qt=spellCheckCompRH_Direct&shards=[ff01::114]:33332/|https://127.0.0.1:43448/&spellcheck=true&spellcheck.q=toyata&fl=id,lowerfilt&version=2&wt=javabin} hits=25 status=0 QTime=186 
   [junit4]   2> 265703 T662 C167 oasc.SolrCore.execute [collection1] webapp= path=/select params={q=*:*&distrib=false&spellcheck.dictionary=direct&shards.qt=spellCheckCompRH_Direct&qt=spellCheckCompRH_Direct&shard.url=[ff01::114]:33332/|https://127.0.0.1:43448/&spellcheck.count=1&spellcheck.q=toyata&spellcheck.onlyMorePopular=false&fl=id,lowerfilt&spellcheck.extendedResults=false&version=2&wt=javabin&ids=1,7,10,2,3,5,8,4,6,9&NOW=1411591453337&spellcheck=false&isShard=true} status=0 QTime=1 
   [junit4]   2> 265705 T642 C167 oashc.SpellCheckComponent.finishStage [ff01::114]:33332/|https://127.0.0.1:43448/ {suggestions={toyata={numFound=1,startOffset=0,endOffset=6,suggestion=[toyota]}},originalTerms=[toyata]}
   [junit4]   2> 265705 T642 C167 oashc.SpellCheckComponent.finishStage [ff01::114]:33332/|https://127.0.0.1:43448/ null
   [junit4]   2> 265705 T642 C167 oasc.SolrCore.execute [collection1] webapp= path=/select params={q=*:*&shards.qt=spellCheckCompRH_Direct&qt=spellCheckCompRH_Direct&shards=[ff01::114]:33332/|https://127.0.0.1:43448/&spellcheck=true&spellcheck.q=toyata&fl=id,lowerfilt&version=2&wt=javabin} hits=25 status=0 QTime=193 
   [junit4]   2> 265716 T663 C167 oasc.SolrCore.execute [collection1] webapp= path=/select params={fsv=true&q=*:*&distrib=false&spellcheck.dictionary=direct&shards.qt=spellCheckCompRH_Direct&qt=spellCheckCompRH_Direct&shard.url=[ff01::114]:33332/|https://127.0.0.1:43448/&spellcheck.count=5&spellcheck.q=toyata&spellcheck.onlyMorePopular=false&fl=id,score&spellcheck.extendedResults=false&rows=10&version=2&wt=javabin&start=0&NOW=1411591453520&spellcheck=true&isShard=true} hits=25 status=0 QTime=13 
   [junit4]   2> 265720 T669 C167 oasc.SolrCore.execute [collection1] webapp= path=/select params={fsv=true&q=*:*&distrib=false&spellcheck.dictionary=direct&shards.qt=spellCheckCompRH_Direct&qt=spellCheckCompRH_Direct&shard.url=[ff01::114]:33332/|https://127.0.0.1:43448/&spellcheck.count=5&spellcheck.q=toyata&spellcheck.onlyMorePopular=false&fl=id,score&spellcheck.extendedResults=false&rows=10&version=2&wt=javabin&start=0&NOW=1411591453530&spellcheck=true&isShard=true} hits=25 status=0 QTime=8 
   [junit4]   2> 265724 T672 C167 oasc.SolrCore.execute [collection1] webapp= path=/select params={fsv=true&q=*:*&distrib=false&spellcheck.dictionary=direct&shards.qt=spellCheckCompRH_Direct&qt=spellCheckCompRH_Direct&shard.url=[ff01::114]:33332/|https://127.0.0.1:43448/&spellcheck.count=5&spellcheck.q=toyata&spellcheck.onlyMorePopular=false&fl=id,score&spellcheck.extendedResults=false&rows=10&version=2&wt=javabin&start=0&NOW=1411591453536&spellcheck=true&isShard=true} hits=25 status=0 QTime=7 
   [junit4]   2> 265725 T674 C167 oasc.SolrCore.execute [collection1] webapp= path=/select params={q=*:*&distrib=false&spellcheck.dictionary=direct&shards.qt=spellCheckCompRH_Direct&qt=spellCheckCompRH_Direct&shard.url=[ff01::114]:33332/|https://127.0.0.1:43448/&spellcheck.count=1&spellcheck.q=toyata&spellcheck.onlyMorePopular=false&fl=id,lowerfilt&spellcheck.extendedResults=false&version=2&wt=javabin&ids=1,7,10,2,3,5,8,4,6,9&NOW=1411591453520&spellcheck=false&isShard=true} status=0 QTime=1 
   [junit4]   2> 265727 T658 C167 oashc.SpellCheckComponent.finishStage [ff01::114]:33332/|https://127.0.0.1:43448/ {suggestions={toyata={numFound=1,startOffset=0,endOffset=6,suggestion=[toyota]}},originalTerms=[toyata]}
   [junit4]   2> 265727 T658 C167 oashc.SpellCheckComponent.finishStage [ff01::114]:33332/|https://127.0.0.1:43448/ null
   [junit4]   2> 265727 T658 C167 oasc.SolrCore.execute [collection1] webapp= path=/select params={q=*:*&shards.qt=spellCheckCompRH_Direct&qt=spellCheckCompRH_Direct&shards=[ff01::114]:33332/|https://127.0.0.1:43448/&spellcheck=true&spellcheck.q=toyata&fl=id,lowerfilt&version=2&wt=javabin} hits=25 status=0 QTime=32 
   [junit4]   2> 265729 T676 C167 oasc.SolrCore.execute [collection1] webapp= path=/select params={q=*:*&distrib=false&spellcheck.dictionary=direct&shards.qt=spellCheckCompRH_Direct&qt=spellCheckCompRH_Direct&shard.url=[ff01::114]:33332/|https://127.0.0.1:43448/&spellcheck.count=1&spellcheck.q=toyata&spellcheck.onlyMorePopular=false&fl=id,lowerfilt&spellcheck.extendedResults=false&version=2&wt=javabin&ids=1,7,10,2,3,5,8,4,6,9&NOW=1411591453530&spellcheck=false&isShard=true} status=0 QTime=1 
   [junit4]   2> 265730 T665 C167 oashc.SpellCheckComponent.finishStage [ff01::114]:33332/|https://127.0.0.1:43448/ {suggestions={toyata={numFound=1,startOffset=0,endOffset=6,suggestion=[toyota]}},originalTerms=[toyata]}
   [junit4]   2> 265731 T665 C167 oashc.SpellCheckComponent.finishStage [ff01::114]:33332/|https://127.0.0.1:43448/ null
   [junit4]   2> 265731 T665 C167 oasc.SolrCore.execute [collection1] webapp= path=/select params={q=*:*&shards.qt=spellCheckCompRH_Direct&qt=spellCheckCompRH_Direct&shards=[ff01::114]:33332/|https://127.0.0.1:43448/&spellcheck=true&spellcheck.q=toyata&fl=id,lowerfilt&version=2&wt=javabin} hits=25 status=0 QTime=26 
   [junit4]   2> 265732 T678 C167 oasc.SolrCore.execute [collection1] webapp= path=/select params={q=*:*&distrib=false&spellcheck.dictionary=direct&shards.qt=spellCheckCompRH_Direct&qt=spellCheckCompRH_Direct&shard.url=[ff01::114]:33332/|https://127.0.0.1:43448/&spellcheck.count=1&spellcheck.q=toyata&spellcheck.onlyMorePopular=false&fl=id,lowerfilt&spellcheck.extendedResults=false&version=2&wt=javabin&ids=1,7,10,2,3,5,8,4,6,9&NOW=1411591453536&spellcheck=false&isShard=true} status=0 QTime=1 
   [junit4]   2> 265734 T668 C167 oashc.SpellCheckComponent.finishStage [ff01::114]:33332/|https://127.0.0.1:43448/ {suggestions={toyata={numFound=1,startOffset=0,endOffset=6,suggestion=[toyota]}},originalTerms=[toyata]}
   [junit4]   2> 265734 T668 C167 oashc.SpellCheckComponent.finishStage [ff01::114]:33332/|https://127.0.0.1:43448/ null
   [junit4]   2> 265735 T668 C167 oasc.SolrCore.execute [collection1] webapp= path=/select params={q=*:*&shards.qt=spellCheckCompRH_Direct&qt=spellCheckCompRH_Direct&shards=[ff01::114]:33332/|https://127.0.0.1:43448/&spellcheck=true&spellcheck.q=toyata&fl=id,lowerfilt&version=2&wt=javabin} hits=25 status=0 QTime=24 
   [junit4]   2> 265753 T680 C166 oasc.SolrCore.execute [collection1] webapp= path=/select params={q=*:*&distrib=false&shards.qt=spellCheckCompRH_Direct&qt=spellCheckCompRH_Direct&spellcheck=true&spellcheck.q=toyata&fl=id,lowerfilt&spellcheck.extendedResults=true&version=2&wt=javabin} hits=25 status=0 QTime=12 
   [junit4]   2> 265779 T684 C167 oasc.SolrCore.execute [collection1] webapp= path=/select params={fsv=true&q=*:*&distrib=false&spellcheck.dictionary=direct&shards.qt=spellCheckCompRH_Direct&qt=spellCheckCompRH_Direct&shard.url=[ff01::114]:33332/|https://127.0.0.1:43448/|[ff01::213]:33332/|[ff01::083]:33332/&spellcheck.count=5&spellcheck.q=toyata&spellcheck.onlyMorePopular=false&fl=id,score&spellcheck.extendedResults=true&rows=10&version=2&wt=javabin&start=0&NOW=1411591453586&spellcheck=true&isShard=true} hits=25 status=0 QTime=10 
   [junit4]   2> 265788 T686 C167 oasc.SolrCore.execute [collection1] webapp= path=/select params={q=*:*&distrib=false&spellcheck.dictionary=direct&shards.qt=spellCheckCompRH_Direct&qt=spellCheckCompRH_Direct&shard.url=[ff01::114]:33332/|https://127.0.0.1:43448/|[ff01::213]:33332/|[ff01::083]:33332/&spellcheck.count=1&spellcheck.q=toyata&spellcheck.onlyMorePopular=false&fl=id,lowerfilt&spellcheck.extendedResults=true&version=2&wt=javabin&ids=1,7,10,2,3,5,8,4,6,9&NOW=1411591453586&spellcheck=false&isShard=true} status=0 QTime=1 
   [junit4]   2> 265789 T682 C167 oashc.SpellCheckComponent.finishStage [ff01::114]:33332/|https://127.0.0.1:43448/|[ff01::213]:33332/|[ff01::083]:33332/ {suggestions={toyata={numFound=1,startOffset=0,endOffset=6,origFreq=0,suggestion=[{word=toyota,freq=1}]}},correctlySpelled=true,originalTerms=[toyata]}
   [junit4]   2> 265790 T682 C167 oashc.SpellCheckComponent.finishStage [ff01::114]:33332/|https://127.0.0.1:43448/|[ff01::213]:33332/|[ff01::083]:33332/ null
   [junit4]   2> 265790 T682 C167 oasc.SolrCore.execute [collection1] webapp= path=/select params={q=*:*&shards.qt=spellCheckCompRH_Direct&qt=spellCheckCompRH_Direct&shards=[ff01::114]:33332/|https://127.0.0.1:43448/|[ff01::213]:33332/|[ff01::083]:33332/&spellcheck=true&spellcheck.q=toyata&fl=id,lowerfilt&spellcheck.extendedResults=true&version=2&wt=javabin} hits=25 status=0 QTime=29 
   [junit4]   2> 265792 T500 oas.BaseDistributedSearchTestCase.query starting stress...
   [junit4]   2> 265815 T699 C167 oasc.SolrCore.execute [collection1] webapp= path=/select params={fsv=true&q=*:*&distrib=false&spellcheck.dictionary=direct&shards.qt=spellCheckCompRH_Direct&qt=spellCheckCompRH_Direct&shard.url=[ff01::114]:33332/|https://127.0.0.1:43448/|[ff01::213]:33332/|[ff01::083]:33332/&spellcheck.count=5&spellcheck.q=toyata&spellcheck.onlyMorePopular=false&fl=id,score&spellcheck.extendedResults=true&rows=10&version=2&wt=javabin&start=0&NOW=1411591453625&spellcheck=true&isShard=true} hits=25 status=0 QTime=8 
   [junit4]   2> 265817 T697 C167 oasc.SolrCore.execute [collection1] webapp= path=/select params={fsv=true&q=*:*&distrib=false&spellcheck.dictionary=direct&shards.qt=spellCheckCompRH_Direct&qt=spellCheckCompRH_Direct&shard.url=[ff01::114]:33332/|https://127.0.0.1:43448/|[ff01::213]:33332/|[ff01::083]:33332/&spellcheck.count=5&spellcheck.q=toyata&spellcheck.onlyMorePopular=false&fl=id,score&spellcheck.extendedResults=true&rows=10&version=2&wt=javabin&start=0&NOW=1411591453624&spellcheck=true&isShard=true} hits=25 status=0 QTime=12 
   [junit4]   2> 265818 T698 C167 oasc.SolrCore.execute [collection1] webapp= path=/select params={fsv=true&q=*:*&distrib=false&spellcheck.dictionary=direct&shards.qt=spellCheckCompRH_Direct&qt=spellCheckCompRH_Direct&shard.url=[ff01::114]:33332/|https://127.0.0.1:43448/|[ff01::213]:33332/|[ff01::083]:33332/&spellcheck.count=5&spellcheck.q=toyata&spellcheck.onlyMorePopular=false&fl=id,score&spellcheck.extendedResults=true&rows=10&version=2&wt=javabin&start=0&NOW=1411591453625&spellcheck=true&isShard=true} hits=25 status=0 QTime=12 
   [junit4]   2> 265825 T628 C167 oasc.SolrCore.execute [collection1] webapp= path=/select params={q=*:*&distrib=false&spellcheck.dictionary=direct&shards.qt=spellCheckCompRH_Direct&qt=spellCheckCompRH_Direct&shard.url=[ff01::114]:33332/|https://127.0.0.1:43448/|[ff01::213]:33332/|[ff01::083]:33332/&spellcheck.count=1&spellcheck.q=toyata&spellcheck.onlyMorePopular=false&fl=id,lowerfilt&spellcheck.extendedResults=true&version=2&wt=javabin&ids=1,7,10,2,3,5,8,4,6,9&NOW=1411591453625&spellcheck=false&isShard=true} status=0 QTime=1 
   [junit4]   2> 265826 T634 C167 oasc.SolrCore.execute [collection1] webapp= path=/select params={q=*:*&distrib=false&spellcheck.dictionary=direct&shards.qt=spellCheckCompRH_Direct&qt=spellCheckCompRH_Direct&shard.url=[ff01::114]:33332/|https://127.0.0.1:43448/|[ff01::213]:33332/|[ff01::083]:33332/&spellcheck.count=1&spellcheck.q=toyata&spellcheck.onlyMorePopular=false&fl=id,lowerfilt&spellcheck.extendedResults=true&version=2&wt=javabin&ids=1,7,10,2,3,5,8,4,6,9&NOW=1411591453624&spellcheck=false&isShard=true} status=0 QTime=1 
   [junit4]   2> 265826 T692 C167 oashc.SpellCheckComponent.finishStage [ff01::114]:33332/|https://127.0.0.1:43448/|[ff01::213]:33332/|[ff01::083]:33332/ {suggestions={toyata={numFound=1,startOffset=0,endOffset=6,origFreq=0,suggestion=[{word=toyota,freq=1}]}},correctlySpelled=true,originalTerms=[toyata]}
   [junit4]   2> 265827 T692 C167 oashc.SpellCheckComponent.finishStage [ff01::114]:33332/|https://127.0.0.1:43448/|[ff01::213]:33332/|[ff01::083]:33332/ null
   [junit4]   2> 265827 T692 C167 oasc.SolrCore.execute [collection1] webapp= path=/select params={q=*:*&shards.qt=spellCheckCompRH_Direct&qt=spellCheckCompRH_Direct&shards=[ff01::114]:33332/|https://127.0.0.1:43448/|[ff01::213]:33332/|[ff01::083]:33332/&spellcheck=true&spellcheck.q=toyata&fl=id,lowerfilt&spellcheck.extendedResults=true&version=2&wt=javabin} hits=25 status=0 QTime=27 
   [junit4]   2> 265827 T636 C167 oasc.SolrCore.execute [collection1] webapp= path=/select params={q=*:*&distrib=false&spellcheck.dictionary=direct&shards.qt=spellCheckCompRH_Direct&qt=spellCheckCompRH_Direct&shard.url=[ff01::114]:33332/|https://127.0.0.1:43448/|[ff01::213]:33332/|[ff01::083]:33332/&spellcheck.count=1&spellcheck.q=toyata&spellcheck.onlyMorePopular=false&fl=id,lowerfilt&spellcheck.extendedResults=true&version=2&wt=javabin&ids=1,7,10,2,3,5,8,4,6,9&NOW=1411591453625&spellcheck=false&isShard=true} status=0 QTime=1 
   [junit4]   2> 265828 T691 C167 oashc.SpellCheckComponent.finishStage [ff01::114]:33332/|https://127.0.0.1:43448/|[ff01::213]:33332/|[ff01::083]:33332/ {suggestions={toyata={numFound=1,startOffset=0,endOffset=6,origFreq=0,suggestion=[{word=toyota,freq=1}]}},correctlySpelled=true,originalTerms=[toyata]}
   [junit4]   2> 265828 T691 C167 oashc.SpellCheckComponent.finishStage [ff01::114]:33332/|https://127.0.0.1:43448/|[ff01::213]:33332/|[ff01::083]:33332/ null
   [junit4]   2> 265828 T691 C167 oasc.SolrCore.execute [collection1] webapp= path=/select params={q=*:*&shards.qt=spellCheckCompRH_Direct&qt=spellCheckCompRH_Direct&shards=[ff01::114]:33332/|https://127.0.0.1:43448/|[ff01::213]:33332/|[ff01::083]:33332/&spellcheck=true&spellcheck.q=toyata&fl=id,lowerfilt&spellcheck.extendedResults=true&version=2&wt=javabin} hits=25 status=0 QTime=29 
   [junit4]   2> 265829 T693 C167 oashc.SpellCheckComponent.finishStage [ff01::114]:33332/|https://127.0.0.1:43448/|[ff01::213]:33332/|[ff01::083]:33332/ {suggestions={toyata={numFound=1,startOffset=0,endOffset=6,origFreq=0,suggestion=[{word=toyota,freq=1}]}},correctlySpelled=true,originalTerms=[toyata]}
   [junit4]   2> 265829 T693 C167 oashc.SpellCheckComponent.finishStage [ff01::114]:33332/|https://127.0.0.1:43448/|[ff01::213]:33332/|[ff01::083]:33332/ null
   [junit4]   2> 265829 T693 C167 oasc.SolrCore.execute [collection1] webapp= path=/select params={q=*:*&shards.qt=spellCheckCompRH_Direct&qt=spellCheckCompRH_Direct&shards=[ff01::114]:33332/|https://127.0.0.1:43448/|[ff01::213]:33332/|[ff01::083]:33332/&spellcheck=true&spellcheck.q=toyata&fl=id,lowerfilt&spellcheck.extendedResults=true&version=2&wt=javabin} hits=25 status=0 QTime=29 
   [junit4]   2> 265855 T656 C167 oasc.SolrCore.execute [collection1] webapp= path=/select params={fsv=true&q=*:*&distrib=false&spellcheck.dictionary=direct&shards.qt=spellCheckCompRH_Direct&qt=spellCheckCompRH_Direct&shard.url=[ff01::114]:33332/|https://127.0.0.1:43448/|[ff01::213]:33332/|[ff01::083]:33332/&spellcheck.count=5&spellcheck.q=toyata&spellcheck.onlyMorePopular=false&fl=id,score&spellcheck.extendedResults=true&rows=10&version=2&wt=javabin&start=0&NOW=1411591453660&spellcheck=true&isShard=true} hits=25 status=0 QTime=10 
   [junit4]   2> 265855 T652 C167 oasc.SolrCore.execute [collection1] webapp= path=/select params={fsv=true&q=*:*&distrib=false&spellcheck.dictionary=direct&shards.qt=spellCheckCompRH_Direct&qt=spellCheckCompRH_Direct&shard.url=[ff01::114]:33332/|https://127.0.0.1:43448/|[ff01::213]:33332/|[ff01::083]:33332/&spellcheck.count=5&spellcheck.q=toyata&spellcheck.onlyMorePopular=false&fl=id,score&spellcheck.extendedResults=true&rows=10&version=2&wt=javabin&start=0&NOW=1411591453661&spellcheck=true&isShard=true} hits=25 status=0 QTime=9 
   [junit4]   2> 265855 T649 C167 oasc.SolrCore.execute [collection1] webapp= path=/select params={fsv=true&q=*:*&distrib=false&spellcheck.dictionary=direct&shards.qt=spellCheckCompRH_Direct&qt=spellCheckCompRH_Direct&shard.url=[ff01::114]:33332/|https://127.0.0.1:43448/|[ff01::213]:33332/|[ff01::083]:33332/&spellcheck.count=5&spellcheck.q=toyata&spellcheck.onlyMorePopular=false&fl=id,score&spellcheck.extendedResults=true&rows=10&version=2&wt=javabin&start=0&NOW=1411591453660&spellcheck=true&isShard=true} hits=25 status=0 QTime=10 
   [junit4]   2> 265865 T659 C167 oasc.SolrCore.execute [collection1] webapp= path=/select params={q=*:*&distrib=false&spellcheck.dictionary=direct&shards.qt=spellCheckCompRH_Direct&qt=spellCheckCompRH_Direct&shard.url=[ff01::114]:33332/|https://127.0.0.1:43448/|[ff01::213]:33332/|[ff01::083]:33332/&spellcheck.count=1&spellcheck.q=toyata&spellcheck.onlyMorePopular=false&fl=id,lowerfilt&spellcheck.extendedResults=true&version=2&wt=javabin&ids=1,7,10,2,3,5,8,4,6,9&NOW=1411591453661&spellcheck=false&isShard=true} status=0 QTime=2 
   [junit4]   2> 265865 T653 C167 oasc.SolrCore.execute [collection1] webapp= path=/select params={q=*:*&distrib=false&spellcheck.dictionary=direct&shards.qt=spellCheckCompRH_Direct&qt=spellCheckCompRH_Direct&shard.url=[ff01::114]:33332/|https://127.0.0.1:43448/|[ff01::213]:33332/|[ff01::083]:33332/&spellcheck.count=1&spellcheck.q=toyata&spellcheck.onlyMorePopular=false&fl=id,lowerfilt&spellcheck.extendedResults=true&version=2&wt=javabin&ids=1,7,10,2,3,5,8,4,6,9&NOW=1411591453660&spellcheck=false&isShard=true} status=0 QTime=2 
   [junit4]   2> 265865 T662 C167 oasc.SolrCore.execute [collection1] webapp= path=/select params={q=*:*&distrib=false&spellcheck.dictionary=direct&shards.qt=spellCheckCompRH_Direct&qt=spellCheckCompRH_Direct&shard.url=[ff01::114]:33332/|https://127.0.0.1:43448/|[ff01::213]:33332/|[ff01::083]:33332/&spellcheck.count=1&spellcheck.q=toyata&spellcheck.onlyMorePopular=false&fl=id,lowerfilt&spellcheck.extendedResults=true&version=2&wt=javabin&ids=1,7,10,2,3,5,8,4,6,9&NOW=1411591453660&spellcheck=false&isShard=true} status=0 QTime=1 
   [junit4]   2> 265866 T708 C167 oashc.SpellCheckComponent.finishStage [ff01::114]:33332/|https://127.0.0.1:43448/|[ff01::213]:33332/|[ff01::083]:33332/ {suggestions={toyata={numFound=1,startOffset=0,endOffset=6,origFreq=0,suggestion=[{word=toyota,freq=1}]}},correctlySpelled=true,originalTerms=[toyata]}
   [junit4]   2> 265867 T708 C167 oashc.SpellCheckComponent.finishStage [ff01::114]:33332/|https://127.0.0.1:43448/|[ff01::213]:33332/|[ff01::083]:33332/ null
   [junit4]   2> 265867 T708 C167 oasc.SolrCore.execute [collection1] webapp= path=/select params={q=*:*&shards.qt=spellCheckCompRH_Direct&qt=spellCheckCompRH_Direct&shards=[ff01::114]:33332/|https://127.0.0.1:43448/|[ff01::213]:33332/|[ff01::083]:33332/&spellcheck=true&spellcheck.q=toyata&fl=id,lowerfilt&spellcheck.extendedResults=true&version=2&wt=javabin} hits=25 status=0 QTime=31 
   [junit4]   2> 265867 T707 C167 oashc.SpellCheckComponent.finishStage [ff01::114]:33332/|https://127.0.0.1:43448/|[ff01::213]:33332/|[ff01::083]:33332/ {suggestions={toyata={numFound=1,startOffset=0,endOffset=6,origFreq=0,suggestion=[{word=toyota,freq=1}]}},correctlySpelled=true,originalTerms=[toyata]}
   [junit4]   2> 265867 T706 C167 oashc.SpellCheckComponent.finishStage [ff01::114]:33332/|https://127.0.0.1:43448/|[ff01::213]:33332/|[ff01::083]:33332/ {suggestions={toyata={numFound=1,startOffset=0,endOffset=6,origFreq=0,suggestion=[{word=toyota,freq=1}]}},correctlySpelled=true,originalTerms=[toyata]}
   [junit4]   2> 265867 T707 C167 oashc.SpellCheckComponent.finishStage [ff01::114]:33332/|https://127.0.0.1:43448/|[ff01::213]:33332/|[ff01::083]:33332/ null
   [junit4]   2> 265868 T706 C167 oashc.SpellCheckComponent.finishStage [ff01::114]:33332/|https://127.0.0.1:43448/|[ff01::213]:33332/|[ff01::083]:33332/ null
   [junit4]   2> 265868 T707 C167 oasc.SolrCore.execute [collection1] webapp= path=/select params={q=*:*&shards.qt=spellCheckCompRH_Direct&qt=spellCheckCompRH_Direct&shards=[ff01::114]:33332/|https://127.0.0.1:43448/|[ff01::213]:33332/|[ff01::083]:33332/&spellcheck=true&spellcheck.q=toyata&fl=id,lowerfilt&spellcheck.extendedResults=true&version=2&wt=javabin} hits=25 status=0 QTime=33 
   [junit4]   2> 265868 T706 C167 oasc.SolrCore.execute [collection1] webapp= path=/select params={q=*:*&shards.qt=spellCheckCompRH_Direct&qt=spellCheckCompRH_Direct&shards=[ff01::114]:33332/|https://127.0.0.1:43448/|[ff01::213]:33332/|[ff01::083]:33332/&spellcheck=true&spellcheck.q=toyata&fl=id,lowerfilt&spellcheck.extendedResults=true&version=2&wt=javabin} hits=25 status=0 QTime=33 
   [junit4]   2> 265881 T718 C166 oasc.SolrCore.execute [collection1] webapp= path=/select params={q=*:*&distrib=false&shards.qt=spellCheckCompRH_Direct&qt=spellCheckCompRH_Direct&spellcheck.count=4&spellcheck=true&spellcheck.q=bluo&fl=id,lowerfilt&spellcheck.extendedResults=true&version=2&wt=javabin} hits=25 status=0 QTime=4 
   [junit4]   2> 265899 T663 C167 oasc.SolrCore.execute [collection1] webapp= path=/select params={fsv=true&q=*:*&distrib=false&spellcheck.dictionary=direct&shards.qt=spellCheckCompRH_Direct&qt=spellCheckCompRH_Direct&shard.url=[ff01::083]:33332/|https://127.0.0.1:43448/&spellcheck.count=5&spellcheck.q=bluo&spellcheck.onlyMorePopular=false&fl=id,score&spellcheck.extendedResults=true&rows=10&version=2&wt=javabin&start=0&NOW=1411591453713&spellcheck=true&isShard=true} hits=25 status=0 QTime=4 
   [junit4]   2> 265908 T669 C167 oasc.SolrCore.execute [collection1] webapp= path=/select params={q=*:*&distrib=false&spellcheck.dictionary=direct&shards.qt=spellCheckCompRH_Direct&qt=spellCheckCompRH_Direct&shard.url=[ff01::083]:33332/|https://127.0.0.1:43448/&spellcheck.count=4&spellcheck.q=bluo&spellcheck.onlyMorePopular=false&fl=id,lowerfilt&spellcheck.extendedResults=true&version=2&wt=javabin&ids=1,7,10,2,3,5,8,4,6,9&NOW=1411591453713&spellcheck=false&isShard=true} status=0 QTime=1 
   [junit4]   2> 265909 T720 C167 oashc.SpellCheckComponent.finishStage [ff01::083]:33332/|https://127.0.0.1:43448/ {suggestions={bluo={numFound=1,startOffset=0,endOffset=4,origFreq=0,suggestion=[{word=blue,freq=1}]}},correctlySpelled=true,originalTerms=[bluo]}
   [junit4]   2> 265910 T720 C167 oashc.SpellCheckComponent.finishStage [ff01::083]:33332/|https://127.0.0.1:43448/ null
   [junit4]   2> 265910 T720 C167 oasc.SolrCore.execute [collection1] webapp= path=/select params={q=*:*&shards.qt=spellCheckCompRH_Direct&qt=spellCheckCompRH_Direct&shards=[ff01::083]:33332/|https://127.0.0.1:43448/&spellcheck.count=4&spellcheck=true&spellcheck.q=bluo&fl=id,lowerfilt&spellcheck.extendedResults=true&version=2&wt=javabin} hits=25 status=0 QTime=22 
   [junit4]   2> 265912 T500 oas.BaseDistributedSearchTestCase.query starting stress...
   [junit4]   2> 265927 T672 C167 oasc.SolrCore.execute [collection1] webapp= path=/select params={fsv=true&q=*:*&distrib=false&spellcheck.dictionary=direct&shards.qt=spellCheckCompRH_Direct&qt=spellCheckCompRH_Direct&shard.url=[ff01::083]:33332/|https://127.0.0.1:43448/&spellcheck.count=5&spellcheck.q=bluo&spellcheck.onlyMorePopular=false&fl=id,score&spellcheck.extendedResults=true&rows=10&version=2&wt=javabin&start=0&NOW=1411591453743&spellcheck=true&isShard=true} hits=25 status=0 QTime=3 
   [junit4]   2> 265929 T674 C167 oasc.SolrCore.execute [collection1] webapp= path=/select params={fsv=true&q=*:*&distrib=false&spellcheck.dictionary=direct&shards.qt=spellCheckCompRH_Direct&qt=spellCheckCompRH_Direct&shard.url=[ff01::083]:33332/|https://127.0.0.1:43448/&spellcheck.count=5&spellcheck.q=bluo&spellcheck.onlyMorePopular=false&fl=id,score&spellcheck.extendedResults=true&rows=10&version=2&wt=javabin&start=0&NOW=1411591453744&spellcheck=true&isShard=true} hits=25 status=0 QTime=3 
   [junit4]   2> 265929 T676 C167 oasc.SolrCore.execute [collection1] webapp= path=/select params={fsv=true&q=*:*&distrib=false&spellcheck.dictionary=direct&shards.qt=spellCheckCompRH_Direct&qt=spellCheckCompRH_Direct&shard.url=[ff01::083]:33332/|https://127.0.0.1:43448/&spellcheck.count=5&spellcheck.q=bluo&spellcheck.onlyMorePopular=false&fl=id,score&spellcheck.extendedResults=true&rows=10&version=2&wt=javabin&start=0&NOW=1411591453744&spellcheck=true&isShard=true} hits=25 status=0 QTime=3 
   [junit4]   2> 265935 T678 C167 oasc.SolrCore.execute [collection1] webapp= path=/select params={q=*:*&distrib=false&spellcheck.dictionary=direct&shards.qt=spellCheckCompRH_Direct&qt=spellCheckCompRH_Direct&shard.url=[ff01::083]:33332/|https://127.0.0.1:43448/&spellcheck.count=4&spellcheck.q=bluo&spellcheck.onlyMorePopular=false&fl=id,lowerfilt&spellcheck.extendedResults=true&version=2&wt=javabin&ids=1,7,10,2,3,5,8,4,6,9&NOW=1411591453743&spellcheck=false&isShard=true} status=0 QTime=1 
   [junit4]   2> 265936 T727 C167 oashc.SpellCheckComponent.finishStage [ff01::083]:33332/|https://127.0.0.1:43448/ {suggestions={bluo={numFound=1,startOffset=0,endOffset=4,origFreq=0,suggestion=[{word=blue,freq=1}]}},correctlySpelled=true,originalTerms=[bluo]}
   [junit4]   2> 265936 T727 C167 oashc.SpellCheckComponent.finishStage [ff01::083]:33332/|https://127.0.0.1:43448/ null
   [junit4]   2> 265937 T727 C167 oasc.SolrCore.execute [collection1] webapp= path=/select params={q=*:*&shards.qt=spellCheckCompRH_Direct&qt=spellCheckCompRH_Direct&shards=[ff01::083]:33332/|https://127.0.0.1:43448/&spellcheck.count=4&spellcheck=true&spellcheck.q=bluo&fl=id,lowerfilt&spellcheck.extendedResults=true&version=2&wt=javabin} hits=25 status=0 QTime=19 
   [junit4]   2> 265937 T686 C167 oasc.SolrCore.execute [collection1] webapp= path=/select params={q=*:*&distrib=false&spellcheck.dictionary=direct&shards.qt=spellCheckCompRH_Direct&qt=spellCheckCompRH_Direct&shard.url=[ff01::083]:33332/|https://127.0.0.1:43448/&spellcheck.count=4&spellcheck.q=bluo&spellcheck.onlyMorePopular=false&fl=id,lowerfilt&spellcheck.extendedResults=true&version=2&wt=javabin&ids=1,7,10,2,3,5,8,4,6,9&NOW=1411591453744&spellcheck=false&isShard=true} status=0 QTime=0 
   [junit4]   2> 265937 T684 C167 oasc.SolrCore.execute [collection1] webapp= path=/select params={q=*:*&distrib=false&spellcheck.dictionary=direct&shards.qt=spellCheckCompRH_Direct&qt=spellCheckCompRH_Direct&shard.url=[ff01::083]:33332/|https://127.0.0.1:43448/&spellcheck.count=4&spellcheck.q=bluo&spellcheck.onlyMorePopular=false&fl=id,lowerfilt&spellcheck.extendedResults=true&version=2&wt=javabin&ids=1,7,10,2,3,5,8,4,6,9&NOW=1411591453744&spellcheck=false&isShard=true} status=0 QTime=0 
   [junit4]   2> 265939 T728 C167 oashc.SpellCheckComponent.finishStage [ff01::083]:33332/|https://127.0.0.1:43448/ {suggestions={bluo={numFound=1,startOffset=0,endOffset=4,origFreq=0,suggestion=[{word=blue,freq=1}]}},correctlySpelled=true,originalTerms=[bluo]}
   [junit4]   2> 265939 T728 C167 oashc.SpellCheckComponent.finishStage [ff01::083]:33332/|https://127.0.0.1:43448/ null
   [junit4]   2> 265939 T729 C167 oashc.SpellCheckComponent.finishStage [ff01::083]:33332/|https://127.0.0.1:43448/ {suggestions={bluo={numFound=1,startOffset=0,endOffset=4,origFreq=0,suggestion=[{word=blue,freq=1}]}},correctlySpelled=true,originalTerms=[bluo]}
   [junit4]   2> 265939 T728 C167 oasc.SolrCore.execute [collection1] webapp= path=/select params={q=*:*&shards.qt=spellCheckCompRH_Direct&qt=spellCheckCompRH_Direct&shards=[ff01::083]:33332/|https://127.0.0.1:43448/&spellcheck.count=4&spellcheck=true&spellcheck.q=bluo&fl=id,lowerfilt&spellcheck.extendedResults=true&version=2&wt=javabin} hits=25 status=0 QTime=20 
   [junit4]   2> 265939 T729 C167 oashc.SpellCheckComponent.finishStage [ff01::083]:33332/|https://127.0.0.1:43448/ null
   [junit4]   2> 265940 T729 C167 oasc.SolrCore.execute [collection1] webapp= path=/select params={q=*:*&shards.qt=spellCheckCompRH_Direct&qt=spellCheckCompRH_Direct&shards=[ff01::083]:33332/|https://127.0.0.1:43448/&spellcheck.count=4&spellcheck=true&spellcheck.q=bluo&fl=id,lowerfilt&spellcheck.extendedResults=true&version=2&wt=javabin} hits=25 status=0 QTime=21 
   [junit4]   2> 265951 T699 C167 oasc.SolrCore.execute [collection1] webapp= path=/select params={fsv=true&q=*:*&distrib=false&spellcheck.dictionary=direct&shards.qt=spellCheckCompRH_Direct&qt=spellCheckCompRH_Direct&shard.url=[ff01::083]:33332/|https://127.0.0.1:43448/&spellcheck.count=5&spellcheck.q=bluo&spellcheck.onlyMorePopular=false&fl=id,score&spellcheck.extendedResults=true&rows=10&version=2&wt=javabin&start=0&NOW=1411591453769&spellcheck=true&isShard=true} hits=25 status=0 QTime=2 
   [junit4]   2> 265956 T697 C167 oasc.SolrCore.execute [collection1] webapp= path=/select params={fsv=true&q=*:*&distrib=false&spellcheck.dictionary=direct&shards.qt=spellCheckCompRH_Direct&qt=spellCheckCompRH_Direct&shard.url=[ff01::083]:33332/|https://127.0.0.1:43448/&spellcheck.count=5&spellcheck.q=bluo&spellcheck.onlyMorePopular=false&fl=id,score&spellcheck.extendedResults=true&rows=10&version=2&wt=javabin&start=0&NOW=1411591453771&spellcheck=true&isShard=true} hits=25 status=0 QTime=3 
   [junit4]   2> 265956 T698 C167 oasc.SolrCore.execute [collection1] webapp= path=/select params={fsv=true&q=*:*&distrib=false&spellcheck.dictionary=direct&shards.qt=spellCheckCompRH_Direct&qt=spellCheckCompRH_Direct&shard.url=[ff01::083]:33332/|https://127.0.0.1:43448/&spellcheck.count=5&spellcheck.q=bluo&spellcheck.onlyMorePopular=false&fl=id,score&spellcheck.extendedResults=true&rows=10&version=2&wt=javabin&start=0&NOW=1411591453773&spellcheck=true&isShard=true} hits=25 status=0 QTime=3 
   [junit4]   2> 265958 T628 C167 oasc.SolrCore.execute [collection1] webapp= path=/select params={q=*:*&distrib=false&spellcheck.dictionary=direct&shards.qt=spellCheckCompRH_Direct&qt=spellCheckCompRH_Direct&shard.url=[ff01::083]:33332/|https://127.0.0.1:43448/&spellcheck.count=4&spellcheck.q=bluo&spellcheck.onlyMorePopular=false&fl=id,lowerfilt&spellcheck.extendedResults=true&version=2&wt=javabin&ids=1,7,10,2,3,5,8,4,6,9&NOW=1411591453769&spellcheck=false&isShard=true} status=0 QTime=0 
   [junit4]   2> 265960 T739 C167 oashc.SpellCheckComponent.finishStage [ff01::083]:33332/|https://127.0.0.1:43448/ {suggestions={bluo={numFound=1,startOffset=0,endOffset=4,origFreq=0,suggestion=[{word=blue,freq=1}]}},correctlySpelled=true,originalTerms=[bluo]}
   [junit4]   2> 265960 T739 C167 oashc.SpellCheckComponent.finishStage [ff01::083]:33332/|https://127.0.0.1:43448/ null
   [junit4]   2> 265960 T739 C167 oasc.SolrCore.execute [collection1] webapp= path=/select params={q=*:*&shards.qt=spellCheckCompRH_Direct&qt=spellCheckCompRH_Direct&shards=[ff01::083]:33332/|https://127.0.0.1:43448/&spellcheck.count=4&spellcheck=true&spellcheck.q=bluo&fl=id,lowerfilt&spellcheck.extendedResults=true&version=2&wt=javabin} hits=25 status=0 QTime=16 
   [junit4]   2> 265965 T634 C167 oasc.SolrCore.execute [collection1] webapp= path=/select params={q=*:*&distrib=false&spellcheck.dictionary=direct&shards.qt=spellCheckCompRH_Direct&qt=spellCheckCompRH_Direct&shard.url=[ff01::083]:33332/|https://127.0.0.1:43448/&spellcheck.count=4&spellcheck.q=bluo&spellcheck.onlyMorePopular=false&fl=id,lowerfilt&spellcheck.extendedResults=true&version=2&wt=javabin&ids=1,7,10,2,3,5,8,4,6,9&NOW=1411591453771&spellcheck=false&isShard=true} status=0 QTime=1 
   [junit4]   2> 265965 T636 C167 oasc.SolrCore.execute [collection1] webapp= path=/select params={q=*:*&distrib=false&spellcheck.dictionary=direct&shards.qt=spellCheckCompRH_Direct&qt=spellCheckCompRH_Direct&shard.url=[ff01::083]:33332/|https://127.0.0.1:43448/&spellcheck.count=4&spellcheck.q=bluo&spellcheck.onlyMorePopular=false&fl=id,lowerfilt&spellcheck.extendedResults=true&version=2&wt=javabin&ids=1,7,10,2,3,5,8,4,6,9&NOW=1411591453773&spellcheck=false&isShard=true} status=0 QTime=0 
   [junit4]   2> 265967 T740 C167 oashc.SpellCheckComponent.finishStage [ff01::083]:33332/|https://127.0.0.1:43448/ {suggestions={bluo={numFound=1,startOffset=0,endOffset=4,origFreq=0,suggestion=[{word=blue,freq=1}]}},correctlySpelled=true,originalTerms=[bluo]}
   [junit4]   2> 265967 T740 C167 oashc.SpellCheckComponent.finishStage [ff01::083]:33332/|https://127.0.0.1:43448/ null
   [junit4]   2> 265967 T742 C167 oashc.SpellCheckComponent.finishStage [ff01::083]:33332/|https://127.0.0.1:43448/ {suggestions={bluo={numFound=1,startOffset=0,endOffset=4,origFreq=0,suggestion=[{word=blue,freq=1}]}},correctlySpelled=true,originalTerms=[bluo]}
   [junit4]   2> 265967 T740 C167 oasc.SolrCore.execute [collection1] webapp= path=/select params={q=*:*&shards.qt=spellCheckCompRH_Direct&qt=spellCheckCompRH_Direct&shards=[ff01::083]:33332/|https://127.0.0.1:43448/&spellcheck.count=4&spellcheck=true&spellcheck.q=bluo&fl=id,lowerfilt&spellcheck.extendedResults=true&version=2&wt=javabin} hits=25 status=0 QTime=21 
   [junit4]   2> 265968 T742 C167 oashc.SpellCheckComponent.finishStage [ff01::083]:33332/|https://127.0.0.1:43448/ null
   [junit4]   2> 265968 T742 C167 oasc.SolrCor

[...truncated too long message...]

992141BC9E4E-001/tempDir-001/shard2/data/index;done=false>>]
   [junit4]   2> 425962 T500 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.component.DistributedSpellCheckComponentTest-4382992141BC9E4E-001/tempDir-001/shard2/data/index
   [junit4]   2> 425962 T500 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.component.DistributedSpellCheckComponentTest-4382992141BC9E4E-001/tempDir-001/shard2/data [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.component.DistributedSpellCheckComponentTest-4382992141BC9E4E-001/tempDir-001/shard2/data;done=false>>]
   [junit4]   2> 425963 T500 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.component.DistributedSpellCheckComponentTest-4382992141BC9E4E-001/tempDir-001/shard2/data
   [junit4]   2> 425970 T500 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
   [junit4]   2> 430957 T500 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=DistributedSpellCheckComponentTest -Dtests.method=testDistribSearch -Dtests.seed=4382992141BC9E4E -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=ar_IQ -Dtests.timezone=Indian/Mayotte -Dtests.file.encoding=US-ASCII
   [junit4] ERROR    170s J0 | DistributedSpellCheckComponentTest.testDistribSearch <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=2263, name=Thread-1076, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([4382992141BC9E4E:C264173936E3FE72]:0)
   [junit4]    > Caused by: java.lang.RuntimeException: org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: https://127.0.0.1:44787
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([4382992141BC9E4E]:0)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase$5.run(BaseDistributedSearchTestCase.java:580)
   [junit4]    > Caused by: org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: https://127.0.0.1:44787
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:558)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
   [junit4]    > 	at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:91)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase$5.run(BaseDistributedSearchTestCase.java:575)
   [junit4]    > Caused by: java.net.SocketTimeoutException: Read timed out
   [junit4]    > 	at java.net.SocketInputStream.socketRead0(Native Method)
   [junit4]    > 	at java.net.SocketInputStream.read(SocketInputStream.java:152)
   [junit4]    > 	at java.net.SocketInputStream.read(SocketInputStream.java:122)
   [junit4]    > 	at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
   [junit4]    > 	at sun.security.ssl.InputRecord.read(InputRecord.java:480)
   [junit4]    > 	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927)
   [junit4]    > 	at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884)
   [junit4]    > 	at sun.security.ssl.AppInputStream.read(AppInputStream.java:102)
   [junit4]    > 	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
   [junit4]    > 	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
   [junit4]    > 	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
   [junit4]    > 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
   [junit4]    > 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
   [junit4]    > 	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:260)
   [junit4]    > 	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
   [junit4]    > 	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
   [junit4]    > 	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
   [junit4]    > 	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:271)
   [junit4]    > 	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
   [junit4]    > 	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:682)
   [junit4]    > 	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:486)
   [junit4]    > 	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
   [junit4]    > 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
   [junit4]    > 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
   [junit4]    > 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:448)
   [junit4]    > 	... 5 more
   [junit4]   2> 430970 T500 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50): {intDefault=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), lowerfilt1and2=PostingsFormat(name=Asserting), end2=PostingsFormat(name=Lucene41VarGapFixedInterval), id=PostingsFormat(name=Lucene41VarGapFixedInterval), gram1=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), start3=PostingsFormat(name=Lucene41VarGapFixedInterval), multiDefault=PostingsFormat(name=Asserting), gram4=PostingsFormat(name=Direct), range_facet_l=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), gram3=PostingsFormat(name=Asserting), start1=PostingsFormat(name=Direct), word=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), start4=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), end3=PostingsFormat(name=Asserting), start2=PostingsFormat(name=Asserting), gram2=PostingsFormat(name=Lucene41VarGapFixedInterval), end1=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), lowerfilt=PostingsFormat(name=Asserting), timestamp=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), end4=PostingsFormat(name=Direct)}, docValues:{timestamp=DocValuesFormat(name=Lucene410)}, sim=RandomSimilarityProvider(queryNorm=false,coord=no): {}, locale=ar_IQ, timezone=Indian/Mayotte
   [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_65 (64-bit)/cpus=16,threads=1,free=106621488,total=343932928
   [junit4]   2> NOTE: All tests run in this JVM: [TestFieldTypeResource, CoreMergeIndexesAdminHandlerTest, TestAnalyzedSuggestions, SSLMigrationTest, TestNRTOpen, IndexSchemaRuntimeFieldTest, TestRangeQuery, AsyncMigrateRouteKeyTest, TestRandomFaceting, TestCoreContainer, ResourceLoaderTest, DistributedSpellCheckComponentTest]
   [junit4] Completed on J0 in 169.90s, 1 test, 1 error <<< FAILURES!

[...truncated 1184 lines...]
BUILD FAILED
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:498: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:471: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:60: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/extra-targets.xml:39: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build.xml:189: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/common-build.xml:496: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:1359: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:962: There were test failures: 429 suites, 1775 tests, 1 error, 55 ignored (11 assumptions)

Total time: 174 minutes 49 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-NightlyTests-5.x #628
Archived 5 artifacts
Archive block size is 32768
Received 0 blocks and 644943698 bytes
Compression is 0.0%
Took 1 min 56 sec
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure