You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2013/11/15 00:56:01 UTC

[JENKINS] Lucene-Solr-4.x-Linux (32bit/jrockit-jdk1.6.0_45-R28.2.7-4.1.0) - Build # 8222 - Still Failing!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/8222/
Java: 32bit/jrockit-jdk1.6.0_45-R28.2.7-4.1.0 -XnoOpt

1 tests failed.
REGRESSION:  org.apache.lucene.index.TestIndexWriterReader.testTooManySegments

Error Message:


Stack Trace:
java.lang.AssertionError
	at __randomizedtesting.SeedInfo.seed([92DE792121089547:D44DB20BF8EF2D94]:0)
	at org.junit.Assert.fail(Assert.java:92)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertTrue(Assert.java:54)
	at org.apache.lucene.index.TestIndexWriterReader.testTooManySegments(TestIndexWriterReader.java:1149)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:774)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:683)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:44)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:56)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:662)




Build Log:
[...truncated 256 lines...]
   [junit4] Suite: org.apache.lucene.index.TestIndexWriterReader
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestIndexWriterReader -Dtests.method=testTooManySegments -Dtests.seed=92DE792121089547 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=mk -Dtests.timezone=Europe/Amsterdam -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 6.42s J0 | TestIndexWriterReader.testTooManySegments <<<
   [junit4]    > Throwable #1: java.lang.AssertionError
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([92DE792121089547:D44DB20BF8EF2D94]:0)
   [junit4]    > 	at org.apache.lucene.index.TestIndexWriterReader.testTooManySegments(TestIndexWriterReader.java:1149)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:662)
   [junit4]   2> NOTE: test params are: codec=Appending, sim=DefaultSimilarity, locale=mk, timezone=Europe/Amsterdam
   [junit4]   2> NOTE: Linux 3.8.0-32-generic i386/Oracle Corporation 1.6.0_45 (32-bit)/cpus=8,threads=1,free=28777776,total=80531456
   [junit4]   2> NOTE: All tests run in this JVM: [TestIndexWriterReader]
   [junit4] Completed on J0 in 25.62s, 23 tests, 1 failure <<< FAILURES!

[...truncated 1321 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:429: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:409: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/build.xml:49: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1276: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:912: There were test failures: 379 suites, 2758 tests, 1 failure, 105 ignored (92 assumptions)

Total time: 24 minutes 21 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jrockit-jdk1.6.0_45-R28.2.7-4.1.0 -XnoOpt
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.7.0_45) - Build # 8224 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/8224/
Java: 32bit/jdk1.7.0_45 -server -XX:+UseConcMarkSweepGC

4 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.DeleteReplicaTest

Error Message:
ERROR: SolrZkClient opens=15 closes=14

Stack Trace:
java.lang.AssertionError: ERROR: SolrZkClient opens=15 closes=14
	at __randomizedtesting.SeedInfo.seed([423FCF3C0998C092]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.SolrTestCaseJ4.endTrackingZkClients(SolrTestCaseJ4.java:343)
	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:135)
	at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:700)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.DeleteReplicaTest

Error Message:
2 threads leaked from SUITE scope at org.apache.solr.cloud.DeleteReplicaTest:     1) Thread[id=2958, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[423FCF3C0998C092]-SendThread(localhost.localdomain:38268), state=TIMED_WAITING, group=TGRP-DeleteReplicaTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)    2) Thread[id=2959, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[423FCF3C0998C092]-EventThread, state=WAITING, group=TGRP-DeleteReplicaTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.DeleteReplicaTest: 
   1) Thread[id=2958, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[423FCF3C0998C092]-SendThread(localhost.localdomain:38268), state=TIMED_WAITING, group=TGRP-DeleteReplicaTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
   2) Thread[id=2959, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[423FCF3C0998C092]-EventThread, state=WAITING, group=TGRP-DeleteReplicaTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
	at __randomizedtesting.SeedInfo.seed([423FCF3C0998C092]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.DeleteReplicaTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=2958, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[423FCF3C0998C092]-SendThread(localhost.localdomain:38268), state=TIMED_WAITING, group=TGRP-DeleteReplicaTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=2958, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[423FCF3C0998C092]-SendThread(localhost.localdomain:38268), state=TIMED_WAITING, group=TGRP-DeleteReplicaTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)
	at __randomizedtesting.SeedInfo.seed([423FCF3C0998C092]:0)


REGRESSION:  org.apache.solr.cloud.DeleteReplicaTest.testDistribSearch

Error Message:
Invalid replica : core_node3 in shard/collection : shard2/delLiveColl available replicas are core_node4

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Invalid replica : core_node3 in shard/collection : shard2/delLiveColl available replicas are core_node4
	at __randomizedtesting.SeedInfo.seed([423FCF3C0998C092:C3D941247EC7A0AE]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:464)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:181)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:268)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:640)
	at org.apache.solr.cloud.DeleteReplicaTest.removeAndWaitForReplicaGone(DeleteReplicaTest.java:154)
	at org.apache.solr.cloud.DeleteReplicaTest.deleteLiveReplicaTest(DeleteReplicaTest.java:141)
	at org.apache.solr.cloud.DeleteReplicaTest.doTest(DeleteReplicaTest.java:115)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)




Build Log:
[...truncated 10954 lines...]
   [junit4] Suite: org.apache.solr.cloud.DeleteReplicaTest
   [junit4]   2> 1241421 T2872 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /mdkl/v
   [junit4]   2> 1241424 T2872 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-DeleteReplicaTest-1384484419524
   [junit4]   2> 1241425 T2872 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 1241425 T2873 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1241525 T2872 oasc.ZkTestServer.run start zk server on port:38268
   [junit4]   2> 1241526 T2872 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1241530 T2879 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@76b842 name:ZooKeeperConnection Watcher:127.0.0.1:38268 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1241530 T2872 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1241531 T2872 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1241533 T2872 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1241534 T2881 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6f5e2 name:ZooKeeperConnection Watcher:127.0.0.1:38268/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1241534 T2872 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1241535 T2872 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 1241537 T2872 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 1241539 T2872 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 1241540 T2872 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 1241542 T2872 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1241542 T2872 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1241546 T2872 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 1241546 T2872 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 1241548 T2872 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1241549 T2872 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1241550 T2872 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1241551 T2872 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1241553 T2872 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1241553 T2872 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1241555 T2872 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1241555 T2872 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 1241557 T2872 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1241557 T2872 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1241559 T2872 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1241560 T2872 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1241562 T2872 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1241563 T2872 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1241565 T2872 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1241566 T2872 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1241568 T2872 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1241568 T2872 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1241628 T2872 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1241632 T2872 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:41326
   [junit4]   2> 1241633 T2872 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1241633 T2872 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1241633 T2872 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1384484419669
   [junit4]   2> 1241633 T2872 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1384484419669/'
   [junit4]   2> 1241654 T2872 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1384484419669/solr.xml
   [junit4]   2> 1241691 T2872 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 1241692 T2872 oasc.CoreContainer.<init> New CoreContainer 8223867
   [junit4]   2> 1241692 T2872 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1384484419669/]
   [junit4]   2> 1241693 T2872 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1241693 T2872 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1241693 T2872 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1241694 T2872 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1241694 T2872 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1241694 T2872 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1241695 T2872 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1241695 T2872 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1241695 T2872 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1241696 T2872 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1241696 T2872 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1241697 T2872 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1241697 T2872 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38268/solr
   [junit4]   2> 1241697 T2872 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1241698 T2872 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1241700 T2892 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@15e31e5 name:ZooKeeperConnection Watcher:127.0.0.1:38268 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1241700 T2872 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1241702 T2872 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1241703 T2894 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@70b955 name:ZooKeeperConnection Watcher:127.0.0.1:38268/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1241703 T2872 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1241705 T2872 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1241707 T2872 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1241709 T2872 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1241710 T2872 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41326_mdkl%2Fv
   [junit4]   2> 1241711 T2872 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:41326_mdkl%2Fv
   [junit4]   2> 1241713 T2872 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1241716 T2872 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1241719 T2872 oasc.Overseer.start Overseer (id=90733570918055939-127.0.0.1:41326_mdkl%2Fv-n_0000000000) starting
   [junit4]   2> 1241724 T2872 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1241727 T2896 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1241727 T2872 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1241729 T2872 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1241730 T2872 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1241732 T2895 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1241734 T2872 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1384484419669
   [junit4]   2> 1241735 T2872 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1384484419669/conf/
   [junit4]   2> 1241761 T2872 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1241762 T2872 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 1241763 T2872 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1241764 T2872 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1241765 T2899 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e56831 name:ZooKeeperConnection Watcher:127.0.0.1:38268/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1241766 T2872 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1241767 T2872 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1241768 T2872 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1241827 T2872 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1241828 T2872 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:59732
   [junit4]   2> 1241829 T2872 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1241829 T2872 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1241829 T2872 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1384484419868
   [junit4]   2> 1241830 T2872 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.DeleteReplicaTest-jetty1-1384484419868/'
   [junit4]   2> 1241842 T2872 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1384484419868/solr.xml
   [junit4]   2> 1241889 T2872 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 1241890 T2872 oasc.CoreContainer.<init> New CoreContainer 27120754
   [junit4]   2> 1241890 T2872 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1384484419868/]
   [junit4]   2> 1241891 T2872 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1241892 T2872 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1241892 T2872 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1241892 T2872 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1241892 T2872 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1241893 T2872 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1241893 T2872 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1241893 T2872 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1241893 T2872 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1241894 T2872 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1241895 T2872 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1241895 T2872 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1241895 T2872 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38268/solr
   [junit4]   2> 1241896 T2872 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1241896 T2872 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1241898 T2910 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1700669 name:ZooKeeperConnection Watcher:127.0.0.1:38268 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1241898 T2872 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1241900 T2872 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1241901 T2912 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ec4066 name:ZooKeeperConnection Watcher:127.0.0.1:38268/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1241901 T2872 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1241907 T2872 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1242912 T2872 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59732_mdkl%2Fv
   [junit4]   2> 1242913 T2872 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:59732_mdkl%2Fv
   [junit4]   2> 1242916 T2894 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1242916 T2899 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1242916 T2912 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1242920 T2872 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1384484419868
   [junit4]   2> 1242921 T2872 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1384484419868/conf/
   [junit4]   2> 1242941 T2872 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1242941 T2872 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 1242941 T2872 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1243000 T2872 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1243002 T2872 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:49670
   [junit4]   2> 1243003 T2872 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1243003 T2872 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1243003 T2872 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.DeleteReplicaTest-jetty2-1384484421041
   [junit4]   2> 1243004 T2872 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.DeleteReplicaTest-jetty2-1384484421041/'
   [junit4]   2> 1243015 T2872 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty2-1384484421041/solr.xml
   [junit4]   2> 1243052 T2872 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 1243052 T2872 oasc.CoreContainer.<init> New CoreContainer 27567373
   [junit4]   2> 1243053 T2872 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.DeleteReplicaTest-jetty2-1384484421041/]
   [junit4]   2> 1243053 T2872 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1243054 T2872 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1243054 T2872 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1243054 T2872 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1243055 T2872 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1243055 T2872 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1243055 T2872 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1243055 T2872 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1243056 T2872 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1243057 T2872 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1243057 T2872 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1243057 T2872 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1243058 T2872 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38268/solr
   [junit4]   2> 1243058 T2872 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1243059 T2872 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1243061 T2924 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@17bc240 name:ZooKeeperConnection Watcher:127.0.0.1:38268 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1243061 T2872 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1243063 T2872 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1243064 T2926 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1dbb468 name:ZooKeeperConnection Watcher:127.0.0.1:38268/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1243064 T2872 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1243067 T2872 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1244069 T2872 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49670_mdkl%2Fv
   [junit4]   2> 1244071 T2872 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49670_mdkl%2Fv
   [junit4]   2> 1244074 T2899 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1244074 T2894 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1244074 T2926 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1244074 T2912 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1244085 T2872 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty2-1384484421041
   [junit4]   2> 1244086 T2872 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty2-1384484421041/conf/
   [junit4]   2> 1244107 T2872 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1244107 T2872 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 1244108 T2872 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1244172 T2872 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1244174 T2872 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:42492
   [junit4]   2> 1244174 T2872 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1244175 T2872 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1244175 T2872 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1384484422208
   [junit4]   2> 1244175 T2872 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1384484422208/'
   [junit4]   2> 1244187 T2872 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1384484422208/solr.xml
   [junit4]   2> 1244223 T2872 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 1244223 T2872 oasc.CoreContainer.<init> New CoreContainer 25600933
   [junit4]   2> 1244224 T2872 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1384484422208/]
   [junit4]   2> 1244224 T2872 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1244225 T2872 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1244225 T2872 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1244225 T2872 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1244226 T2872 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1244226 T2872 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1244226 T2872 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1244226 T2872 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1244227 T2872 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1244228 T2872 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1244228 T2872 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1244228 T2872 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1244229 T2872 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38268/solr
   [junit4]   2> 1244229 T2872 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1244230 T2872 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1244231 T2938 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@628c4c name:ZooKeeperConnection Watcher:127.0.0.1:38268 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1244231 T2872 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1244233 T2872 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1244234 T2940 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@c520e name:ZooKeeperConnection Watcher:127.0.0.1:38268/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1244235 T2872 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1244238 T2872 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1245241 T2872 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42492_mdkl%2Fv
   [junit4]   2> 1245242 T2872 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:42492_mdkl%2Fv
   [junit4]   2> 1245245 T2899 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1245246 T2926 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1245246 T2940 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1245246 T2912 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1245246 T2894 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1245250 T2872 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1384484422208
   [junit4]   2> 1245251 T2872 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1384484422208/conf/
   [junit4]   2> 1245276 T2872 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1245277 T2872 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 1245277 T2872 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1245342 T2872 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1245344 T2872 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:39331
   [junit4]   2> 1245345 T2872 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1245345 T2872 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1245345 T2872 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1384484423377
   [junit4]   2> 1245346 T2872 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1384484423377/'
   [junit4]   2> 1245357 T2872 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1384484423377/solr.xml
   [junit4]   2> 1245399 T2872 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 1245400 T2872 oasc.CoreContainer.<init> New CoreContainer 17942655
   [junit4]   2> 1245400 T2872 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1384484423377/]
   [junit4]   2> 1245401 T2872 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1245402 T2872 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1245402 T2872 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1245402 T2872 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1245403 T2872 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1245403 T2872 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1245403 T2872 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1245403 T2872 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1245404 T2872 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1245405 T2872 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1245405 T2872 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1245405 T2872 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1245405 T2872 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38268/solr
   [junit4]   2> 1245406 T2872 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1245406 T2872 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1245408 T2952 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e9e887 name:ZooKeeperConnection Watcher:127.0.0.1:38268 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1245408 T2872 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1245410 T2872 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1245411 T2954 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@9ee27a name:ZooKeeperConnection Watcher:127.0.0.1:38268/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1245411 T2872 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1245414 T2872 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1246417 T2872 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39331_mdkl%2Fv
   [junit4]   2> 1246418 T2872 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:39331_mdkl%2Fv
   [junit4]   2> 1246422 T2926 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1246422 T2940 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1246423 T2899 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1246423 T2954 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1246423 T2894 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1246422 T2912 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1246427 T2872 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1384484423377
   [junit4]   2> 1246427 T2872 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1384484423377/conf/
   [junit4]   2> 1246454 T2872 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1246454 T2872 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 1246455 T2872 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1246456 T2872 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1246457 T2957 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a1bb1d name:ZooKeeperConnection Watcher:127.0.0.1:38268/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1246458 T2872 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1246459 T2872 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1246460 T2872 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1246461 T2959 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c267e2 name:ZooKeeperConnection Watcher:127.0.0.1:38268/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1246461 T2872 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1246462 T2872 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1246465 T2933 oasha.CollectionsHandler.handleCreateAction Creating Collection : numShards=2&maxShardsPerNode=2&name=delLiveColl&replicationFactor=2&action=CREATE&wt=javabin&version=2
   [junit4]   2> 1246466 T2894 oasc.DistributedQueue$LatchChildWatcher.process Watcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1246466 T2894 oasc.DistributedQueue$LatchChildWatcher.process Watcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1246466 T2896 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"delLiveColl",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "maxShardsPerNode":"2"}
   [junit4]   2> 1246467 T2896 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : createcollection , {
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"delLiveColl",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "maxShardsPerNode":"2"}
   [junit4]   2> 1247741 T2895 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1247741 T2895 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: delLiveColl
   [junit4]   2> 1247741 T2895 oasc.Overseer$ClusterStateUpdater.createCollection Create collection delLiveColl with shards [shard1, shard2]
   [junit4]   2> 1247745 T2954 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1247745 T2894 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1247745 T2899 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1247745 T2959 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1247745 T2912 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1247745 T2926 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1247745 T2940 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1247745 T2957 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1247770 T2896 oasc.OverseerCollectionProcessor.createCollection going to create cores replicas shardNames [shard1, shard2] , repFactor : 2
   [junit4]   2> 1247770 T2896 oasc.OverseerCollectionProcessor.createCollection Creating shard delLiveColl_shard1_replica1 as part of slice shard1 of collection delLiveColl on 127.0.0.1:59732_mdkl%2Fv
   [junit4]   2> 1247771 T2896 oasc.OverseerCollectionProcessor.createCollection Creating shard delLiveColl_shard1_replica2 as part of slice shard1 of collection delLiveColl on 127.0.0.1:39331_mdkl%2Fv
   [junit4]   2> 1247771 T2896 oasc.OverseerCollectionProcessor.createCollection Creating shard delLiveColl_shard2_replica1 as part of slice shard2 of collection delLiveColl on 127.0.0.1:49670_mdkl%2Fv
   [junit4]   2> 1247772 T2896 oasc.OverseerCollectionProcessor.createCollection Creating shard delLiveColl_shard2_replica2 as part of slice shard2 of collection delLiveColl on 127.0.0.1:41326_mdkl%2Fv
   [junit4]   2> 1247773 T2908 oasc.ZkController.publish publishing core=delLiveColl_shard1_replica1 state=down
   [junit4]   2> 1247773 T2947 oasc.ZkController.publish publishing core=delLiveColl_shard1_replica2 state=down
   [junit4]   2> 1247773 T2920 oasc.ZkController.publish publishing core=delLiveColl_shard2_replica1 state=down
   [junit4]   2> 1247774 T2908 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1247774 T2947 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1247774 T2889 oasc.ZkController.publish publishing core=delLiveColl_shard2_replica2 state=down
   [junit4]   2> 1247774 T2920 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1247776 T2889 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1249254 T2895 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1249255 T2895 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:59732/mdkl/v",
   [junit4]   2> 	  "core":"delLiveColl_shard1_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:59732_mdkl%2Fv",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"delLiveColl",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1249257 T2895 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:39331/mdkl/v",
   [junit4]   2> 	  "core":"delLiveColl_shard1_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:39331_mdkl%2Fv",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"delLiveColl",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1249259 T2895 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:49670/mdkl/v",
   [junit4]   2> 	  "core":"delLiveColl_shard2_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:49670_mdkl%2Fv",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"delLiveColl",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1249262 T2895 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:41326/mdkl/v",
   [junit4]   2> 	  "core":"delLiveColl_shard2_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:41326_mdkl%2Fv",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"delLiveColl",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1249264 T2940 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1249264 T2954 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1249264 T2957 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1249265 T2894 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1249264 T2912 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1249264 T2899 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1249264 T2926 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1249264 T2959 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1249775 T2908 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for delLiveColl_shard1_replica1
   [junit4]   2> 1249775 T2947 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for delLiveColl_shard1_replica2
   [junit4]   2> 1249775 T2920 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for delLiveColl_shard2_replica1
   [junit4]   2> 1249776 T2908 oasc.CoreContainer.create Creating SolrCore 'delLiveColl_shard1_replica1' using instanceDir: ./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1384484419868/delLiveColl_shard1_replica1
   [junit4]   2> 1249776 T2920 oasc.CoreContainer.create Creating SolrCore 'delLiveColl_shard2_replica1' using instanceDir: ./org.apache.solr.cloud.DeleteReplicaTest-jetty2-1384484421041/delLiveColl_shard2_replica1
   [junit4]   2> 1249777 T2920 oasc.ZkController.createCollectionZkNode Check for collection zkNode:delLiveColl
   [junit4]   2> 1249776 T2889 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for delLiveColl_shard2_replica2
   [junit4]   2> 1249776 T2947 oasc.CoreContainer.create Creating SolrCore 'delLiveColl_shard1_replica2' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1384484423377/delLiveColl_shard1_replica2
   [junit4]   2> 1249777 T2947 oasc.ZkController.createCollectionZkNode Check for collection zkNode:delLiveColl
   [junit4]   2> 1249777 T2920 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:delLiveColl
   [junit4]   2> 1249776 T2908 oasc.ZkController.createCollectionZkNode Check for collection zkNode:delLiveColl
   [junit4]   2> 1249778 T2947 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:delLiveColl
   [junit4]   2> 1249778 T2920 oasc.ZkController.getConfName Looking for collection configName
   [junit4]   2> 1249778 T2889 oasc.CoreContainer.create Creating SolrCore 'delLiveColl_shard2_replica2' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1384484419669/delLiveColl_shard2_replica2
   [junit4]   2> 1249779 T2908 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:delLiveColl
   [junit4]   2> 1249779 T2947 oasc.ZkController.getConfName Looking for collection configName
   [junit4]   2> 1249779 T2908 oasc.ZkController.getConfName Looking for collection configName
   [junit4]   2> 1249779 T2889 oasc.ZkController.createCollectionZkNode Check for collection zkNode:delLiveColl
   [junit4]   2> 1249780 T2920 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
   [junit4]   2> 1249780 T2947 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
   [junit4]   2> 1249780 T2889 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:delLiveColl
   [junit4]   2> 1249780 T2920 oascc.SolrZkClient.makePath makePath: /collections/delLiveColl
   [junit4]   2> 1249781 T2889 oasc.ZkController.getConfName Looking for collection configName
   [junit4]   2> 1249781 T2908 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
   [junit4]   2> 1249781 T2947 oascc.SolrZkClient.makePath makePath: /collections/delLiveColl
   [junit4]   2> 1249782 T2908 oascc.SolrZkClient.makePath makePath: /collections/delLiveColl
   [junit4]   2> 1249782 T2889 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
   [junit4]   2> 1249783 T2889 oascc.SolrZkClient.makePath makePath: /collections/delLiveColl
   [junit4]   2> 1249783 T2920 oasc.ZkController.readConfigName Load collection config from:/collections/delLiveColl
   [junit4]   2> 1249784 T2908 oasc.ZkController.readConfigName Load collection config from:/collections/delLiveColl
   [junit4]   2> 1249784 T2947 oasc.ZkController.readConfigName Load collection config from:/collections/delLiveColl
   [junit4]   2> 1249794 T2889 oasc.ZkController.readConfigName Load collection config from:/collections/delLiveColl
   [junit4]   2> 1249794 T2920 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.DeleteReplicaTest-jetty2-1384484421041/delLiveColl_shard2_replica1/'
   [junit4]   2> 1249795 T2908 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.DeleteReplicaTest-jetty1-1384484419868/delLiveColl_shard1_replica1/'
   [junit4]   2> 1249796 T2889 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1384484419669/delLiveColl_shard2_replica2/'
   [junit4]   2> 1249797 T2947 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1384484423377/delLiveColl_shard1_replica2/'
   [junit4]   2> 1249883 T2920 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 1249883 T2908 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 1249883 T2947 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 1249884 T2889 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 1249960 T2908 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1249960 T2889 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1249960 T2920 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1249961 T2947 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1249962 T2889 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1249962 T2908 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1249962 T2947 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1249962 T2920 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1249968 T2908 oass.IndexSchema.readSchema [delLiveColl_shard1_replica1] Schema name=test
   [junit4]   2> 1249968 T2947 oass.IndexSchema.readSchema [delLiveColl_shard1_replica2] Schema name=test
   [junit4]   2> 1249969 T2889 oass.IndexSchema.readSchema [delLiveColl_shard2_replica2] Schema name=test
   [junit4]   2> 1249978 T2920 oass.IndexSchema.readSchema [delLiveColl_shard2_replica1] Schema name=test
   [junit4]   2> 1250817 T2908 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1250818 T2908 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1250818 T2908 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1250824 T2908 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1250826 T2908 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1250838 T2947 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1250838 T2947 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1250839 T2947 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1250845 T2947 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1250845 T2920 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1250845 T2908 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1250845 T2920 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1250846 T2920 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1250847 T2947 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1250856 T2908 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1250859 T2908 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1250860 T2920 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1250860 T2908 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1250861 T2908 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1250861 T2908 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1250862 T2908 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1250862 T2908 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1250863 T2908 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1250863 T2908 oasc.SolrCore.<init> [delLiveColl_shard1_replica1] Opening new SolrCore at ./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1384484419868/delLiveColl_shard1_replica1/, dataDir=./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1384484419868/delLiveColl_shard1_replica1/data/
   [junit4]   2> 1250863 T2920 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1250863 T2908 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1e8c82f
   [junit4]   2> 1250864 T2908 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1384484419868/delLiveColl_shard1_replica1/data
   [junit4]   2> 1250865 T2908 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1384484419868/delLiveColl_shard1_replica1/data/index/
   [junit4]   2> 1250865 T2908 oasc.SolrCore.initIndex WARN [delLiveColl_shard1_replica1] Solr index directory './org.apache.solr.cloud.DeleteReplicaTest-jetty1-1384484419868/delLiveColl_shard1_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1250866 T2908 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1384484419868/delLiveColl_shard1_replica1/data/index
   [junit4]   2> 1250866 T2947 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1250866 T2908 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=26, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.33875771996404735]
   [junit4]   2> 1250867 T2908 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@d0954e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@51ccba),segFN=segments_1,generation=1}
   [junit4]   2> 1250867 T2908 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1250869 T2947 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1250871 T2908 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1250871 T2908 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1250871 T2908 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1250872 T2908 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1250872 T2908 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1250872 T2908 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1250872 T2947 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1250873 T2908 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1250873 T2908 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1250873 T2908 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1250873 T2947 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1250874 T2908 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1250874 T2947 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1250874 T2920 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1250875 T2908 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1250875 T2908 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1250875 T2947 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1250875 T2908 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1250876 T2908 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1250876 T2908 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1250876 T2947 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1250885 T2947 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1250885 T2947 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1250886 T2947 oasc.SolrCore.<init> [delLiveColl_shard1_replica2] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1384484423377/delLiveColl_shard1_replica2/, dataDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1384484423377/delLiveColl_shard1_replica2/data/
   [junit4]   2> 1250886 T2920 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1250886 T2908 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1250886 T2947 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1e8c82f
   [junit4]   2> 1250888 T2947 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1384484423377/delLiveColl_shard1_replica2/data
   [junit4]   2> 1250888 T2947 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1384484423377/delLiveColl_shard1_replica2/data/index/
   [junit4]   2> 1250888 T2947 oasc.SolrCore.initIndex WARN [delLiveColl_shard1_replica2] Solr index directory '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1384484423377/delLiveColl_shard1_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1250889 T2947 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1384484423377/delLiveColl_shard1_replica2/data/index
   [junit4]   2> 1250889 T2908 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1250889 T2947 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=26, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.33875771996404735]
   [junit4]   2> 1250889 T2920 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1250889 T2908 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1250890 T2947 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@10273f9 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1293525),segFN=segments_1,generation=1}
   [junit4]   2> 1250890 T2908 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=33, maxMergeAtOnceExplicit=40, maxMergedSegmentMB=46.333984375, floorSegmentMB=1.44140625, forceMergeDeletesPctAllowed=1.8643109108065081, segmentsPerTier=36.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1250890 T2920 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1250891 T2920 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1250890 T2947 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1250891 T2920 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1250891 T2908 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@d0954e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@51ccba),segFN=segments_1,generation=1}
   [junit4]   2> 1250892 T2889 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1250892 T2920 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1250893 T2920 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1250892 T2908 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1250893 T2920 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1250892 T2889 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1250894 T2920 oasc.SolrCore.<init> [delLiveColl_shard2_replica1] Opening new SolrCore at ./org.apache.solr.cloud.DeleteReplicaTest-jetty2-1384484421041/delLiveColl_shard2_replica1/, dataDir=./org.apache.solr.cloud.DeleteReplicaTest-jetty2-1384484421041/delLiveColl_shard2_replica1/data/
   [junit4]   2> 1250893 T2908 oass.SolrIndexSearcher.<init> Opening Searcher@b687b3 main
   [junit4]   2> 1250894 T2889 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1250894 T2947 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1250894 T2920 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1e8c82f
   [junit4]   2> 1250895 T2947 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1250896 T2920 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.DeleteReplicaTest-jetty2-1384484421041/delLiveColl_shard2_replica1/data
   [junit4]   2> 1250896 T2920 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.DeleteReplicaTest-jetty2-1384484421041/delLiveColl_shard2_replica1/data/index/
   [junit4]   2> 1250896 T2947 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1250896 T2920 oasc.SolrCore.initIndex WARN [delLiveColl_shard2_replica1] Solr index directory './org.apache.solr.cloud.DeleteReplicaTest-jetty2-1384484421041/delLiveColl_shard2_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1250897 T2947 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1250897 T2920 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.DeleteReplicaTest-jetty2-1384484421041/delLiveColl_shard2_replica1/data/index
   [junit4]   2> 1250897 T2964 oasc.SolrCore.registerSearcher [delLiveColl_shard1_replica1] Registered new searcher Searcher@b687b3 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1250898 T2920 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=26, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.33875771996404735]
   [junit4]   2> 1250897 T2947 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1250898 T2947 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1250898 T2920 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@11cbdad lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@105b1cf),segFN=segments_1,generation=1}
   [junit4]   2> 1250899 T2920 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1250899 T2908 oasc.CoreContainer.registerCore registering core: delLiveColl_shard1_replica1
   [junit4]   2> 1250899 T2908 oasc.ZkController.register Register replica - core:delLiveColl_shard1_replica1 address:http://127.0.0.1:59732/mdkl/v collection:delLiveColl shard:shard1
   [junit4]   2> 1250899 T2947 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1250900 T2947 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1250900 T2947 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1250900 T2908 oascc.SolrZkClient.makePath makePath: /collections/delLiveColl/leader_elect/shard1/election
   [junit4]   2> 1250900 T2947 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1250901 T2947 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1250901 T2947 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1250902 T2947 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1250902 T2889 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1250902 T2920 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1250902 T2947 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1250903 T2920 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1250903 T2947 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1250903 T2920 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1250904 T2920 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1250904 T2920 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1250904 T2947 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1250904 T2920 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1250905 T2920 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1250905 T2920 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1250906 T2908 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 1250906 T2889 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1250906 T2920 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1250907 T2920 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1250907 T2947 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1250908 T2920 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1250908 T2908 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Waiting until we see more replicas up for shard shard1: total=2 found=1 timeoutin=179999
   [junit4]   2> 1250908 T2947 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1250908 T2920 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1250909 T2920 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1250909 T2947 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=33, maxMergeAtOnceExplicit=40, maxMergedSegmentMB=46.333984375, floorSegmentMB=1.44140625, forceMergeDeletesPctAllowed=1.8643109108065081, segmentsPerTier=36.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1250909 T2920 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1250910 T2947 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@10273f9 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1293525),segFN=segments_1,generation=1}
   [junit4]   2> 1250910 T2920 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1250910 T2947 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1250911 T2947 oass.SolrIndexSearcher.<init> Opening Searcher@3481b0 main
   [junit4]   2> 1250911 T2920 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1250914 T2920 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1250914 T2920 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1250914 T2965 oasc.SolrCore.registerSearcher [delLiveColl_shard1_replica2] Registered new searcher Searcher@3481b0 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1250923 T2920 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=33, maxMergeAtOnceExplicit=40, maxMergedSegmentMB=46.333984375, floorSegmentMB=1.44140625, forceMergeDeletesPctAllowed=1.8643109108065081, segmentsPerTier=36.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1250923 T2920 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@11cbdad lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@105b1cf),segFN=segments_1,generation=1}
   [junit4]   2> 1250924 T2920 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1250924 T2947 oasc.CoreContainer.registerCore registering core: delLiveColl_shard1_replica2
   [junit4]   2> 1250924 T2920 oass.SolrIndexSearcher.<init> Opening Searcher@1db64d3 main
   [junit4]   2> 1250924 T2947 oasc.ZkController.register Register replica - core:delLiveColl_shard1_replica2 address:http://127.0.0.1:39331/mdkl/v collection:delLiveColl shard:shard1
   [junit4]   2> 1250926 T2889 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1250926 T2966 oasc.SolrCore.registerSearcher [delLiveColl_shard2_replica1] Registered new searcher Searcher@1db64d3 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1250928 T2920 oasc.CoreContainer.registerCore registering core: delLiveColl_shard2_replica1
   [junit4]   2> 1250928 T2920 oasc.ZkController.register Register replica - core:delLiveColl_shard2_replica1 address:http://127.0.0.1:49670/mdkl/v collection:delLiveColl shard:shard2
   [junit4]   2> 1250929 T2920 oascc.SolrZkClient.makePath makePath: /collections/delLiveColl/leader_elect/shard2/election
   [junit4]   2> 1250929 T2889 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1250932 T2889 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1250934 T2889 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1250934 T2889 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1250934 T2920 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 1250934 T2889 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1250935 T2889 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1250936 T2889 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1250936 T2889 oas

[...truncated too long message...]

r.next WARN Unexpected exception java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
   [junit4]   2> 
   [junit4]   2> 1347737 T2958 oaz.ClientCnxn$SendThread.run WARN Session 0x14259b3b7c6000e for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1348237 T2958 oaz.ClientCnxn$SendThread.run WARN Unexpected exception java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)
   [junit4]   2> 
   [junit4]   2> 1348737 T2958 oazc.StaticHostProvider.next WARN Unexpected exception java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
   [junit4]   2> 
   [junit4]   2> 1348738 T2958 oaz.ClientCnxn$SendThread.run WARN Session 0x14259b3b7c6000e for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 107818 T2871 ccr.ThreadLeakControl.tryToInterruptAll SEVERE There are still zombie threads that couldn't be terminated:
   [junit4]   2> 	   1) Thread[id=2958, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[423FCF3C0998C092]-SendThread(localhost.localdomain:38268), state=TIMED_WAITING, group=TGRP-DeleteReplicaTest]
   [junit4]   2> 	        at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)
   [junit4]   2> NOTE: test params are: codec=Lucene46: {}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=true,coord=yes): {}, locale=ar_KW, timezone=Europe/Skopje
   [junit4]   2> NOTE: Linux 3.8.0-32-generic i386/Oracle Corporation 1.7.0_45 (32-bit)/cpus=8,threads=3,free=110151640,total=265453568
   [junit4]   2> NOTE: All tests run in this JVM: [TestSweetSpotSimilarityFactory, HighlighterTest, PingRequestHandlerTest, TestSolr4Spatial, ShardRoutingCustomTest, TestAnalyzedSuggestions, TermVectorComponentDistributedTest, TestCollapseQParserPlugin, MinimalSchemaTest, TestRandomFaceting, PluginInfoTest, RAMDirectoryFactoryTest, HdfsUnloadDistributedZkTest, DistanceFunctionTest, DirectSolrConnectionTest, DocValuesMultiTest, PrimUtilsTest, TestSolrQueryParserDefaultOperatorResource, SolrPluginUtilsTest, TestUniqueKeyFieldResource, TestRangeQuery, TestDynamicFieldCollectionResource, TestSolrXml, TestUtils, TestQuerySenderNoQuery, TestManagedSchemaFieldResource, TestNumberUtils, HdfsSyncSliceTest, TestFastWriter, TestQuerySenderListener, RequestHandlersTest, TestFastLRUCache, TestFoldingMultitermQuery, TestZkChroot, TestFastOutputStream, MultiTermTest, TimeZoneUtilsTest, SolrTestCaseJ4Test, SolrIndexConfigTest, SpellingQueryConverterTest, DateFieldTest, ChaosMonkeySafeLeaderTest, CurrencyFieldOpenExchangeTest, HdfsDirectoryTest, RequiredFieldsTest, TestQueryTypes, TestCloudManagedSchemaAddField, SolrRequestParserTest, TestWordDelimiterFilterFactory, OpenExchangeRatesOrgProviderTest, TestLazyCores, UniqFieldsUpdateProcessorFactoryTest, DirectSolrSpellCheckerTest, TestRecoveryHdfs, TestCSVLoader, HighlighterConfigTest, InfoHandlerTest, BasicDistributedZk2Test, SortByFunctionTest, TestDefaultSimilarityFactory, XmlUpdateRequestHandlerTest, BadIndexSchemaTest, SpellCheckComponentTest, TestAtomicUpdateErrorCases, BlockCacheTest, ZkNodePropsTest, XsltUpdateRequestHandlerTest, HdfsChaosMonkeySafeLeaderTest, CoreAdminCreateDiscoverTest, TestSolrJ, DeleteInactiveReplicaTest, TestPhraseSuggestions, PreAnalyzedFieldTest, QueryResultKeyTest, FileUtilsTest, NotRequiredUniqueKeyTest, TestRTGBase, TestRecovery, HardAutoCommitTest, IndexBasedSpellCheckerTest, DefaultValueUpdateProcessorTest, TestJmxIntegration, SyncSliceTest, LeaderElectionIntegrationTest, BasicZkTest, LeaderElectionTest, TestRandomDVFaceting, TestStressReorder, SolrCoreTest, SolrCmdDistributorTest, QueryElevationComponentTest, ConvertedLegacyTest, TestFiltering, SpellCheckCollatorTest, TestStressLucene, NoCacheHeaderTest, WordBreakSolrSpellCheckerTest, SolrCoreCheckLockOnStartupTest, DebugComponentTest, StandardRequestHandlerTest, TestOmitPositions, TermsComponentTest, TestValueSourceCache, TestIndexingPerformance, TestArbitraryIndexDir, IndexSchemaRuntimeFieldTest, TestCSVResponseWriter, BinaryUpdateRequestHandlerTest, CSVRequestHandlerTest, CoreContainerCoreInitFailuresTest, TestLFUCache, UpdateParamsTest, TestStressRecovery, AssignTest, DeleteReplicaTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=DeleteReplicaTest -Dtests.seed=423FCF3C0998C092 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ar_KW -Dtests.timezone=Europe/Skopje -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   0.00s J1 | DeleteReplicaTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: ERROR: SolrZkClient opens=15 closes=14
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([423FCF3C0998C092]:0)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.endTrackingZkClients(SolrTestCaseJ4.java:343)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:135)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:744)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.DeleteReplicaTest: 
   [junit4]    >    1) Thread[id=2958, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[423FCF3C0998C092]-SendThread(localhost.localdomain:38268), state=TIMED_WAITING, group=TGRP-DeleteReplicaTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
   [junit4]    >    2) Thread[id=2959, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[423FCF3C0998C092]-EventThread, state=WAITING, group=TGRP-DeleteReplicaTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([423FCF3C0998C092]:0)Throwable #3: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=2958, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[423FCF3C0998C092]-SendThread(localhost.localdomain:38268), state=TIMED_WAITING, group=TGRP-DeleteReplicaTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([423FCF3C0998C092]:0)
   [junit4] Completed on J1 in 107.86s, 1 test, 1 failure, 3 errors <<< FAILURES!

[...truncated 127 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:429: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:409: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:489: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1276: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:912: There were test failures: 336 suites, 1478 tests, 3 suite-level errors, 1 error, 230 ignored (6 assumptions)

Total time: 52 minutes 15 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_45 -server -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-4.x-Linux (64bit/jdk1.6.0_45) - Build # 8223 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/8223/
Java: 64bit/jdk1.6.0_45 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC

2 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.core.OpenCloseCoreStressTest

Error Message:
10 threads leaked from SUITE scope at org.apache.solr.core.OpenCloseCoreStressTest:     1) Thread[id=2031, name=commitScheduler-1674-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)    2) Thread[id=2033, name=commitScheduler-1678-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)    3) Thread[id=2061, name=commitScheduler-1690-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)    4) Thread[id=2065, name=commitScheduler-1696-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)    5) Thread[id=2019, name=commitScheduler-1666-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)    6) Thread[id=2063, name=commitScheduler-1693-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)    7) Thread[id=2032, name=commitScheduler-1673-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)    8) Thread[id=2059, name=commitScheduler-1687-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)    9) Thread[id=2057, name=commitScheduler-1685-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)   10) Thread[id=2029, name=commitScheduler-1676-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 10 threads leaked from SUITE scope at org.apache.solr.core.OpenCloseCoreStressTest: 
   1) Thread[id=2031, name=commitScheduler-1674-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
   2) Thread[id=2033, name=commitScheduler-1678-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
   3) Thread[id=2061, name=commitScheduler-1690-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
   4) Thread[id=2065, name=commitScheduler-1696-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
   5) Thread[id=2019, name=commitScheduler-1666-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
   6) Thread[id=2063, name=commitScheduler-1693-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
   7) Thread[id=2032, name=commitScheduler-1673-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
   8) Thread[id=2059, name=commitScheduler-1687-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
   9) Thread[id=2057, name=commitScheduler-1685-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
  10) Thread[id=2029, name=commitScheduler-1676-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
	at __randomizedtesting.SeedInfo.seed([8DFFD4DC74EE63F7]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.core.OpenCloseCoreStressTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=2031, name=commitScheduler-1674-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)    2) Thread[id=2033, name=commitScheduler-1678-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)    3) Thread[id=2061, name=commitScheduler-1690-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)    4) Thread[id=2065, name=commitScheduler-1696-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)    5) Thread[id=2019, name=commitScheduler-1666-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)    6) Thread[id=2063, name=commitScheduler-1693-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)    7) Thread[id=2032, name=commitScheduler-1673-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)    8) Thread[id=2059, name=commitScheduler-1687-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)    9) Thread[id=2057, name=commitScheduler-1685-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)   10) Thread[id=2029, name=commitScheduler-1676-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=2031, name=commitScheduler-1674-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
   2) Thread[id=2033, name=commitScheduler-1678-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
   3) Thread[id=2061, name=commitScheduler-1690-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
   4) Thread[id=2065, name=commitScheduler-1696-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
   5) Thread[id=2019, name=commitScheduler-1666-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
   6) Thread[id=2063, name=commitScheduler-1693-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
   7) Thread[id=2032, name=commitScheduler-1673-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
   8) Thread[id=2059, name=commitScheduler-1687-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
   9) Thread[id=2057, name=commitScheduler-1685-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
  10) Thread[id=2029, name=commitScheduler-1676-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
	at __randomizedtesting.SeedInfo.seed([8DFFD4DC74EE63F7]:0)




Build Log:
[...truncated 9598 lines...]
   [junit4] Suite: org.apache.solr.core.OpenCloseCoreStressTest
   [junit4] IGNOR/A 0.01s J1 | OpenCloseCoreStressTest.test10MinutesOld
   [junit4]    > Assumption #1: 'nightly' test group is disabled (@Nightly)
   [junit4]   2> 857219 T1995 oas.SolrTestCaseJ4.setUp ###Starting test15SecondsOld
   [junit4]   2> 857229 T1995 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 857235 T1995 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:34247
   [junit4]   2> 857235 T1995 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 857236 T1995 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 857236 T1995 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_
   [junit4]   2> 857237 T1995 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/'
   [junit4]   2> 857249 T1995 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/solr.xml
   [junit4]   2> 857297 T1995 oasc.CoreContainer.<init> New CoreContainer 1985104213
   [junit4]   2> 857298 T1995 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/]
   [junit4]   2> 857299 T1995 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 857299 T1995 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 857300 T1995 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 857300 T1995 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 857301 T1995 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 857301 T1995 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 857301 T1995 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 857302 T1995 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 857302 T1995 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 857303 T1995 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 857304 T1995 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 857304 T1995 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 857332 T2005 oasc.CoreContainer.create Creating SolrCore '00000_core' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00000_core
   [junit4]   2> 857333 T2005 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00000_core/'
   [junit4]   2> 857379 T2005 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 857423 T2005 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 857423 T2005 oass.IndexSchema.readSchema Reading Solr Schema from schema-tiny.xml
   [junit4]   2> 857427 T2005 oass.IndexSchema.readSchema [00000_core] Schema name=tiny
   [junit4]   2> 857439 T2005 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 857441 T2005 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 857441 T2005 oasc.SolrCore.<init> [00000_core] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00000_core/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00000_core/data/
   [junit4]   2> 857441 T2005 oasc.SolrCore.<init> JMX monitoring not detected for core: 00000_core
   [junit4]   2> 857442 T2005 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00000_core/data
   [junit4]   2> 857442 T2005 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00000_core/data/index/
   [junit4]   2> 857442 T2005 oasc.SolrCore.initIndex WARN [00000_core] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00000_core/data/index' doesn't exist. Creating new index...
   [junit4]   2> 857442 T2005 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00000_core/data/index
   [junit4]   2> 857443 T2005 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=42, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 857443 T2005 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@7d6f5d7f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7ac81960),segFN=segments_1,generation=1}
   [junit4]   2> 857444 T2005 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 857445 T2005 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 857445 T2005 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
   [junit4]   2> 857445 T2005 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 857445 T2005 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 857447 T2005 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 857449 T2005 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 857449 T2005 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 857450 T2005 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=39, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 857451 T2005 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@7d6f5d7f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7ac81960),segFN=segments_1,generation=1}
   [junit4]   2> 857467 T2005 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 857468 T2005 oass.SolrIndexSearcher.<init> Opening Searcher@2c8e5e85 main
   [junit4]   2> 857468 T2005 oasc.CoreContainer.registerCore registering core: 00000_core
   [junit4]   2> 857469 T2006 oasc.SolrCore.registerSearcher [00000_core] Registered new searcher Searcher@2c8e5e85 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 857469 T1995 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 857470 T1995 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 857473 T1995 oasc.OpenCloseCoreStressTest.doStress Starting indexing and querying
   [junit4]   2> 857474 T1995 oasc.OpenCloseCoreStressTest.doStress 
   [junit4]   2> 	
   [junit4]   2> 	
   [junit4]   2> 	
   [junit4]   2> 	
   [junit4]   2> 	Starting a 15 second cycle, seconds left: 15. Seconds run so far: 0.
   [junit4]   2> 857476 T2009 oasc.OneIndexer.run Starting indexing thread: 2009
   [junit4]   2> 857476 T2012 oasc.OneIndexer.run Starting indexing thread: 2012
   [junit4]   2> 857476 T2011 oasc.OneIndexer.run Starting indexing thread: 2011
   [junit4]   2> 857476 T2010 oasc.OneIndexer.run Starting indexing thread: 2010
   [junit4]   2> 857476 T2008 oasc.OneIndexer.run Starting indexing thread: 2008
   [junit4]   2> 857479 T2013 oasc.OneQuery.run Starting query thread: 2013
   [junit4]   2> 857480 T2002 oasc.CoreContainer.create Creating SolrCore '00004_core' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00004_core
   [junit4]   2> 857480 T2014 oasc.OneQuery.run Starting query thread: 2014
   [junit4]   2> 857480 T2017 oasc.OneQuery.run Starting query thread: 2017
   [junit4]   2> 857480 T2016 oasc.OneQuery.run Starting query thread: 2016
   [junit4]   2> 857480 T2015 oasc.OneQuery.run Starting query thread: 2015
   [junit4]   2> 857481 T2002 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00004_core/'
   [junit4]   2> 857480 T2004 oasc.CoreContainer.create Creating SolrCore '00002_core' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00002_core
   [junit4]   2> 857480 T2003 oasc.CoreContainer.create Creating SolrCore '00001_core' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00001_core
   [junit4]   2> 857483 T2004 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00002_core/'
   [junit4]   2> 857482 T2018 oasc.CoreContainer.create Creating SolrCore '00003_core' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00003_core
   [junit4]   2> 857483 T2003 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00001_core/'
   [junit4]   2> 857484 T2018 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00003_core/'
   [junit4]   2> ASYNC  NEW_CORE C6576 name=00000_core org.apache.solr.core.SolrCore@93f21ef
   [junit4]   2> 857487 T2001 C6576 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id3]} 0 2
   [junit4]   2> 857551 T2018 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 857554 T2003 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 857565 T2002 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> ASYNC  NEW_CORE C6577 name=00000_core org.apache.solr.core.SolrCore@93f21ef
   [junit4]   2> 857609 T2023 C6577 oasc.SolrCore.execute [00000_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=15 
   [junit4]   2> 857629 T2004 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 857666 T2018 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 857667 T2018 oass.IndexSchema.readSchema Reading Solr Schema from schema-tiny.xml
   [junit4]   2> 857669 T2003 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 857670 T2003 oass.IndexSchema.readSchema Reading Solr Schema from schema-tiny.xml
   [junit4]   2> 857682 T2018 oass.IndexSchema.readSchema [00003_core] Schema name=tiny
   [junit4]   2> 857683 T2003 oass.IndexSchema.readSchema [00001_core] Schema name=tiny
   [junit4]   2> 857688 T2002 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 857689 T2002 oass.IndexSchema.readSchema Reading Solr Schema from schema-tiny.xml
   [junit4]   2> 857703 T2002 oass.IndexSchema.readSchema [00004_core] Schema name=tiny
   [junit4]   2> 857734 T2004 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 857734 T2018 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 857735 T2004 oass.IndexSchema.readSchema Reading Solr Schema from schema-tiny.xml
   [junit4]   2> 857736 T2018 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 857736 T2018 oasc.SolrCore.<init> [00003_core] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00003_core/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00003_core/data/
   [junit4]   2> 857736 T2003 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 857736 T2018 oasc.SolrCore.<init> JMX monitoring not detected for core: 00003_core
   [junit4]   2> 857738 T2018 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00003_core/data
   [junit4]   2> 857738 T2003 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 857739 T2018 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00003_core/data/index/
   [junit4]   2> 857739 T2003 oasc.SolrCore.<init> [00001_core] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00001_core/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00001_core/data/
   [junit4]   2> 857740 T2003 oasc.SolrCore.<init> JMX monitoring not detected for core: 00001_core
   [junit4]   2> 857741 T2004 oass.IndexSchema.readSchema [00002_core] Schema name=tiny
   [junit4]   2> 857739 T2018 oasc.SolrCore.initIndex WARN [00003_core] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00003_core/data/index' doesn't exist. Creating new index...
   [junit4]   2> 857743 T2002 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 857741 T2003 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00001_core/data
   [junit4]   2> 857758 T2003 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00001_core/data/index/
   [junit4]   2> 857759 T2003 oasc.SolrCore.initIndex WARN [00001_core] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00001_core/data/index' doesn't exist. Creating new index...
   [junit4]   2> 857758 T2018 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00003_core/data/index
   [junit4]   2> 857760 T2003 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00001_core/data/index
   [junit4]   2> 857759 T2002 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 857761 T2003 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=42, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 857760 T2018 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=42, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 857762 T2003 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@1c8ff4ae lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2c624362),segFN=segments_1,generation=1}
   [junit4]   2> 857762 T2018 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@487164a7 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4c42455),segFN=segments_1,generation=1}
   [junit4]   2> 857763 T2018 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 857761 T2002 oasc.SolrCore.<init> [00004_core] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00004_core/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00004_core/data/
   [junit4]   2> 857764 T2002 oasc.SolrCore.<init> JMX monitoring not detected for core: 00004_core
   [junit4]   2> 857764 T2018 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 857762 T2003 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 857765 T2002 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00004_core/data
   [junit4]   2> 857765 T2002 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00004_core/data/index/
   [junit4]   2> 857765 T2004 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 857764 T2018 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
   [junit4]   2> 857766 T2002 oasc.SolrCore.initIndex WARN [00004_core] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00004_core/data/index' doesn't exist. Creating new index...
   [junit4]   2> 857766 T2003 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 857767 T2002 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00004_core/data/index
   [junit4]   2> 857767 T2018 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 857768 T2002 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=42, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 857767 T2003 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
   [junit4]   2> 857767 T2004 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 857769 T2002 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@354523c6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@166ea673),segFN=segments_1,generation=1}
   [junit4]   2> 857770 T2002 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 857769 T2003 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 857768 T2018 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 857771 T2002 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 857771 T2003 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 857769 T2004 oasc.SolrCore.<init> [00002_core] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00002_core/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00002_core/data/
   [junit4]   2> 857772 T2018 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 857772 T2004 oasc.SolrCore.<init> JMX monitoring not detected for core: 00002_core
   [junit4]   2> 857772 T2003 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 857771 T2002 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
   [junit4]   2> 857773 T2004 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00002_core/data
   [junit4]   2> 857773 T2002 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 857773 T2004 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00002_core/data/index/
   [junit4]   2> 857774 T2002 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 857774 T2018 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 857775 T2003 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 857774 T2004 oasc.SolrCore.initIndex WARN [00002_core] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00002_core/data/index' doesn't exist. Creating new index...
   [junit4]   2> 857776 T2002 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 857776 T2004 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00002_core/data/index
   [junit4]   2> 857776 T2003 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 857775 T2018 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 857777 T2003 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=39, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 857777 T2004 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=42, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 857778 T2003 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@1c8ff4ae lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2c624362),segFN=segments_1,generation=1}
   [junit4]   2> 857779 T2002 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 857779 T2002 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 857777 T2018 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=39, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 857780 T2002 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=39, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 857779 T2004 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@2930dd1a lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2baea459),segFN=segments_1,generation=1}
   [junit4]   2> 857781 T2004 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 857779 T2003 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 857781 T2002 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@354523c6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@166ea673),segFN=segments_1,generation=1}
   [junit4]   2> 857782 T2002 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 857780 T2018 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@487164a7 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4c42455),segFN=segments_1,generation=1}
   [junit4]   2> 857783 T2018 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 857782 T2002 oass.SolrIndexSearcher.<init> Opening Searcher@64598f1 main
   [junit4]   2> 857782 T2004 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 857782 T2003 oass.SolrIndexSearcher.<init> Opening Searcher@20272646 main
   [junit4]   2> 857784 T2004 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
   [junit4]   2> 857785 T2002 oasc.SolrCores.putTransientCore Opening transient core 00004_core
   [junit4]   2> 857783 T2018 oass.SolrIndexSearcher.<init> Opening Searcher@1674c73e main
   [junit4]   2> 857786 T2002 oasc.CoreContainer.registerCore registering core: 00004_core
   [junit4]   2> 857787 T2018 oasc.SolrCores.putTransientCore Opening transient core 00003_core
   [junit4]   2> 857785 T2004 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 857785 T2027 oasc.SolrCore.registerSearcher [00001_core] Registered new searcher Searcher@20272646 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 857785 T2003 oasc.SolrCores.putTransientCore Opening transient core 00001_core
   [junit4]   2> 857785 T2028 oasc.SolrCore.registerSearcher [00004_core] Registered new searcher Searcher@64598f1 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 857800 T2003 oasc.CoreContainer.registerCore registering core: 00001_core
   [junit4]   2> ASYNC  NEW_CORE C6578 name=00004_core org.apache.solr.core.SolrCore@79397428
   [junit4]   2> 857800 T2023 C6578 oasc.SolrCore.execute [00004_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=1 
   [junit4]   2> 857796 T2004 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 857787 T2018 oasc.CoreContainer.registerCore registering core: 00003_core
   [junit4]   2> 857787 T2026 oasc.SolrCore.registerSearcher [00003_core] Registered new searcher Searcher@1674c73e main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 857803 T2004 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> ASYNC  NEW_CORE C6579 name=00004_core org.apache.solr.core.SolrCore@79397428
   [junit4]   2> 857801 T2002 C6579 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id1]} 0 1
   [junit4]   2> 857800 T2025 C6579 oasc.SolrCore.execute [00004_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=1 
   [junit4]   2> 857813 T2004 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> ASYNC  NEW_CORE C6580 name=00001_core org.apache.solr.core.SolrCore@59f485c2
   [junit4]   2> 857815 T2003 C6580 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id2]} 0 1
   [junit4]   2> 857815 T2020 C6580 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id6]} 0 2
   [junit4]   2> 857817 T2022 C6580 oasc.SolrCore.execute [00001_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 857817 T2021 C6580 oasc.SolrCore.execute [00001_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 857817 T2002 C6580 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id7]} 0 1
   [junit4]   2> 857816 T2004 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> ASYNC  NEW_CORE C6581 name=00003_core org.apache.solr.core.SolrCore@2bca32d4
   [junit4]   2> 857818 T2024 C6581 oasc.SolrCore.execute [00003_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 857819 T2004 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=39, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 857820 T2025 C6580 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id9]} 0 1
   [junit4]   2> 857819 T2018 C6581 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id4]} 0 1
   [junit4]   2> 857819 T2023 C6580 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id8]} 0 0
   [junit4]   2> 857821 T2003 C6579 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id10]} 0 1
   [junit4]   2> 857820 T2004 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@2930dd1a lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2baea459),segFN=segments_1,generation=1}
   [junit4]   2> 857822 T2004 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 857822 T2004 oass.SolrIndexSearcher.<init> Opening Searcher@5cf5b025 main
   [junit4]   2> 857823 T2004 oasc.SolrCores.putTransientCore Opening transient core 00002_core
   [junit4]   2> 857823 T2022 C6581 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id12]} 0 1
   [junit4]   2> 857823 T2030 oasc.SolrCore.registerSearcher [00002_core] Registered new searcher Searcher@5cf5b025 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 857823 T2004 oasc.CoreContainer.registerCore registering core: 00002_core
   [junit4]   2> ASYNC  NEW_CORE C6582 name=00004_core org.apache.solr.core.SolrCore@79397428
   [junit4]   2> 857834 T2024 C6582 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id15]} 0 1
   [junit4]   2> ASYNC  NEW_CORE C6583 name=00000_core org.apache.solr.core.SolrCore@93f21ef
   [junit4]   2> 857839 T2025 C6583 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id16]} 0 2
   [junit4]   2> ASYNC  NEW_CORE C6584 name=00002_core org.apache.solr.core.SolrCore@74160aa8
   [junit4]   2> 857839 T2004 C6584 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id5]} 0 1
   [junit4]   2> 857839 T2021 C6584 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id13]} 0 2
   [junit4]   2> 857840 T2002 C6584 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id14]} 0 4
   [junit4]   2> 857840 T2020 C6584 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id11]} 0 1
   [junit4]   2> 857842 T2020 C6584 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id17]} 0 1
   [junit4]   2> 857843 T2003 C6582 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id19]} 0 1
   [junit4]   2> ASYNC  NEW_CORE C6585 name=00001_core org.apache.solr.core.SolrCore@59f485c2
   [junit4]   2> 857842 T2023 C6585 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id18]} 0 0
   [junit4]   2> 857844 T2024 C6583 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id21]} 0 1
   [junit4]   2> 857843 T2022 C6583 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id20]} 0 1
   [junit4]   2> 857846 T2025 C6583 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id22]} 0 1
   [junit4]   2> ASYNC  NEW_CORE C6586 name=00003_core org.apache.solr.core.SolrCore@2bca32d4
   [junit4]   2> 857846 T2004 C6586 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id23]} 0 1
   [junit4]   2> 857847 T2002 C6585 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id25]} 0 0
   [junit4]   2> 857847 T2021 C6583 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id24]} 0 1
   [junit4]   2> 857848 T2018 C6586 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id26]} 0 1
   [junit4]   2> 857849 T2021 C6584 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id28]} 0 0
   [junit4]   2> 857849 T2020 C6582 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id27]} 0 1
   [junit4]   2> 857850 T2023 C6586 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id29]} 0 0
   [junit4]   2> 857851 T2024 C6585 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id30]} 0 1
   [junit4]   2> ASYNC  NEW_CORE C6587 name=00002_core org.apache.solr.core.SolrCore@74160aa8
   [junit4]   2> 857860 T2022 C6587 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id31]} 0 9
   [junit4]   2> 857861 T2003 C6587 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id35]} 0 1
   [junit4]   2> ASYNC  NEW_CORE C6588 name=00004_core org.apache.solr.core.SolrCore@79397428
   [junit4]   2> 857861 T2002 C6588 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id34]} 0 1
   [junit4]   2> 857860 T2025 C6588 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id32]} 0 9
   [junit4]   2> ASYNC  NEW_CORE C6589 name=00003_core org.apache.solr.core.SolrCore@2bca32d4
   [junit4]   2> 857864 T2018 C6589 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id36]} 0 1
   [junit4]   2> ASYNC  NEW_CORE C6590 name=00001_core org.apache.solr.core.SolrCore@59f485c2
   [junit4]   2> 857864 T2021 C6590 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id37]} 0 1
   [junit4]   2> 857865 T2004 C6589 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id33]} 0 5
   [junit4]   2> ASYNC  NEW_CORE C6591 name=00000_core org.apache.solr.core.SolrCore@93f21ef
   [junit4]   2> 857865 T2020 C6591 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id39]} 0 1
   [junit4]   2> 857867 T2024 C6590 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id40]} 0 1
   [junit4]   2> 857867 T2004 C6587 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id41]} 0 0
   [junit4]   2> 857866 T2023 C6588 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id38]} 0 1
   [junit4]   2> 857869 T2003 C6587 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id42]} 0 1
   [junit4]   2> 857870 T2025 C6590 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id44]} 0 1
   [junit4]   2> 857869 T2002 C6588 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id43]} 0 0
   [junit4]   2> 857870 T2021 C6590 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id46]} 0 0
   [junit4]   2> 857870 T2018 C6588 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id45]} 0 1
   [junit4]   2> 857873 T2021 C6590 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id47]} 0 1
   [junit4]   2> 857873 T2024 C6587 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id48]} 0 1
   [junit4]   2> 857874 T2020 C6591 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id49]} 0 1
   [junit4]   2> 857874 T2004 C6589 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id50]} 0 1
   [junit4]   2> 857874 T2023 C6588 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id51]} 0 1
   [junit4]   2> 857876 T2003 C6590 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id53]} 0 1
   [junit4]   2> 857876 T2004 C6590 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id52]} 0 1
   [junit4]   2> 857877 T2002 C6591 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id54]} 0 1
   [junit4]   2> 857878 T2022 C6589 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id55]} 0 1
   [junit4]   2> 857878 T2018 C6590 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id56]} 0 1
   [junit4]   2> 857879 T2021 C6591 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id57]} 0 1
   [junit4]   2> 857881 T2018 C6588 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id58]} 0 2
   [junit4]   2> 857881 T2024 C6587 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id59]} 0 1
   [junit4]   2> 857881 T2020 C6591 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id60]} 0 1
   [junit4]   2> 857883 T2003 C6589 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id62]} 0 1
   [junit4]   2> 857883 T2023 C6588 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id61]} 0 1
   [junit4]   2> 857884 T2004 C6591 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id63]} 0 1
   [junit4]   2> 857885 T2022 C6588 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id65]} 0 1
   [junit4]   2> 857885 T2002 C6587 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id64]} 0 1
   [junit4]   2> 857886 T2025 C6591 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id66]} 0 1
   [junit4]   2> 857886 T2018 C6587 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id67]} 0 0
   [junit4]   2> 857888 T2024 C6589 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id68]} 0 1
   [junit4]   2> 857888 T2021 C6589 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id69]} 0 1
   [junit4]   2> 857888 T2020 C6588 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id70]} 0 1
   [junit4]   2> 857890 T2003 C6588 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id72]} 0 0
   [junit4]   2> 857890 T2024 C6588 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id71]} 0 1
   [junit4]   2> 857892 T2004 C6588 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id73]} 0 1
   [junit4]   2> 857893 T2022 C6591 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id74]} 0 1
   [junit4]   2> 857894 T2002 C6588 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id75]} 0 1
   [junit4]   2> 857894 T2018 C6591 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id76]} 0 1
   [junit4]   2> 857895 T2022 C6590 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id77]} 0 1
   [junit4]   2> 857895 T2002 C6591 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id78]} 0 1
   [junit4]   2> 857896 T2018 C6589 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id79]} 0 1
   [junit4]   2> 857898 T2018 C6589 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id80]} 0 1
   [junit4]   2> 857899 T2018 C6588 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id81]} 0 1
   [junit4]   2> 857900 T2018 C6587 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id82]} 0 1
   [junit4]   2> 857901 T2018 C6590 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id83]} 0 0
   [junit4]   2> 857903 T2018 C6590 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id85]} 0 1
   [junit4]   2> 857904 T2018 C6587 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id84]} 0 1
   [junit4]   2> 857904 T2021 C6589 oasc.SolrCore.execute [00003_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 857905 T2023 C6589 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id87]} 0 1
   [junit4]   2> 857905 T2025 C6590 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id86]} 0 1
   [junit4]   2> 857906 T2021 C6591 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id89]} 0 1
   [junit4]   2> 857906 T2018 C6587 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id88]} 0 1
   [junit4]   2> 857907 T2022 C6589 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id90]} 0 1
   [junit4]   2> 857908 T2020 C6591 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id91]} 0 1
   [junit4]   2> 857909 T2018 C6588 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id92]} 0 1
   [junit4]   2> 857909 T2024 C6591 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id93]} 0 1
   [junit4]   2> 857910 T2004 C6589 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id94]} 0 1
   [junit4]   2> 857911 T2023 C6588 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id96]} 0 1
   [junit4]   2> 857912 T2025 C6589 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id97]} 0 1
   [junit4]   2> 857912 T2021 C6588 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id98]} 0 1
   [junit4]   2> 857912 T2002 C6590 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id95]} 0 3
   [junit4]   2> 857913 T2003 C6591 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id99]} 0 1
   [junit4]   2> 857914 T2022 C6587 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id100]} 0 1
   [junit4]   2> 857915 T2020 C6587 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id102]} 0 1
   [junit4]   2> 857916 T2003 C6588 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id101]} 0 2
   [junit4]   2> 857917 T2022 C6590 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id103]} 0 2
   [junit4]   2> 857917 T2023 C6589 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id105]} 0 0
   [junit4]   2> 857917 T2025 C6588 oasc.SolrCore.execute [00004_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 857916 T2024 C6587 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id104]} 0 1
   [junit4]   2> 857919 T2002 C6589 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id107]} 0 1
   [junit4]   2> 857920 T2021 C6588 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id106]} 0 1
   [junit4]   2> 857920 T2004 C6591 oasc.SolrCore.execute [00000_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 857920 T2018 C6587 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id108]} 0 1
   [junit4]   2> 857921 T2024 C6588 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id109]} 0 1
   [junit4]   2> 857922 T2003 C6587 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id110]} 0 1
   [junit4]   2> 857921 T2020 C6587 oasc.SolrCore.execute [00002_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=1 
   [junit4]   2> 857922 T2023 C6588 oasc.SolrCore.execute [00004_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 857923 T2025 C6587 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id112]} 0 1
   [junit4]   2> 857925 T2022 C6589 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id113]} 0 2
   [junit4]   2> 857925 T2004 C6588 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id114]} 0 1
   [junit4]   2> 857925 T2021 C6590 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id115]} 0 1
   [junit4]   2> 857925 T2023 C6588 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id111]} 0 2
   [junit4]   2> 857927 T2024 C6588 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id116]} 0 1
   [junit4]   2> ASYNC  NEW_CORE C6592 name=00000_core org.apache.solr.core.SolrCore@93f21ef
   [junit4]   2> 857937 T2020 C6592 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id118]} 0 10
   [junit4]   2> ASYNC  NEW_CORE C6593 name=00001_core org.apache.solr.core.SolrCore@59f485c2
   [junit4]   2> 857937 T2003 C6593 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id117]} 0 10
   [junit4]   2> ASYNC  NEW_CORE C6594 name=00002_core org.apache.solr.core.SolrCore@74160aa8
   [junit4]   2> 857938 T2002 C6594 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id119]} 0 1
   [junit4]   2> ASYNC  NEW_CORE C6595 name=00003_core org.apache.solr.core.SolrCore@2bca32d4
   [junit4]   2> 857938 T2025 C6595 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id120]} 0 1
   [junit4]   2> 857939 T2018 C6594 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id121]} 0 1
   [junit4]   2> 857941 T2022 C6594 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id122]} 0 1
   [junit4]   2> 857942 T2004 C6593 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id123]} 0 1
   [junit4]   2> 857942 T2022 C6592 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id125]} 0 1
   [junit4]   2> 857942 T2023 C6593 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id124]} 0 1
   [junit4]   2> 857942 T2024 C6592 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id126]} 0 0
   [junit4]   2> 857944 T2020 C6592 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id127]} 0 1
   [junit4]   2> ASYNC  NEW_CORE C6596 name=00004_core org.apache.solr.core.SolrCore@79397428
   [junit4]   2> 857945 T2003 C6596 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2}

[...truncated too long message...]


   [junit4]    >         at java.lang.Thread.run(Thread.java:662)
   [junit4]    >    3) Thread[id=2061, name=commitScheduler-1690-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
   [junit4]    >         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
   [junit4]    >         at java.lang.Thread.run(Thread.java:662)
   [junit4]    >    4) Thread[id=2065, name=commitScheduler-1696-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
   [junit4]    >         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
   [junit4]    >         at java.lang.Thread.run(Thread.java:662)
   [junit4]    >    5) Thread[id=2019, name=commitScheduler-1666-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
   [junit4]    >         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
   [junit4]    >         at java.lang.Thread.run(Thread.java:662)
   [junit4]    >    6) Thread[id=2063, name=commitScheduler-1693-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
   [junit4]    >         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
   [junit4]    >         at java.lang.Thread.run(Thread.java:662)
   [junit4]    >    7) Thread[id=2032, name=commitScheduler-1673-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
   [junit4]    >         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
   [junit4]    >         at java.lang.Thread.run(Thread.java:662)
   [junit4]    >    8) Thread[id=2059, name=commitScheduler-1687-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
   [junit4]    >         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
   [junit4]    >         at java.lang.Thread.run(Thread.java:662)
   [junit4]    >    9) Thread[id=2057, name=commitScheduler-1685-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
   [junit4]    >         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
   [junit4]    >         at java.lang.Thread.run(Thread.java:662)
   [junit4]    >   10) Thread[id=2029, name=commitScheduler-1676-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
   [junit4]    >         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
   [junit4]    >         at java.lang.Thread.run(Thread.java:662)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([8DFFD4DC74EE63F7]:0)
   [junit4] Completed on J1 in 54.40s, 6 tests, 2 errors, 4 skipped <<< FAILURES!

[...truncated 600 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:429: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:409: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:489: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1276: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:912: There were test failures: 336 suites, 1478 tests, 2 suite-level errors, 842 ignored (5 assumptions)

Total time: 41 minutes 52 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.6.0_45 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Re: [JENKINS] Lucene-Solr-4.x-Linux (32bit/jrockit-jdk1.6.0_45-R28.2.7-4.1.0) - Build # 8222 - Still Failing!

Posted by Michael McCandless <lu...@mikemccandless.com>.
I already committed a fix for this.

Mike McCandless

http://blog.mikemccandless.com


On Thu, Nov 14, 2013 at 6:56 PM, Policeman Jenkins Server
<je...@thetaphi.de> wrote:
> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/8222/
> Java: 32bit/jrockit-jdk1.6.0_45-R28.2.7-4.1.0 -XnoOpt
>
> 1 tests failed.
> REGRESSION:  org.apache.lucene.index.TestIndexWriterReader.testTooManySegments
>
> Error Message:
>
>
> Stack Trace:
> java.lang.AssertionError
>         at __randomizedtesting.SeedInfo.seed([92DE792121089547:D44DB20BF8EF2D94]:0)
>         at org.junit.Assert.fail(Assert.java:92)
>         at org.junit.Assert.assertTrue(Assert.java:43)
>         at org.junit.Assert.assertTrue(Assert.java:54)
>         at org.apache.lucene.index.TestIndexWriterReader.testTooManySegments(TestIndexWriterReader.java:1149)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:738)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:774)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
>         at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>         at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>         at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>         at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:683)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>         at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>         at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:44)
>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
>         at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:56)
>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
>         at java.lang.Thread.run(Thread.java:662)
>
>
>
>
> Build Log:
> [...truncated 256 lines...]
>    [junit4] Suite: org.apache.lucene.index.TestIndexWriterReader
>    [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestIndexWriterReader -Dtests.method=testTooManySegments -Dtests.seed=92DE792121089547 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=mk -Dtests.timezone=Europe/Amsterdam -Dtests.file.encoding=UTF-8
>    [junit4] FAILURE 6.42s J0 | TestIndexWriterReader.testTooManySegments <<<
>    [junit4]    > Throwable #1: java.lang.AssertionError
>    [junit4]    >        at __randomizedtesting.SeedInfo.seed([92DE792121089547:D44DB20BF8EF2D94]:0)
>    [junit4]    >        at org.apache.lucene.index.TestIndexWriterReader.testTooManySegments(TestIndexWriterReader.java:1149)
>    [junit4]    >        at java.lang.Thread.run(Thread.java:662)
>    [junit4]   2> NOTE: test params are: codec=Appending, sim=DefaultSimilarity, locale=mk, timezone=Europe/Amsterdam
>    [junit4]   2> NOTE: Linux 3.8.0-32-generic i386/Oracle Corporation 1.6.0_45 (32-bit)/cpus=8,threads=1,free=28777776,total=80531456
>    [junit4]   2> NOTE: All tests run in this JVM: [TestIndexWriterReader]
>    [junit4] Completed on J0 in 25.62s, 23 tests, 1 failure <<< FAILURES!
>
> [...truncated 1321 lines...]
> BUILD FAILED
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:429: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:409: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/build.xml:49: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1276: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:912: There were test failures: 379 suites, 2758 tests, 1 failure, 105 ignored (92 assumptions)
>
> Total time: 24 minutes 21 seconds
> Build step 'Invoke Ant' marked build as failure
> Description set: Java: 32bit/jrockit-jdk1.6.0_45-R28.2.7-4.1.0 -XnoOpt
> Archiving artifacts
> Recording test results
> Email was triggered for: Failure
> Sending email for trigger: Failure
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org