You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2015/03/22 20:27:30 UTC
[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.9.0-ea-b54) - Build #
12049 - Still Failing!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/12049/
Java: 64bit/jdk1.9.0-ea-b54 -XX:+UseCompressedOops -XX:+UseSerialGC
4 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.search.json.TestJsonRequest
Error Message:
ERROR: SolrIndexSearcher opens=10 closes=4
Stack Trace:
java.lang.AssertionError: ERROR: SolrIndexSearcher opens=10 closes=4
at __randomizedtesting.SeedInfo.seed([B02D844189A0D216]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.SolrTestCaseJ4.endTrackingSearchers(SolrTestCaseJ4.java:496)
at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:232)
at sun.reflect.GeneratedMethodAccessor36.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:502)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:799)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at java.lang.Thread.run(Thread.java:745)
FAILED: junit.framework.TestSuite.org.apache.solr.search.json.TestJsonRequest
Error Message:
72 threads leaked from SUITE scope at org.apache.solr.search.json.TestJsonRequest: 1) Thread[id=9023, name=qtp1065840628-9023-acceptor-0@58df53aa-ServerConnector@5fde24fa{HTTP/1.1}{127.0.0.1:59207}, state=RUNNABLE, group=TGRP-TestJsonRequest] at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:377) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:500) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) 2) Thread[id=9049, name=qtp516957905-9049-selector-ServerConnectorManager@29f0793e/2, state=RUNNABLE, group=TGRP-TestJsonRequest] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:596) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545) at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) 3) Thread[id=9086, name=searcherExecutor-4635-thread-1, state=WAITING, group=TGRP-TestJsonRequest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 4) Thread[id=9048, name=qtp516957905-9048-selector-ServerConnectorManager@29f0793e/1, state=RUNNABLE, group=TGRP-TestJsonRequest] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:596) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545) at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) 5) Thread[id=9021, name=qtp1065840628-9021-selector-ServerConnectorManager@528f767b/2, state=RUNNABLE, group=TGRP-TestJsonRequest] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:596) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545) at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) 6) Thread[id=9012, name=qtp2084900029-9012, state=TIMED_WAITING, group=TGRP-TestJsonRequest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) at java.lang.Thread.run(Thread.java:745) 7) Thread[id=9016, name=searcherExecutor-4590-thread-1, state=WAITING, group=TGRP-TestJsonRequest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 8) Thread[id=9020, name=qtp1065840628-9020-selector-ServerConnectorManager@528f767b/1, state=RUNNABLE, group=TGRP-TestJsonRequest] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:596) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545) at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) 9) Thread[id=9011, name=qtp2084900029-9011, state=TIMED_WAITING, group=TGRP-TestJsonRequest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) at java.lang.Thread.run(Thread.java:745) 10) Thread[id=9005, name=qtp2084900029-9005-selector-ServerConnectorManager@7576e54a/0, state=RUNNABLE, group=TGRP-TestJsonRequest] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:596) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545) at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) 11) Thread[id=9007, name=qtp2084900029-9007-selector-ServerConnectorManager@7576e54a/2, state=RUNNABLE, group=TGRP-TestJsonRequest] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:596) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545) at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) 12) Thread[id=9080, name=qtp1026430202-9080, state=TIMED_WAITING, group=TGRP-TestJsonRequest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) at java.lang.Thread.run(Thread.java:745) 13) Thread[id=9056, name=org.eclipse.jetty.server.session.HashSessionManager@17fb53deTimer, state=TIMED_WAITING, group=TGRP-TestJsonRequest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 14) Thread[id=9077, name=qtp1026430202-9077-selector-ServerConnectorManager@4f9b9598/2, state=RUNNABLE, group=TGRP-TestJsonRequest] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:596) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545) at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) 15) Thread[id=9027, name=Scheduler-335371227, state=TIMED_WAITING, group=TGRP-TestJsonRequest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 16) Thread[id=9044, name=searcherExecutor-4608-thread-1, state=WAITING, group=TGRP-TestJsonRequest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 17) Thread[id=9054, name=qtp516957905-9054, state=TIMED_WAITING, group=TGRP-TestJsonRequest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) at java.lang.Thread.run(Thread.java:745) 18) Thread[id=9052, name=qtp516957905-9052, state=TIMED_WAITING, group=TGRP-TestJsonRequest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) at java.lang.Thread.run(Thread.java:745) 19) Thread[id=9087, name=Thread-3870, state=WAITING, group=TGRP-TestJsonRequest] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:508) at org.apache.solr.core.CloserThread.run(CoreContainer.java:907) 20) Thread[id=9019, name=qtp1065840628-9019-selector-ServerConnectorManager@528f767b/0, state=RUNNABLE, group=TGRP-TestJsonRequest] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:596) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545) at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) 21) Thread[id=9045, name=Thread-3840, state=WAITING, group=TGRP-TestJsonRequest] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:508) at org.apache.solr.core.CloserThread.run(CoreContainer.java:907) 22) Thread[id=9034, name=qtp1960869387-9034-selector-ServerConnectorManager@1697a6c9/1, state=RUNNABLE, group=TGRP-TestJsonRequest] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:596) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545) at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) 23) Thread[id=9024, name=qtp1065840628-9024, state=TIMED_WAITING, group=TGRP-TestJsonRequest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) at java.lang.Thread.run(Thread.java:745) 24) Thread[id=9082, name=qtp1026430202-9082, state=TIMED_WAITING, group=TGRP-TestJsonRequest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) at java.lang.Thread.run(Thread.java:745) 25) Thread[id=9076, name=qtp1026430202-9076-selector-ServerConnectorManager@4f9b9598/1, state=RUNNABLE, group=TGRP-TestJsonRequest] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:596) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545) at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) 26) Thread[id=9072, name=searcherExecutor-4626-thread-1, state=WAITING, group=TGRP-TestJsonRequest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 27) Thread[id=9036, name=qtp1960869387-9036-selector-ServerConnectorManager@1697a6c9/3, state=RUNNABLE, group=TGRP-TestJsonRequest] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:596) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545) at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) 28) Thread[id=9035, name=qtp1960869387-9035-selector-ServerConnectorManager@1697a6c9/2, state=RUNNABLE, group=TGRP-TestJsonRequest] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:596) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545) at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) 29) Thread[id=9022, name=qtp1065840628-9022-selector-ServerConnectorManager@528f767b/3, state=RUNNABLE, group=TGRP-TestJsonRequest] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:596) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545) at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) 30) Thread[id=9028, name=org.eclipse.jetty.server.session.HashSessionManager@51f2b1a5Timer, state=TIMED_WAITING, group=TGRP-TestJsonRequest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 31) Thread[id=9079, name=qtp1026430202-9079-acceptor-0@2c48f69d-ServerConnector@7ff08b41{HTTP/1.1}{127.0.0.1:33554}, state=RUNNABLE, group=TGRP-TestJsonRequest] at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:377) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:500) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) 32) Thread[id=9033, name=qtp1960869387-9033-selector-ServerConnectorManager@1697a6c9/0, state=RUNNABLE, group=TGRP-TestJsonRequest] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:596) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545) at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) 33) Thread[id=9064, name=qtp600385744-9064-selector-ServerConnectorManager@4b10acfb/3, state=RUNNABLE, group=TGRP-TestJsonRequest] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:596) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545) at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) 34) Thread[id=9026, name=qtp1065840628-9026, state=TIMED_WAITING, group=TGRP-TestJsonRequest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) at java.lang.Thread.run(Thread.java:745) 35) Thread[id=9047, name=qtp516957905-9047-selector-ServerConnectorManager@29f0793e/0, state=RUNNABLE, group=TGRP-TestJsonRequest] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:596) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545) at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) 36) Thread[id=9006, name=qtp2084900029-9006-selector-ServerConnectorManager@7576e54a/1, state=RUNNABLE, group=TGRP-TestJsonRequest] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:596) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545) at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) 37) Thread[id=9084, name=org.eclipse.jetty.server.session.HashSessionManager@33f33c9Timer, state=TIMED_WAITING, group=TGRP-TestJsonRequest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 38) Thread[id=9051, name=qtp516957905-9051-acceptor-0@2f6df613-ServerConnector@773f0edc{HTTP/1.1}{127.0.0.1:36264}, state=RUNNABLE, group=TGRP-TestJsonRequest] at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:377) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:500) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) 39) Thread[id=9063, name=qtp600385744-9063-selector-ServerConnectorManager@4b10acfb/2, state=RUNNABLE, group=TGRP-TestJsonRequest] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:596) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545) at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) 40) Thread[id=9039, name=qtp1960869387-9039, state=TIMED_WAITING, group=TGRP-TestJsonRequest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) at java.lang.Thread.run(Thread.java:745) 41) Thread[id=9059, name=Thread-3850, state=WAITING, group=TGRP-TestJsonRequest] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:508) at org.apache.solr.core.CloserThread.run(CoreContainer.java:907) 42) Thread[id=9010, name=qtp2084900029-9010, state=TIMED_WAITING, group=TGRP-TestJsonRequest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) at java.lang.Thread.run(Thread.java:745) 43) Thread[id=9075, name=qtp1026430202-9075-selector-ServerConnectorManager@4f9b9598/0, state=RUNNABLE, group=TGRP-TestJsonRequest] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:596) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545) at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) 44) Thread[id=9081, name=qtp1026430202-9081, state=TIMED_WAITING, group=TGRP-TestJsonRequest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) at java.lang.Thread.run(Thread.java:745) 45) Thread[id=9083, name=Scheduler-250326847, state=TIMED_WAITING, group=TGRP-TestJsonRequest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 46) Thread[id=9050, name=qtp516957905-9050-selector-ServerConnectorManager@29f0793e/3, state=RUNNABLE, group=TGRP-TestJsonRequest] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:596) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545) at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) 47) Thread[id=9073, name=Thread-3860, state=WAITING, group=TGRP-TestJsonRequest] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:508) at org.apache.solr.core.CloserThread.run(CoreContainer.java:907) 48) Thread[id=9069, name=Scheduler-371716121, state=TIMED_WAITING, group=TGRP-TestJsonRequest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 49) Thread[id=9066, name=qtp600385744-9066, state=TIMED_WAITING, group=TGRP-TestJsonRequest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) at java.lang.Thread.run(Thread.java:745) 50) Thread[id=9042, name=org.eclipse.jetty.server.session.HashSessionManager@115b457bTimer, state=TIMED_WAITING, group=TGRP-TestJsonRequest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 51) Thread[id=9025, name=qtp1065840628-9025, state=TIMED_WAITING, group=TGRP-TestJsonRequest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) at java.lang.Thread.run(Thread.java:745) 52) Thread[id=9061, name=qtp600385744-9061-selector-ServerConnectorManager@4b10acfb/0, state=RUNNABLE, group=TGRP-TestJsonRequest] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:596) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545) at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) 53) Thread[id=9041, name=Scheduler-336927040, state=TIMED_WAITING, group=TGRP-TestJsonRequest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 54) Thread[id=9058, name=searcherExecutor-4617-thread-1, state=WAITING, group=TGRP-TestJsonRequest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 55) Thread[id=9037, name=qtp1960869387-9037-acceptor-0@48c10929-ServerConnector@7f6913c1{HTTP/1.1}{127.0.0.1:59647}, state=RUNNABLE, group=TGRP-TestJsonRequest] at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:377) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:500) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) 56) Thread[id=9055, name=Scheduler-1764057369, state=TIMED_WAITING, group=TGRP-TestJsonRequest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 57) Thread[id=9013, name=Scheduler-2017733747, state=TIMED_WAITING, group=TGRP-TestJsonRequest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 58) Thread[id=9040, name=qtp1960869387-9040, state=TIMED_WAITING, group=TGRP-TestJsonRequest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) at java.lang.Thread.run(Thread.java:745) 59) Thread[id=9078, name=qtp1026430202-9078-selector-ServerConnectorManager@4f9b9598/3, state=RUNNABLE, group=TGRP-TestJsonRequest] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:596) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545) at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) 60) Thread[id=9067, name=qtp600385744-9067, state=TIMED_WAITING, group=TGRP-TestJsonRequest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) at java.lang.Thread.run(Thread.java:745) 61) Thread[id=9065, name=qtp600385744-9065-acceptor-0@1608d4bd-ServerConnector@6f765e3{HTTP/1.1}{127.0.0.1:50989}, state=RUNNABLE, group=TGRP-TestJsonRequest] at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:377) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:500) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) 62) Thread[id=9062, name=qtp600385744-9062-selector-ServerConnectorManager@4b10acfb/1, state=RUNNABLE, group=TGRP-TestJsonRequest] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:596) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545) at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) 63) Thread[id=9009, name=qtp2084900029-9009-acceptor-0@37f545e4-ServerConnector@7acfb140{HTTP/1.1}{127.0.0.1:50549}, state=RUNNABLE, group=TGRP-TestJsonRequest] at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:377) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:500) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) 64) Thread[id=9070, name=org.eclipse.jetty.server.session.HashSessionManager@2305f873Timer, state=TIMED_WAITING, group=TGRP-TestJsonRequest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 65) Thread[id=9053, name=qtp516957905-9053, state=TIMED_WAITING, group=TGRP-TestJsonRequest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) at java.lang.Thread.run(Thread.java:745) 66) Thread[id=9030, name=searcherExecutor-4599-thread-1, state=WAITING, group=TGRP-TestJsonRequest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 67) Thread[id=9014, name=org.eclipse.jetty.server.session.HashSessionManager@572c0f33Timer, state=TIMED_WAITING, group=TGRP-TestJsonRequest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 68) Thread[id=9068, name=qtp600385744-9068, state=TIMED_WAITING, group=TGRP-TestJsonRequest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) at java.lang.Thread.run(Thread.java:745) 69) Thread[id=9008, name=qtp2084900029-9008-selector-ServerConnectorManager@7576e54a/3, state=RUNNABLE, group=TGRP-TestJsonRequest] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:596) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545) at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) 70) Thread[id=9038, name=qtp1960869387-9038, state=TIMED_WAITING, group=TGRP-TestJsonRequest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) at java.lang.Thread.run(Thread.java:745) 71) Thread[id=9031, name=Thread-3830, state=WAITING, group=TGRP-TestJsonRequest] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:508) at org.apache.solr.core.CloserThread.run(CoreContainer.java:907) 72) Thread[id=9017, name=Thread-3820, state=WAITING, group=TGRP-TestJsonRequest] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:508) at org.apache.solr.core.CloserThread.run(CoreContainer.java:907)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 72 threads leaked from SUITE scope at org.apache.solr.search.json.TestJsonRequest:
1) Thread[id=9023, name=qtp1065840628-9023-acceptor-0@58df53aa-ServerConnector@5fde24fa{HTTP/1.1}{127.0.0.1:59207}, state=RUNNABLE, group=TGRP-TestJsonRequest]
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:377)
at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:500)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
2) Thread[id=9049, name=qtp516957905-9049-selector-ServerConnectorManager@29f0793e/2, state=RUNNABLE, group=TGRP-TestJsonRequest]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:596)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545)
at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
3) Thread[id=9086, name=searcherExecutor-4635-thread-1, state=WAITING, group=TGRP-TestJsonRequest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
4) Thread[id=9048, name=qtp516957905-9048-selector-ServerConnectorManager@29f0793e/1, state=RUNNABLE, group=TGRP-TestJsonRequest]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:596)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545)
at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
5) Thread[id=9021, name=qtp1065840628-9021-selector-ServerConnectorManager@528f767b/2, state=RUNNABLE, group=TGRP-TestJsonRequest]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:596)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545)
at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
6) Thread[id=9012, name=qtp2084900029-9012, state=TIMED_WAITING, group=TGRP-TestJsonRequest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
at java.lang.Thread.run(Thread.java:745)
7) Thread[id=9016, name=searcherExecutor-4590-thread-1, state=WAITING, group=TGRP-TestJsonRequest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
8) Thread[id=9020, name=qtp1065840628-9020-selector-ServerConnectorManager@528f767b/1, state=RUNNABLE, group=TGRP-TestJsonRequest]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:596)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545)
at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
9) Thread[id=9011, name=qtp2084900029-9011, state=TIMED_WAITING, group=TGRP-TestJsonRequest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
at java.lang.Thread.run(Thread.java:745)
10) Thread[id=9005, name=qtp2084900029-9005-selector-ServerConnectorManager@7576e54a/0, state=RUNNABLE, group=TGRP-TestJsonRequest]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:596)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545)
at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
11) Thread[id=9007, name=qtp2084900029-9007-selector-ServerConnectorManager@7576e54a/2, state=RUNNABLE, group=TGRP-TestJsonRequest]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:596)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545)
at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
12) Thread[id=9080, name=qtp1026430202-9080, state=TIMED_WAITING, group=TGRP-TestJsonRequest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
at java.lang.Thread.run(Thread.java:745)
13) Thread[id=9056, name=org.eclipse.jetty.server.session.HashSessionManager@17fb53deTimer, state=TIMED_WAITING, group=TGRP-TestJsonRequest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
14) Thread[id=9077, name=qtp1026430202-9077-selector-ServerConnectorManager@4f9b9598/2, state=RUNNABLE, group=TGRP-TestJsonRequest]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:596)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545)
at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
15) Thread[id=9027, name=Scheduler-335371227, state=TIMED_WAITING, group=TGRP-TestJsonRequest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
16) Thread[id=9044, name=searcherExecutor-4608-thread-1, state=WAITING, group=TGRP-TestJsonRequest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
17) Thread[id=9054, name=qtp516957905-9054, state=TIMED_WAITING, group=TGRP-TestJsonRequest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
at java.lang.Thread.run(Thread.java:745)
18) Thread[id=9052, name=qtp516957905-9052, state=TIMED_WAITING, group=TGRP-TestJsonRequest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
at java.lang.Thread.run(Thread.java:745)
19) Thread[id=9087, name=Thread-3870, state=WAITING, group=TGRP-TestJsonRequest]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:508)
at org.apache.solr.core.CloserThread.run(CoreContainer.java:907)
20) Thread[id=9019, name=qtp1065840628-9019-selector-ServerConnectorManager@528f767b/0, state=RUNNABLE, group=TGRP-TestJsonRequest]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:596)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545)
at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
21) Thread[id=9045, name=Thread-3840, state=WAITING, group=TGRP-TestJsonRequest]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:508)
at org.apache.solr.core.CloserThread.run(CoreContainer.java:907)
22) Thread[id=9034, name=qtp1960869387-9034-selector-ServerConnectorManager@1697a6c9/1, state=RUNNABLE, group=TGRP-TestJsonRequest]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:596)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545)
at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
23) Thread[id=9024, name=qtp1065840628-9024, state=TIMED_WAITING, group=TGRP-TestJsonRequest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
at java.lang.Thread.run(Thread.java:745)
24) Thread[id=9082, name=qtp1026430202-9082, state=TIMED_WAITING, group=TGRP-TestJsonRequest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
at java.lang.Thread.run(Thread.java:745)
25) Thread[id=9076, name=qtp1026430202-9076-selector-ServerConnectorManager@4f9b9598/1, state=RUNNABLE, group=TGRP-TestJsonRequest]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:596)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545)
at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
26) Thread[id=9072, name=searcherExecutor-4626-thread-1, state=WAITING, group=TGRP-TestJsonRequest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
27) Thread[id=9036, name=qtp1960869387-9036-selector-ServerConnectorManager@1697a6c9/3, state=RUNNABLE, group=TGRP-TestJsonRequest]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:596)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545)
at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
28) Thread[id=9035, name=qtp1960869387-9035-selector-ServerConnectorManager@1697a6c9/2, state=RUNNABLE, group=TGRP-TestJsonRequest]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:596)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545)
at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
29) Thread[id=9022, name=qtp1065840628-9022-selector-ServerConnectorManager@528f767b/3, state=RUNNABLE, group=TGRP-TestJsonRequest]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:596)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545)
at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
at
[...truncated too long message...]
nio.ch.SelectorImpl.select(SelectorImpl.java:101)
[junit4] > at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:596)
[junit4] > at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545)
[junit4] > at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] > 41) Thread[id=9047, name=qtp516957905-9047-selector-ServerConnectorManager@29f0793e/0, state=RUNNABLE, group=TGRP-TestJsonRequest]
[junit4] > at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method)
[junit4] > at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:317)
[junit4] > at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:193)
[junit4] > at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:213)
[junit4] > at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:219)
[junit4] > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:78)
[junit4] > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
[junit4] > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
[junit4] > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
[junit4] > at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:596)
[junit4] > at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545)
[junit4] > at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] > 42) Thread[id=9030, name=searcherExecutor-4599-thread-1, state=WAITING, group=TGRP-TestJsonRequest]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
[junit4] > at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] > 43) Thread[id=9014, name=org.eclipse.jetty.server.session.HashSessionManager@572c0f33Timer, state=TIMED_WAITING, group=TGRP-TestJsonRequest]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
[junit4] > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
[junit4] > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] > 44) Thread[id=9006, name=qtp2084900029-9006-selector-ServerConnectorManager@7576e54a/1, state=RUNNABLE, group=TGRP-TestJsonRequest]
[junit4] > at sun.nio.ch.IOUtil.drain(Native Method)
[junit4] > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:90)
[junit4] > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
[junit4] > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
[junit4] > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
[junit4] > at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:596)
[junit4] > at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545)
[junit4] > at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] > 45) Thread[id=9084, name=org.eclipse.jetty.server.session.HashSessionManager@33f33c9Timer, state=TIMED_WAITING, group=TGRP-TestJsonRequest]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
[junit4] > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
[junit4] > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] > 46) Thread[id=9008, name=qtp2084900029-9008-selector-ServerConnectorManager@7576e54a/3, state=RUNNABLE, group=TGRP-TestJsonRequest]
[junit4] > at sun.nio.ch.IOUtil.drain(Native Method)
[junit4] > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:90)
[junit4] > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
[junit4] > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
[junit4] > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
[junit4] > at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:596)
[junit4] > at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545)
[junit4] > at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] > 47) Thread[id=9031, name=Thread-3830, state=WAITING, group=TGRP-TestJsonRequest]
[junit4] > at java.lang.Object.wait(Native Method)
[junit4] > at java.lang.Object.wait(Object.java:508)
[junit4] > at org.apache.solr.core.CloserThread.run(CoreContainer.java:907)
[junit4] > 48) Thread[id=9017, name=Thread-3820, state=WAITING, group=TGRP-TestJsonRequest]
[junit4] > at java.lang.Object.wait(Native Method)
[junit4] > at java.lang.Object.wait(Object.java:508)
[junit4] > at org.apache.solr.core.CloserThread.run(CoreContainer.java:907)
[junit4] > at __randomizedtesting.SeedInfo.seed([B02D844189A0D216]:0)
[junit4] Completed on J0 in 146.31s, 2 tests, 1 failure, 3 errors <<< FAILURES!
[...truncated 12 lines...]
BUILD FAILED
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:519: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:61: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:191: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:510: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1351: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:958: There were test failures: 476 suites, 1894 tests, 3 suite-level errors, 1 error, 48 ignored (21 assumptions)
Total time: 43 minutes 32 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.9.0-ea-b54 -XX:+UseCompressedOops -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_60-ea-b06) -
Build # 12050 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/12050/
Java: 64bit/jdk1.8.0_60-ea-b06 -XX:+UseCompressedOops -XX:+UseG1GC
1 tests failed.
FAILED: org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.test
Error Message:
There were too many update fails - we expect it can happen, but shouldn't easily
Stack Trace:
java.lang.AssertionError: There were too many update fails - we expect it can happen, but shouldn't easily
at __randomizedtesting.SeedInfo.seed([7B49B40F4E87B816:F31D8BD5E07BD5EE]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertFalse(Assert.java:68)
at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.test(ChaosMonkeyNothingIsSafeTest.java:222)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:958)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:933)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at java.lang.Thread.run(Thread.java:745)
Build Log:
[...truncated 10080 lines...]
[junit4] Suite: org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest
[junit4] 2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/init-core-data-001
[junit4] 2> 751736 T5519 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4] 2> 751740 T5519 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 751747 T5520 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 751847 T5519 oasc.ZkTestServer.run start zk server on port:58253
[junit4] 2> 751848 T5519 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 751848 T5519 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 751871 T5527 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@11d969e3 name:ZooKeeperConnection Watcher:127.0.0.1:58253 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 751872 T5519 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 751872 T5519 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 751872 T5519 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 751884 T5519 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 751885 T5519 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 751887 T5530 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ee8bf2c name:ZooKeeperConnection Watcher:127.0.0.1:58253/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 751888 T5519 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 751888 T5519 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 751888 T5519 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 751898 T5519 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 751900 T5519 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 751901 T5519 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 751902 T5519 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 751902 T5519 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 751904 T5519 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
[junit4] 2> 751904 T5519 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 751905 T5519 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 751906 T5519 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 751907 T5519 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 751907 T5519 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 751908 T5519 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 751909 T5519 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 751910 T5519 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 751911 T5519 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 751913 T5519 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 751913 T5519 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
[junit4] 2> 751914 T5519 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 751914 T5519 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 751915 T5519 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 751915 T5519 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 751917 T5519 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 751917 T5519 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 751924 T5519 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 751924 T5519 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 752052 T5519 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/control-001/cores/collection1
[junit4] 2> 752053 T5519 oejs.Server.doStart jetty-9.2.9.v20150224
[junit4] 2> 752060 T5519 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@11665869{/,null,AVAILABLE}
[junit4] 2> 752071 T5519 oejs.AbstractConnector.doStart Started ServerConnector@ac72a35{HTTP/1.1}{127.0.0.1:46147}
[junit4] 2> 752071 T5519 oejs.Server.doStart Started @753243ms
[junit4] 2> 752072 T5519 oascse.JettySolrRunner$1.lifeCycleStarted Jetty properties: {solr.data.dir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/tempDir-001/control/data, hostContext=/, hostPort=46147, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/control-001/cores}
[junit4] 2> 752073 T5519 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@4e0e2f2a
[junit4] 2> 752073 T5519 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/control-001/'
[junit4] 2> 752089 T5519 oasc.SolrXmlConfig.fromFile Loading container configuration from /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/control-001/solr.xml
[junit4] 2> 752093 T5519 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/control-001/cores
[junit4] 2> 752094 T5519 oasc.CoreContainer.<init> New CoreContainer 418085395
[junit4] 2> 752094 T5519 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/control-001/]
[junit4] 2> 752094 T5519 oasc.CoreContainer.load loading shared library: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/control-001/lib
[junit4] 2> 752095 T5519 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/control-001/lib).
[junit4] 2> 752101 T5519 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 752102 T5519 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 752102 T5519 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 752102 T5519 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 752102 T5519 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
[junit4] 2> 752103 T5519 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 752103 T5519 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 752103 T5519 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 752103 T5519 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 752104 T5519 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 752104 T5519 oashc.HttpShardHandlerFactory.getParameter Setting useRetries to: false
[junit4] 2> 752104 T5519 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 752105 T5519 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 752105 T5519 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 752105 T5519 oasc.CoreContainer.load Node Name: 127.0.0.1
[junit4] 2> 752105 T5519 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:58253/solr
[junit4] 2> 752106 T5519 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 752106 T5519 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 752106 T5519 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 752108 T5544 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2a9792f name:ZooKeeperConnection Watcher:127.0.0.1:58253 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 752109 T5519 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 752109 T5519 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 752113 T5519 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 752114 T5547 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6061c35f name:ZooKeeperConnection Watcher:127.0.0.1:58253/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 752115 T5519 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 752116 T5519 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 752118 T5519 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 752119 T5519 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 752121 T5519 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 752122 T5519 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 752127 T5519 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 752128 T5519 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 752129 T5519 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 752130 T5519 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46147_
[junit4] 2> 752130 T5519 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:46147_
[junit4] 2> 752132 T5519 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 752133 T5519 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 752133 T5519 oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 752135 T5519 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:46147_
[junit4] 2> 752136 T5519 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 752137 T5519 oasc.Overseer.start Overseer (id=93523738476478467-127.0.0.1:46147_-n_0000000000) starting
[junit4] 2> 752138 T5519 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 752143 T5519 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 752143 T5549 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 752144 T5548 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 752164 T5519 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 752166 T5519 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/control-001/cores
[junit4] 2> 752166 T5519 oasc.CoreDescriptor.<init> CORE DESCRIPTOR: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, instanceDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/control-001/cores/collection1, collection=control_collection, absoluteInstDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/control-001/cores/collection1/, coreNodeName=, dataDir=data/, shard=}
[junit4] 2> 752167 T5519 oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/control-001/cores/collection1/
[junit4] 2> 752167 T5519 oasc.CorePropertiesLocator.discover Found 1 core definitions
[junit4] 2> 752167 T5551 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
[junit4] 2> 752167 T5551 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 752168 T5547 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 752168 T5551 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 752169 T5548 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"http://127.0.0.1:46147",
[junit4] 2> "node_name":"127.0.0.1:46147_",
[junit4] 2> "numShards":"1",
[junit4] 2> "state":"down",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "operation":"state"} current state version: 0
[junit4] 2> 752169 T5548 oasco.ReplicaMutator.updateState Update state numShards=1 message={
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"http://127.0.0.1:46147",
[junit4] 2> "node_name":"127.0.0.1:46147_",
[junit4] 2> "numShards":"1",
[junit4] 2> "state":"down",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "operation":"state"}
[junit4] 2> 752169 T5548 oasco.ClusterStateMutator.createCollection building a new cName: control_collection
[junit4] 2> 752170 T5548 oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard1
[junit4] 2> 752171 T5547 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 753168 T5551 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 753169 T5551 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 753169 T5551 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 753169 T5551 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 753170 T5551 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 753170 T5551 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/control-001/cores/collection1/'
[junit4] 2> 753180 T5551 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
[junit4] 2> 753182 T5551 oasc.Config.<init> loaded config solrconfig.xml with version 0
[junit4] 2> 753186 T5551 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
[junit4] 2> 753191 T5551 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
[junit4] 2> 753200 T5551 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 753201 T5551 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
[junit4] 2> 753209 T5551 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 753294 T5551 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 753295 T5551 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 753296 T5551 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 753298 T5551 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 753309 T5551 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
[junit4] 2> 753309 T5551 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 753309 T5551 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/control-001/cores/collection1/, dataDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/control-001/cores/collection1/data/
[junit4] 2> 753310 T5551 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2695e2ad
[junit4] 2> 753310 T5551 oasc.CachingDirectoryFactory.get return new directory for /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/control-001/cores/collection1/data
[junit4] 2> 753311 T5551 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/control-001/cores/collection1/data/index/
[junit4] 2> 753311 T5551 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/control-001/cores/collection1/data/index' doesn't exist. Creating new index...
[junit4] 2> 753312 T5551 oasc.CachingDirectoryFactory.get return new directory for /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/control-001/cores/collection1/data/index
[junit4] 2> 753312 T5551 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=34, maxMergeAtOnceExplicit=43, maxMergedSegmentMB=85.4970703125, floorSegmentMB=1.8056640625, forceMergeDeletesPctAllowed=9.519055254924117, segmentsPerTier=46.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
[junit4] 2> 753346 T5551 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/control-001/cores/collection1/data/index,segFN=segments_1,generation=1}
[junit4] 2> 753346 T5551 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 753351 T5551 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 753351 T5551 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 753352 T5551 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 753352 T5551 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 753352 T5551 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 753352 T5551 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 753352 T5551 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 753353 T5551 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 753353 T5551 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 753355 T5551 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 753356 T5551 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 753356 T5551 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 753357 T5551 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 753360 T5551 oasc.RequestHandlers.initHandlersFromConfig Registered paths: /admin/mbeans,standard,/update/csv,/update/json/docs,/admin/luke,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/schema,/admin/plugins,/admin/logging,/update/json,/admin/threads,/admin/ping,/update,/admin/file
[junit4] 2> 753360 T5551 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
[junit4] 2> 753361 T5551 oasu.UpdateHandler.<init> Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 753361 T5551 oasu.UpdateLog.init Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
[junit4] 2> 753361 T5551 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 753361 T5551 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 753362 T5551 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=31, maxMergeAtOnceExplicit=10, maxMergedSegmentMB=79.779296875, floorSegmentMB=0.591796875, forceMergeDeletesPctAllowed=3.1114218803298144, segmentsPerTier=35.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 753362 T5551 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/control-001/cores/collection1/data/index,segFN=segments_1,generation=1}
[junit4] 2> 753362 T5551 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 753362 T5551 oass.SolrIndexSearcher.<init> Opening Searcher@3d2586e8[collection1] main
[junit4] 2> 753362 T5551 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 753363 T5551 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 753363 T5551 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 753363 T5551 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 753363 T5551 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 753364 T5551 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 753364 T5551 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 753364 T5551 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 753364 T5551 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 753364 T5551 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4] 2> 753365 T5552 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3d2586e8[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 753365 T5551 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 753366 T5555 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:46147 collection:control_collection shard:shard1
[junit4] 2> 753366 T5519 oass.SolrDispatchFilter.init user.dir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4] 2> 753366 T5519 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 753367 T5519 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 753367 T5555 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 753367 T5519 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 753369 T5558 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@69e9fcd0 name:ZooKeeperConnection Watcher:127.0.0.1:58253/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 753370 T5519 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 753370 T5519 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 753370 T5519 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 753370 T5555 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 753373 T5547 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 753373 T5555 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 753373 T5519 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 753373 T5548 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"control_collection"} current state version: 1
[junit4] 2> 753373 T5555 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C1702 name=collection1 org.apache.solr.core.SolrCore@7f1a1b6d url=http://127.0.0.1:46147/collection1 node=127.0.0.1:46147_ C1702_STATE=coll:control_collection core:collection1 props:{core=collection1, base_url=http://127.0.0.1:46147, node_name=127.0.0.1:46147_, state=down}
[junit4] 2> 753373 T5555 C1702 P46147 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:46147/collection1/
[junit4] 2> 753374 T5555 C1702 P46147 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 753374 T5555 C1702 P46147 oasc.SyncStrategy.syncToMe http://127.0.0.1:46147/collection1/ has no replicas
[junit4] 2> 753374 T5555 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:46147/collection1/ shard1
[junit4] 2> 753374 T5555 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 753379 T5547 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 753379 T5548 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "base_url":"http://127.0.0.1:46147",
[junit4] 2> "core":"collection1",
[junit4] 2> "state":"active"} current state version: 1
[junit4] 2> 753424 T5519 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/shard-1-001/cores/collection1
[junit4] 2> 753425 T5519 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1 in directory /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/shard-1-001
[junit4] 2> 753425 T5519 oejs.Server.doStart jetty-9.2.9.v20150224
[junit4] 2> 753426 T5519 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@5e5e71ee{/,null,AVAILABLE}
[junit4] 2> 753427 T5519 oejs.AbstractConnector.doStart Started ServerConnector@475e4474{HTTP/1.1}{127.0.0.1:38503}
[junit4] 2> 753427 T5519 oejs.Server.doStart Started @754598ms
[junit4] 2> 753427 T5519 oascse.JettySolrRunner$1.lifeCycleStarted Jetty properties: {solr.data.dir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/tempDir-001/jetty1, solrconfig=solrconfig.xml, hostContext=/, hostPort=38503, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/shard-1-001/cores}
[junit4] 2> 753428 T5519 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@4e0e2f2a
[junit4] 2> 753428 T5519 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/shard-1-001/'
[junit4] 2> 753439 T5519 oasc.SolrXmlConfig.fromFile Loading container configuration from /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/shard-1-001/solr.xml
[junit4] 2> 753443 T5519 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/shard-1-001/cores
[junit4] 2> 753443 T5519 oasc.CoreContainer.<init> New CoreContainer 407722908
[junit4] 2> 753443 T5519 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/shard-1-001/]
[junit4] 2> 753444 T5519 oasc.CoreContainer.load loading shared library: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/shard-1-001/lib
[junit4] 2> 753444 T5519 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/shard-1-001/lib).
[junit4] 2> 753448 T5519 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 753448 T5519 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 753449 T5519 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 753449 T5519 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 753449 T5519 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
[junit4] 2> 753449 T5519 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 753449 T5519 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 753449 T5519 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 753450 T5519 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 753450 T5519 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 753450 T5519 oashc.HttpShardHandlerFactory.getParameter Setting useRetries to: false
[junit4] 2> 753450 T5519 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 753450 T5519 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 753451 T5519 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 753451 T5519 oasc.CoreContainer.load Node Name: 127.0.0.1
[junit4] 2> 753451 T5519 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:58253/solr
[junit4] 2> 753451 T5519 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 753451 T5519 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 753452 T5519 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 753462 T5572 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@38fa2660 name:ZooKeeperConnection Watcher:127.0.0.1:58253 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 753462 T5519 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 753463 T5519 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 753470 T5519 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 753472 T5575 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4d331773 name:ZooKeeperConnection Watcher:127.0.0.1:58253/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 753472 T5519 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 753475 T5519 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 753481 T5558 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 753481 T5575 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 753481 T5547 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 753526 T5555 oasc.ZkController.register We are http://127.0.0.1:46147/collection1/ and leader is http://127.0.0.1:46147/collection1/
[junit4] 2> 753527 T5555 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:46147
[junit4] 2> 753527 T5555 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 753527 T5555 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
[junit4] 2> 753527 T5555 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 753528 T5547 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 753529 T5548 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "core":"collection1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"http://127.0.0.1:46147",
[junit4] 2> "node_name":"127.0.0.1:46147_",
[junit4] 2> "numShards":"1",
[junit4] 2> "state":"active",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "operation":"state"} current state version: 2
[junit4] 2> 753529 T5548 oasco.ReplicaMutator.updateState Update state numShards=1 message={
[junit4] 2> "core":"collection1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"http://127.0.0.1:46147",
[junit4] 2> "node_name":"127.0.0.1:46147_",
[junit4] 2> "numShards":"1",
[junit4] 2> "state":"active",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "operation":"state"}
[junit4] 2> 753630 T5558 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 753631 T5547 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 753631 T5575 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 754477 T5519 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38503_
[junit4] 2> 754478 T5519 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:38503_
[junit4] 2> 754479 T5519 oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 754481 T5519 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/shard-1-001/cores
[junit4] 2> 754481 T5519 oasc.CoreDescriptor.<init> CORE DESCRIPTOR: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, instanceDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/shard-1-001/cores/collection1, collection=collection1, absoluteInstDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/shard-1-001/cores/collection1/, coreNodeName=, dataDir=data/, shard=}
[junit4] 2> 754482 T5519 oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/shard-1-001/cores/collection1/
[junit4] 2> 754482 T5519 oasc.CorePropertiesLocator.discover Found 1 core definitions
[junit4] 2> 754482 T5576 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 754482 T5576 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 754483 T5576 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 754483 T5547 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 754488 T5548 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"http://127.0.0.1:38503",
[junit4] 2> "node_name":"127.0.0.1:38503_",
[junit4] 2> "numShards":"1",
[junit4] 2> "state":"down",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "operation":"state"} current state version: 3
[junit4] 2> 754488 T5548 oasco.ReplicaMutator.updateState Update state numShards=1 message={
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"http://127.0.0.1:38503",
[junit4] 2> "node_name":"127.0.0.1:38503_",
[junit4] 2> "numShards":"1",
[junit4] 2> "state":"down",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "operation":"state"}
[junit4] 2> 754488 T5548 oasco.ClusterStateMutator.createCollection building a new cName: collection1
[junit4] 2> 754489 T5548 oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard1
[junit4] 2> 754590 T5558 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 754590 T5547 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 754590 T5575 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 755483 T5576 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 755483 T5576 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 755484 T5576 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 755484 T5576 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 755484 T5576 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 755484 T5576 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/shard-1-001/cores/collection1/'
[junit4] 2> 755492 T5576 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
[junit4] 2> 755492 T5576 oasc.Config.<init> loaded config solrconfig.xml with version 0
[junit4] 2> 755496 T5576 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
[junit4] 2> 755498 T5576 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
[junit4] 2> 755503 T5576 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 755503 T5576 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
[junit4] 2> 755506 T5576 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 755568 T5576 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 755569 T5576 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 755570 T5576 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 755571 T5576 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 755579 T5576 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 755579 T5576 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 755580 T5576 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/shard-1-001/cores/collection1/, dataDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/shard-1-001/cores/collection1/data/
[junit4] 2> 755580 T5576 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2695e2ad
[junit4] 2> 755580 T5576 oasc.CachingDirectoryFactory.get return new directory for /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/shard-1-001/cores/collection1/data
[junit4] 2> 755580 T5576 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/shard-1-001/cores/collection1/data/index/
[junit4] 2> 755581 T5576 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/shard-1-001/cores/collection1/data/index' doesn't exist. Creating new index...
[junit4] 2> 755581 T5576 oasc.CachingDirectoryFactory.get return new directory for /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/shard-1-001/cores/collection1/data/index
[junit4] 2> 755581 T5576 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=34, maxMergeAtOnceExplicit=43, maxMergedSegmentMB=85.4970703125, floorSegmentMB=1.8056640625, forceMergeDeletesPctAllowed=9.519055254924117, segmentsPerTier=46.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
[junit4] 2> 755602 T5576 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/shard-1-001/cores/collection1/data/index,segFN=segments_1,generation=1}
[junit4] 2> 755602 T5576 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 755606 T5576 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 755606 T5576 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 755606 T5576 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 755606 T5576 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 755606 T5576 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 755607 T5576 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 755607 T5576 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 755607 T5576 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 755607 T5576 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 755608 T5576 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 755609 T5576 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 755610 T5576 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 755610 T5576 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 755617 T5576 oasc.RequestHandlers.initHandlersFromConfig Registered paths: /admin/mbeans,standard,/update/csv,/update/json/docs,/admin/luke,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/schema,/admin/plugins,/admin/logging,/update/json,/admin/threads,/admin/ping,/update,/admin/file
[junit4] 2> 755617 T5576 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
[junit4] 2> 755618 T5576 oasu.UpdateHandler.<init> Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 755618 T5576 oasu.UpdateLog.init Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
[junit4] 2> 755618 T5576 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 755619 T5576 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 755619 T5576 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=31, maxMergeAtOnceExplicit=10, maxMergedSegmentMB=79.779296875, floorSegmentMB=0.591796875, forceMergeDeletesPctAllowed=3.1114218803298144, segmentsPerTier=35.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 755620 T5576 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/shard-1-001/cores/collection1/data/index,segFN=segments_1,generation=1}
[junit4] 2> 755620 T5576 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 755621 T5576 oass.SolrIndexSearcher.<init> Opening Searcher@2b42ed3b[collection1] main
[junit4] 2> 755621 T5576 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 755621 T5576 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 755621 T5576 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 755622 T5576 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 755622 T5576 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 755622 T5576 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 755622 T5576 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 755622 T5576 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 755623 T5576 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 755623 T5576 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4] 2> 755623 T5577 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2b42ed3b[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 755623 T5576 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 755624 T5580 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:38503 collection:collection1 shard:shard1
[junit4] 2> 755624 T5519 oass.SolrDispatchFilter.init user.dir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4] 2> 755624 T5519 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 755624 T5580 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 755626 T5580 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 755628 T5547 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 755628 T5580 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 755628 T5580 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 755628 T5548 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1"} current state version: 4
[junit4] 2> ASYNC NEW_CORE C1703 name=collection1 org.apache.solr.core.SolrCore@3c0e1cdf url=http://127.0.0.1:38503/collection1 node=127.0.0.1:38503_ C1703_STATE=coll:collection1 core:collection1 props:{core=collection1, base_url=http://127.0.0.1:38503, node_name=127.0.0.1:38503_, state=down}
[junit4] 2> 755628 T5580 C1703 P38503 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:38503/collection1/
[junit4] 2> 755629 T5580 C1703 P38503 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 755629 T5580 C1703 P38503 oasc.SyncStrategy.syncToMe http://127.0.0.1:38503/collection1/ has no replicas
[junit4] 2> 755629 T5580 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:38503/collection1/ shard1
[junit4] 2> 755629 T5580 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 755638 T5547 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 755639 T5548 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "base_url":"http://127.0.0.1:38503",
[junit4] 2> "core":"collection1",
[junit4] 2> "state":"active"} current state version: 4
[junit4] 2> 755681 T5519 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/shard-2-001/cores/collection1
[junit4] 2> 755682 T5519 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2 in directory /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/shard-2-001
[junit4] 2> 755682 T5519 oejs.Server.doStart jetty-9.2.9.v20150224
[junit4] 2> 755683 T5519 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@1f862133{/,null,AVAILABLE}
[junit4] 2> 755684 T5519 oejs.AbstractConnector.doStart Started ServerConnector@17c051d7{HTTP/1.1}{127.0.0.1:49688}
[junit4] 2> 755684 T5519 oejs.Server.doStart Started @756856ms
[junit4] 2> 755684 T5519 oascse.JettySolrRunner$1.lifeCycleStarted Jetty properties: {solr.data.dir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/tempDir-001/jetty2, solrconfig=solrconfig.xml, hostContext=/, hostPort=49688, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/shard-2-001/cores}
[junit4] 2> 755685 T5519 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@4e0e2f2a
[junit4] 2> 755685 T5519 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/shard-2-001/'
[junit4] 2> 755696 T5519 oasc.SolrXmlConfig.fromFile Loading container configuration from /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/shard-2-001/solr.xml
[junit4] 2> 755699 T5519 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/shard-2-001/cores
[junit4] 2> 755699 T5519 oasc.CoreContainer.<init> New CoreContainer 521765856
[junit4] 2> 755699 T5519 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/shard-2-001/]
[junit4] 2> 755700 T5519 oasc.CoreContainer.load loading shared library: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/shard-2-001/lib
[junit4] 2> 755700 T5519 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/shard-2-001/lib).
[junit4] 2> 755704 T5519 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 755704 T5519 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 755705 T5519 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 755705 T5519 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 755705 T5519 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
[junit4] 2> 755705 T5519 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 755705 T5519 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 755705 T5519 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 755706 T5519 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 755706 T5519 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 755706 T5519 oashc.HttpShardHandlerFactory.getParameter Setting useRetries to: false
[junit4] 2> 755706 T5519 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 755706 T5519 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 755706 T5519 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 755707 T5519 oasc.CoreContainer.load Node Name: 127.0.0.1
[junit4] 2> 755707 T5519 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:58253/solr
[junit4] 2> 755707 T5519 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 755707 T5519 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 755708 T5519 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 755710 T5594 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e2f9070 name:ZooKeeperConnection Watcher:127.0.0.1:58253 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 755710 T5519 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 755710 T5519 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 755713 T5519 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 755714 T5597 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@68c47797 name:ZooKeeperConnection Watcher:127.0.0.1:58253/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 755714 T5519 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 755718 T5519 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 755741 T5558 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 755741 T5597 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 755741 T5575 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 755741 T5547 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 755789 T5580 oasc.ZkController.register We are http://127.0.0.1:38503/collection1/ and leader is http://127.0.0.1:38503/collection1/
[junit4] 2> 755789 T5580 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:38503
[junit4] 2> 755789 T5580 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 755789 T5580 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 755789 T5580 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 755790 T5547 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 755790 T5548 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "core":"collection1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"http://127.0.0.1:38503",
[junit4] 2> "node_name":"127.0.0.1:38503_",
[junit4] 2> "numShards":"1",
[junit4] 2> "state":"active",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "operation":"state"} current state version: 5
[junit4] 2> 755791 T5548 oasco.ReplicaMutator.updateState Update state numShards=1 message={
[junit4] 2> "core":"collection1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"http://127.0.0.1:38503",
[junit4] 2> "node_name":"127.0.0.1:38503_",
[junit4] 2> "numShards":"1",
[junit4] 2> "state":"active",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "operation":"state"}
[junit4] 2> 755892 T5558 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 755892 T5597 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 755892 T5575 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 755892 T5547 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 756720 T5519 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49688_
[junit4] 2> 756721 T5519 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49688_
[junit4] 2> 756722 T5519 oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 756723 T5519 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/shard-2-001/cores
[junit4] 2> 756724 T5519 oasc.CoreDescriptor.<init> CORE DESCRIPTOR: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, instanceDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/shard-2-001/cores/collection1, collection=collection1, absoluteInstDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/shard-2-001/cores/collection1/, coreNodeName=, dataDir=data/, shard=}
[junit4] 2> 756724 T5519 oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/shard-2-001/cores/collection1/
[junit4] 2> 756724 T5519 oasc.CorePropertiesLocator.discover Found 1 core definitions
[junit4] 2> 756725 T5598 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 756725 T5598 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 756725 T5598 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 756725 T5547 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 756726 T5548 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"http://127.0.0.1:49688",
[junit4] 2> "node_name":"127.0.0.1:49688_",
[junit4] 2> "numShards":"1",
[junit4] 2> "state":"down",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "operation":"state"} current state version: 6
[junit4] 2> 756726 T5548 oasco.ReplicaMutator.updateState Update state numShards=1 message={
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"http://127.0.0.1:49688",
[junit4] 2> "node_name":"127.0.0.1:49688_",
[junit4] 2> "numShards":"1",
[junit4] 2> "state":"down",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "operation":"state"}
[junit4] 2> 756726 T5548 oasco.ReplicaMutator.updateState Collection already exists with numShards=1
[junit4] 2> 756726 T5548 oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard1
[junit4] 2> 756828 T5558 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 756828 T5575 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 756828 T5547 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 756828 T5597 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 757726 T5598 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 757726 T5598 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 757726 T5598 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 757726 T5598 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 757727 T5598 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 757727 T5598 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/shard-2-001/cores/collection1/'
[junit4] 2> 757737 T5598 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
[junit4] 2> 757738 T5598 oasc.Config.<init> loaded config solrconfig.xml with version 0
[junit4] 2> 757742 T5598 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
[junit4] 2> 757745 T5598 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
[junit4] 2> 757750 T5598 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 757752 T5598 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
[junit4] 2> 757755 T5598 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 757828 T5598 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 757829 T5598 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 757829 T5598 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 757831 T5598 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 757843 T5598 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 757843 T5598 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 757843 T5598 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/shard-2-001/cores/collection1/, dataDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/shard-2-001/cores/collection1/data/
[junit4] 2> 757843 T5598 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2695e2ad
[junit4] 2> 757844 T5598 oasc.CachingDirectoryFactory.get return new directory for /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/shard-2-001/cores/collection1/data
[junit4] 2> 757844 T5598 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/shard-2-001/cores/collection1/data/index/
[junit4] 2> 757844 T5598 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/shard-2-001/cores/collection1/data/index' doesn't exist. Creating new index...
[junit4] 2> 757845 T5598 oasc.CachingDirectoryFactory.get return new directory for /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/shard-2-001/cores/collection1/data/index
[junit4] 2> 757846 T5598 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=34, maxMergeAtOnceExplicit=43, maxMergedSegmentMB=85.4970703125, floorSegmentMB=1.8056640625, forceMergeDeletesPctAllowed=9.519055254924117, segmentsPerTier=46.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
[junit4] 2> 757866 T5598 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/shard-2-001/cores/collection1/data/index,segFN=segments_1,generation=1}
[junit4] 2> 757866 T5598 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 757871 T5598 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 757872 T5598 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 757872 T5598 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 757872 T5598 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 757872 T5598 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 757872 T5598 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 757873 T5598 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 757873 T5598 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 757873 T5598 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 757875 T5598 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 757876 T5598 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 757877 T5598 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 757878 T5598 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 757884 T5598 oasc.RequestHandlers.initHandlersFromConfig Registered paths: /admin/mbeans,standard,/update/csv,/update/json/docs,/admin/luke,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/schema,/admin/plugins,/admin/logging,/update/json,/admin/threads,/admin/ping,/update,/admin/file
[junit4] 2> 757885 T5598 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
[junit4] 2> 757885 T5598 oasu.UpdateHandler.<init> Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 757886 T5598 oasu.UpdateLog.init Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
[junit4] 2> 757886 T5598 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 757886 T5598 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 757887 T5598 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=31, maxMergeAtOnceExplicit=10, maxMergedSegmentMB=79.779296875, floorSegmentMB=0.591796875, forceMergeDeletesPctAllowed=3.1114218803298144, segmentsPerTier=35.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 757888 T5598 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/shard-2-001/cores/collection1/data/index,segFN=segments_1,generation=1}
[junit4] 2> 757888 T5598 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 757888 T5598 oass.SolrIndexSearcher.<init> Opening Searcher@6a065db9[collection1] main
[junit4] 2> 757888 T5598 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 757889 T5598 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 757889 T5598 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 757890 T5598 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 757890 T5598 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 757890 T5598 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 757890 T5598 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 757890 T5598 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 757891 T5598 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 757891 T5598 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4] 2> 757891 T5599 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6a065db9[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 757892 T5598 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 757893 T5602 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:49688 collection:collection1 shard:shard1
[junit4] 2> 757893 T5519 oass.SolrDispatchFilter.init user.dir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4] 2> 757893 T5519 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 757909 T5602 oasc.ZkController.register We are http://127.0.0.1:49688/collection1/ and leader is http://127.0.0.1:38503/collection1/
[junit4] 2> 757909 T5602 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:49688
[junit4] 2> 757909 T5602 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4] 2> 757910 T5602 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4] 2> ASYNC NEW_CORE C1704 name=collection1 org.apache.solr.core.SolrCore@46754d2b url=http://127.0.0.1:49688/collection1 node=127.0.0.1:49688_ C1704_STATE=coll:collection1 core:collection1 props:{core=collection1, base_url=http://127.0.0.1:49688, node_name=127.0.0.1:49688_, state=down}
[junit4] 2> 757910 T5603 C1704 P49688 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4] 2> 757910 T5603 C1704 P49688 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4] 2> 757910 T5603 C1704 P49688 oasc.RecoveryStrategy.doRecovery Publishing state of core collection1 as recovering, leader is http://127.0.0.1:38503/collection1/ and I am http://127.0.0.1:49688/collection1/
[junit4] 2> 757911 T5603 C1704 P49688 oasc.ZkController.publish publishing core=collection1 state=recovering collection=collection1
[junit4] 2> 757911 T5603 C1704 P49688 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 757912 T5547 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 757912 T5603 C1704 P49688 oasc.RecoveryStrategy.sendPrepRecoveryCmd Sending prep recovery command to http://127.0.0.1:38503; WaitFo
[...truncated too long message...]
tSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4] 2> 793636 T5689 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4] 2> 793636 T5689 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4] 2> 793639 T5689 oasc.CachingDirectoryFactory.close Closing StandardDirectoryFactory - 2 directories currently being tracked
[junit4] 2> 793639 T5689 oasc.CachingDirectoryFactory.closeCacheValue looking to close /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/shard-3-001/cores/collection1/data [CachedDir<<refCount=0;path=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/shard-3-001/cores/collection1/data;done=false>>]
[junit4] 2> 793639 T5689 oasc.CachingDirectoryFactory.close Closing directory: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/shard-3-001/cores/collection1/data
[junit4] 2> 793639 T5689 oasc.CachingDirectoryFactory.closeCacheValue looking to close /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/shard-3-001/cores/collection1/data/index [CachedDir<<refCount=0;path=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/shard-3-001/cores/collection1/data/index;done=false>>]
[junit4] 2> 793640 T5689 oasc.CachingDirectoryFactory.close Closing directory: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001/shard-3-001/cores/collection1/data/index
[junit4] 2> 794136 T5690 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 794137 T5519 oejs.AbstractConnector.doStop Stopped ServerConnector@3ece352{HTTP/1.1}{127.0.0.1:0}
[junit4] 2> 794138 T5519 oejsh.ContextHandler.doStop Stopped o.e.j.s.ServletContextHandler@147bdf6a{/,null,UNAVAILABLE}
[junit4] 2> 794139 T5519 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:58253 58253
[junit4] 2> 794151 T5520 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:58253 58253
[junit4] 2> 794152 T5520 oasc.ZkTestServer$ZKServerMain.runFromConfig WARN Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 6 /solr/aliases.json
[junit4] 2> 6 /solr/clusterstate.json
[junit4] 2> 5 /solr/configs/conf1
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 2 /solr/overseer_elect/election/93523738476478467-127.0.0.1:46147_-n_0000000000
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 5 /solr/live_nodes
[junit4] 2> 3 /solr/overseer/queue
[junit4] 2> 3 /solr/overseer/collection-queue-work
[junit4] 2>
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=ChaosMonkeyNothingIsSafeTest -Dtests.method=test -Dtests.seed=7B49B40F4E87B816 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=sr_CS -Dtests.timezone=Antarctica/Mawson -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
[junit4] FAILURE 42.4s J1 | ChaosMonkeyNothingIsSafeTest.test <<<
[junit4] > Throwable #1: java.lang.AssertionError: There were too many update fails - we expect it can happen, but shouldn't easily
[junit4] > at __randomizedtesting.SeedInfo.seed([7B49B40F4E87B816:F31D8BD5E07BD5EE]:0)
[junit4] > at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.test(ChaosMonkeyNothingIsSafeTest.java:222)
[junit4] > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:958)
[junit4] > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:933)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 794161 T5519 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4] 2> NOTE: leaving temporary files on disk at: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 7B49B40F4E87B816-001
[junit4] 2> 42428 T5518 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
[junit4] 2> NOTE: test params are: codec=FastDecompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST_DECOMPRESSION, chunkSize=1785, maxDocsPerChunk=5, blockSize=5), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST_DECOMPRESSION, chunkSize=1785, blockSize=5)), sim=RandomSimilarityProvider(queryNorm=false,coord=crazy): {}, locale=sr_CS, timezone=Antarctica/Mawson
[junit4] 2> NOTE: Linux 3.13.0-46-generic amd64/Oracle Corporation 1.8.0_60-ea (64-bit)/cpus=12,threads=1,free=326029448,total=536870912
[junit4] 2> NOTE: All tests run in this JVM: [TestCloudManagedSchema, TestOmitPositions, JSONWriterTest, FieldMutatingUpdateProcessorTest, TestCustomSort, SolrCmdDistributorTest, IndexSchemaRuntimeFieldTest, DistributedFacetPivotSmallTest, TestRTGBase, WordBreakSolrSpellCheckerTest, TestManagedSchema, CacheHeaderTest, ReturnFieldsTest, ShardRoutingTest, LukeRequestHandlerTest, TestSolrDeletionPolicy2, TestSerializedLuceneMatchVersion, DistributedMLTComponentTest, TestReplicationHandlerBackup, TestPivotHelperCode, TestCollationFieldDocValues, TestReplicaProperties, OpenCloseCoreStressTest, TestSolrQueryParserDefaultOperatorResource, BasicFunctionalityTest, TestSolrConfigHandlerConcurrent, SpatialFilterTest, TestCoreContainer, SharedFSAutoReplicaFailoverTest, HdfsBasicDistributedZk2Test, SuggesterTSTTest, FieldAnalysisRequestHandlerTest, TestBulkSchemaConcurrent, DocValuesMultiTest, TestConfigSets, TestFastWriter, CoreAdminHandlerTest, ShardRoutingCustomTest, AlternateDirectoryTest, RequiredFieldsTest, TestLMDirichletSimilarityFactory, SpellPossibilityIteratorTest, TestJettySolrRunner, DisMaxRequestHandlerTest, TestOrdValues, TestImplicitCoreProperties, TriLevelCompositeIdRoutingTest, TestValueSourceCache, BadComponentTest, SystemInfoHandlerTest, CopyFieldTest, JsonLoaderTest, BlockCacheTest, TestConfigOverlay, TestFastLRUCache, TestElisionMultitermQuery, SpellCheckComponentTest, DistanceFunctionTest, TestRecoveryHdfs, SimpleMLTQParserTest, ResourceLoaderTest, TestFuzzyAnalyzedSuggestions, TestManagedSchemaDynamicFieldResource, TestWriterPerf, TestSolrCoreProperties, TestLRUStatsCache, TestNRTOpen, DocumentBuilderTest, CurrencyFieldOpenExchangeTest, ClusterStateUpdateTest, TestExceedMaxTermLength, TestTolerantSearch, TestCloudSchemaless, TestAddFieldRealTimeGet, DirectSolrConnectionTest, IgnoreCommitOptimizeUpdateProcessorFactoryTest, TestFiltering, DateRangeFieldTest, ChaosMonkeySafeLeaderTest, UnloadDistributedZkTest, CollectionsAPIDistributedZkTest, SyncSliceTest, FullSolrCloudDistribCmdsTest, TestDistributedGrouping, TestFaceting, TestHashPartitioner, DistributedSpellCheckComponentTest, HardAutoCommitTest, TestRangeQuery, StatsComponentTest, TestFunctionQuery, TestBadConfig, TestIndexSearcher, ShowFileRequestHandlerTest, DistributedQueryElevationComponentTest, SolrIndexSplitterTest, SimplePostToolTest, TestCoreDiscovery, AnalysisAfterCoreReloadTest, SignatureUpdateProcessorFactoryTest, TestExtendedDismaxParser, TestFoldingMultitermQuery, NoCacheHeaderTest, SchemaVersionSpecificBehaviorTest, TestUpdate, TestAtomicUpdateErrorCases, DefaultValueUpdateProcessorTest, SortByFunctionTest, TestRemoteStreaming, TestSolrDeletionPolicy1, XsltUpdateRequestHandlerTest, DebugComponentTest, PrimitiveFieldTypeTest, FileBasedSpellCheckerTest, SolrPluginUtilsTest, TestCSVResponseWriter, TestAnalyzedSuggestions, TestLFUCache, TestSolrIndexConfig, ResponseLogComponentTest, MultiTermTest, SampleTest, TestCharFilters, TestCodecSupport, SynonymTokenizerTest, EchoParamsTest, TestDFRSimilarityFactory, TestLMJelinekMercerSimilarityFactory, TestJmxMonitoredMap, OpenExchangeRatesOrgProviderTest, ChaosMonkeyNothingIsSafeTest]
[junit4] Completed on J1 in 43.52s, 1 test, 1 failure <<< FAILURES!
[...truncated 368 lines...]
BUILD FAILED
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:519: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:61: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:191: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:510: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1351: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:958: There were test failures: 476 suites, 1894 tests, 1 failure, 48 ignored (21 assumptions)
Total time: 43 minutes 23 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.8.0_60-ea-b06 -XX:+UseCompressedOops -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any