You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2017/12/21 15:19:48 UTC
[JENKINS] Lucene-Solr-Tests-7.x - Build # 290 - Still Failing
Build: https://builds.apache.org/job/Lucene-Solr-Tests-7.x/290/
No tests ran.
Build Log:
[...truncated 95 lines...]
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 216006656 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/hs_err_pid13636.log
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[Fast Archiver] No artifacts from Lucene-Solr-Tests-7.x #288 to compare, so performing full copy of artifacts
Recording test results
ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error?
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Lucene-Solr-Tests-7.x - Build # 294 - Still unstable
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-7.x/294/
7 tests failed.
FAILED: org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testCollectionsAPI
Error Message:
We have shards using the same indexDir. E.g. shards [collection2.shard6.replica_n31, collection2.shard5.replica_n24] all use indexDir (closed)
Stack Trace:
java.lang.AssertionError: We have shards using the same indexDir. E.g. shards [collection2.shard6.replica_n31, collection2.shard5.replica_n24] all use indexDir (closed)
at __randomizedtesting.SeedInfo.seed([1A946A2BB09AB81:49DC3216BD3A8414]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.checkNoTwoShardsUseTheSameIndexDir(CollectionsAPIDistributedZkTest.java:602)
at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testCollectionsAPI(CollectionsAPIDistributedZkTest.java:477)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: org.apache.solr.cloud.LeaderInitiatedRecoveryOnCommitTest.test
Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:39702
Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:39702
at __randomizedtesting.SeedInfo.seed([1A946A2BB09AB81:89FD797815F5C679]:0)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1104)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:409)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:334)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:991)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
... 44 more
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.LeaderInitiatedRecoveryOnCommitTest
Error Message:
36 threads leaked from SUITE scope at org.apache.solr.cloud.LeaderInitiatedRecoveryOnCommitTest: 1) Thread[id=17503, name=org.eclipse.jetty.server.session.HashSessionManager@57a59fc8Timer, state=TIMED_WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2) Thread[id=17525, name=Scheduler-1563745040, state=TIMED_WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 3) Thread[id=17559, name=searcherExecutor-5887-thread-1, state=WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 4) Thread[id=17500, name=qtp797491010-17500, state=TIMED_WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748) 5) Thread[id=17553, name=ScheduledTrigger-5879-thread-2, state=WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 6) Thread[id=17497, name=qtp797491010-17497-acceptor-0@71f27863-ServerConnector@14b5a9ad{HTTP/1.1,[http/1.1]}{127.0.0.1:34828}, state=RUNNABLE, group=TGRP-LeaderInitiatedRecoveryOnCommitTest] at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:371) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:601) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang.Thread.run(Thread.java:748) 7) Thread[id=17519, name=Connection evictor, state=TIMED_WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest] at java.lang.Thread.sleep(Native Method) at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.lang.Thread.run(Thread.java:748) 8) Thread[id=17581, name=AutoscalingActionExecutor-5880-thread-1, state=WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 9) Thread[id=17507, name=Connection evictor, state=TIMED_WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest] at java.lang.Thread.sleep(Native Method) at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.lang.Thread.run(Thread.java:748) 10) Thread[id=17545, name=OverseerCollectionConfigSetProcessor-99232747526422539-127.0.0.1:43705_-n_0000000002, state=TIMED_WAITING, group=Overseer collection creation process.] at java.lang.Thread.sleep(Native Method) at org.apache.solr.common.cloud.ZkCmdExecutor.retryDelay(ZkCmdExecutor.java:111) at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:70) at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:339) at org.apache.solr.cloud.OverseerTaskProcessor.amILeader(OverseerTaskProcessor.java:387) at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:193) at java.lang.Thread.run(Thread.java:748) 11) Thread[id=17499, name=qtp797491010-17499, state=TIMED_WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748) 12) Thread[id=17577, name=indexFetcher-5893-thread-1, state=TIMED_WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 13) Thread[id=17585, name=zkCallback-3025-thread-7, state=TIMED_WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 14) Thread[id=17508, name=Connection evictor, state=TIMED_WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest] at java.lang.Thread.sleep(Native Method) at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.lang.Thread.run(Thread.java:748) 15) Thread[id=17501, name=qtp797491010-17501, state=TIMED_WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748) 16) Thread[id=17498, name=qtp797491010-17498, state=RUNNABLE, group=TGRP-LeaderInitiatedRecoveryOnCommitTest] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang.Thread.run(Thread.java:748) 17) Thread[id=17550, name=ScheduledTrigger-5879-thread-1, state=TIMED_WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 18) Thread[id=17555, name=ScheduledTrigger-5879-thread-3, state=WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 19) Thread[id=17527, name=zkCallback-3025-thread-2, state=TIMED_WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 20) Thread[id=17495, name=qtp797491010-17495, state=RUNNABLE, group=TGRP-LeaderInitiatedRecoveryOnCommitTest] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang.Thread.run(Thread.java:748) 21) Thread[id=17528, name=zkCallback-3025-thread-3, state=TIMED_WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 22) Thread[id=17543, name=zkConnectionManagerCallback-3026-thread-1-processing-n:127.0.0.1:43705_-EventThread, state=WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501) 23) Thread[id=17556, name=searcherExecutor-5882-thread-1, state=WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 24) Thread[id=17515, name=zkCallback-3025-thread-1, state=TIMED_WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 25) Thread[id=17542, name=zkConnectionManagerCallback-3026-thread-1-processing-n:127.0.0.1:43705_-SendThread(127.0.0.1:59987), state=TIMED_WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest] at java.lang.Thread.sleep(Native Method) at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060) 26) Thread[id=17544, name=OverseerStateUpdate-99232747526422539-127.0.0.1:43705_-n_0000000002, state=TIMED_WAITING, group=Overseer state updater.] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.apache.solr.cloud.ZkDistributedQueue.peekElements(ZkDistributedQueue.java:463) at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:203) at java.lang.Thread.run(Thread.java:748) 27) Thread[id=17546, name=OverseerAutoScalingTriggerThread-99232747526422539-127.0.0.1:43705_-n_0000000002, state=WAITING, group=Overseer autoscaling triggers] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:166) at java.lang.Thread.run(Thread.java:748) 28) Thread[id=17541, name=zkCallback-3025-thread-5, state=TIMED_WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 29) Thread[id=17584, name=zkCallback-3025-thread-6, state=TIMED_WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 30) Thread[id=17502, name=qtp797491010-17502, state=TIMED_WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748) 31) Thread[id=17548, name=OverseerThreadFactory-5881-thread-1, state=WAITING, group=Overseer collection creation process.] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 32) Thread[id=17529, name=zkCallback-3025-thread-4, state=TIMED_WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 33) Thread[id=17557, name=ScheduledTrigger-5879-thread-4, state=WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 34) Thread[id=17514, name=zkConnectionManagerCallback-3026-thread-1, state=WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 35) Thread[id=17517, name=Thread-2506, state=WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:502) at org.apache.solr.core.CloserThread.run(CoreContainer.java:1719) 36) Thread[id=17496, name=qtp797491010-17496, state=TIMED_WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 36 threads leaked from SUITE scope at org.apache.solr.cloud.LeaderInitiatedRecoveryOnCommitTest:
1) Thread[id=17503, name=org.eclipse.jetty.server.session.HashSessionManager@57a59fc8Timer, state=TIMED_WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2) Thread[id=17525, name=Scheduler-1563745040, state=TIMED_WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
3) Thread[id=17559, name=searcherExecutor-5887-thread-1, state=WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
4) Thread[id=17500, name=qtp797491010-17500, state=TIMED_WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
5) Thread[id=17553, name=ScheduledTrigger-5879-thread-2, state=WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
6) Thread[id=17497, name=qtp797491010-17497-acceptor-0@71f27863-ServerConnector@14b5a9ad{HTTP/1.1,[http/1.1]}{127.0.0.1:34828}, state=RUNNABLE, group=TGRP-LeaderInitiatedRecoveryOnCommitTest]
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:371)
at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:601)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
7) Thread[id=17519, name=Connection evictor, state=TIMED_WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.lang.Thread.run(Thread.java:748)
8) Thread[id=17581, name=AutoscalingActionExecutor-5880-thread-1, state=WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
9) Thread[id=17507, name=Connection evictor, state=TIMED_WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.lang.Thread.run(Thread.java:748)
10) Thread[id=17545, name=OverseerCollectionConfigSetProcessor-99232747526422539-127.0.0.1:43705_-n_0000000002, state=TIMED_WAITING, group=Overseer collection creation process.]
at java.lang.Thread.sleep(Native Method)
at org.apache.solr.common.cloud.ZkCmdExecutor.retryDelay(ZkCmdExecutor.java:111)
at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:70)
at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:339)
at org.apache.solr.cloud.OverseerTaskProcessor.amILeader(OverseerTaskProcessor.java:387)
at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:193)
at java.lang.Thread.run(Thread.java:748)
11) Thread[id=17499, name=qtp797491010-17499, state=TIMED_WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
12) Thread[id=17577, name=indexFetcher-5893-thread-1, state=TIMED_WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
13) Thread[id=17585, name=zkCallback-3025-thread-7, state=TIMED_WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
14) Thread[id=17508, name=Connection evictor, state=TIMED_WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.lang.Thread.run(Thread.java:748)
15) Thread[id=17501, name=qtp797491010-17501, state=TIMED_WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
16) Thread[id=17498, name=qtp797491010-17498, state=RUNNABLE, group=TGRP-LeaderInitiatedRecoveryOnCommitTest]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
17) Thread[id=17550, name=ScheduledTrigger-5879-thread-1, state=TIMED_WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
18) Thread[id=17555, name=ScheduledTrigger-5879-thread-3, state=WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
19) Thread[id=17527, name=zkCallback-3025-thread-2, state=TIMED_WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
20) Thread[id=17495, name=qtp797491010-17495, state=RUNNABLE, group=TGRP-LeaderInitiatedRecoveryOnCommitTest]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
21) Thread[id=17528, name=zkCallback-3025-thread-3, state=TIMED_WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
22) Thread[id=17543, name=zkConnectionManagerCallback-3026-thread-1-processing-n:127.0.0.1:43705_-EventThread, state=WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)
23) Thread[id=17556, name=searcherExecutor-5882-thread-1, state=WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
24) Thread[id=17515, name=zkCallback-3025-thread-1, state=TIMED_WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
25) Thread[id=17542, name=zkConnectionManagerCallback-3026-thread-1-processing-n:127.0.0.1:43705_-SendThread(127.0.0.1:59987), state=TIMED_WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
26) Thread[id=17544, name=OverseerStateUpdate-99232747526422539-127.0.0.1:43705_-n_0000000002, state=TIMED_WAITING, group=Overseer state updater.]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.apache.solr.cloud.ZkDistributedQueue.peekElements(ZkDistributedQueue.java:463)
at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:203)
at java.lang.Thread.run(Thread.java:748)
27) Thread[id=17546, name=OverseerAutoScalingTriggerThread-99232747526422539-127.0.0.1:43705_-n_0000000002, state=WAITING, group=Overseer autoscaling triggers]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:166)
at java.lang.Thread.run(Thread.java:748)
28) Thread[id=17541, name=zkCallback-3025-thread-5, state=TIMED_WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
29) Thread[id=17584, name=zkCallback-3025-thread-6, state=TIMED_WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
30) Thread[id=17502, name=qtp797491010-17502, state=TIMED_WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
31) Thread[id=17548, name=OverseerThreadFactory-5881-thread-1, state=WAITING, group=Overseer collection creation process.]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
32) Thread[id=17529, name=zkCallback-3025-thread-4, state=TIMED_WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
33) Thread[id=17557, name=ScheduledTrigger-5879-thread-4, state=WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
34) Thread[id=17514, name=zkConnectionManagerCallback-3026-thread-1, state=WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
35) Thread[id=17517, name=Thread-2506, state=WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at org.apache.solr.core.CloserThread.run(CoreContainer.java:1719)
36) Thread[id=17496, name=qtp797491010-17496, state=TIMED_WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
at __randomizedtesting.SeedInfo.seed([1A946A2BB09AB81]:0)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.LeaderInitiatedRecoveryOnCommitTest
Error Message:
There are still zombie threads that couldn't be terminated: 1) Thread[id=17528, name=zkCallback-3025-thread-3, state=TIMED_WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2) Thread[id=17503, name=org.eclipse.jetty.server.session.HashSessionManager@57a59fc8Timer, state=TIMED_WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 3) Thread[id=17556, name=searcherExecutor-5882-thread-1, state=WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 4) Thread[id=17525, name=Scheduler-1563745040, state=WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 5) Thread[id=17559, name=searcherExecutor-5887-thread-1, state=WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 6) Thread[id=17553, name=ScheduledTrigger-5879-thread-2, state=WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 7) Thread[id=17515, name=zkCallback-3025-thread-1, state=TIMED_WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 8) Thread[id=17581, name=AutoscalingActionExecutor-5880-thread-1, state=WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 9) Thread[id=17542, name=zkConnectionManagerCallback-3026-thread-1-processing-n:127.0.0.1:43705_-SendThread(127.0.0.1:59987), state=TIMED_WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest] at java.lang.Thread.sleep(Native Method) at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060) 10) Thread[id=17577, name=indexFetcher-5893-thread-1, state=TIMED_WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 11) Thread[id=17585, name=zkCallback-3025-thread-7, state=TIMED_WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 12) Thread[id=17541, name=zkCallback-3025-thread-5, state=TIMED_WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 13) Thread[id=17584, name=zkCallback-3025-thread-6, state=TIMED_WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 14) Thread[id=17529, name=zkCallback-3025-thread-4, state=TIMED_WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 15) Thread[id=17557, name=ScheduledTrigger-5879-thread-4, state=WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 16) Thread[id=17514, name=zkConnectionManagerCallback-3026-thread-1, state=WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 17) Thread[id=17498, name=qtp797491010-17498, state=RUNNABLE, group=TGRP-LeaderInitiatedRecoveryOnCommitTest] at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method) at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:317) at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:207) at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:213) at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:219) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:92) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang.Thread.run(Thread.java:748) 18) Thread[id=17550, name=ScheduledTrigger-5879-thread-1, state=TIMED_WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 19) Thread[id=17517, name=Thread-2506, state=WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:502) at org.apache.solr.core.CloserThread.run(CoreContainer.java:1719) 20) Thread[id=17555, name=ScheduledTrigger-5879-thread-3, state=WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 21) Thread[id=17527, name=zkCallback-3025-thread-2, state=TIMED_WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 22) Thread[id=17495, name=qtp797491010-17495, state=RUNNABLE, group=TGRP-LeaderInitiatedRecoveryOnCommitTest] at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method) at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:317) at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:207) at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:213) at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:219) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:92) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang.Thread.run(Thread.java:748)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
1) Thread[id=17528, name=zkCallback-3025-thread-3, state=TIMED_WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2) Thread[id=17503, name=org.eclipse.jetty.server.session.HashSessionManager@57a59fc8Timer, state=TIMED_WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
[...truncated too long message...]
nchronousQueue.poll(SynchronousQueue.java:941)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] > 15) Thread[id=17557, name=ScheduledTrigger-5879-thread-4, state=WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
[junit4] > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
[junit4] > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] > 16) Thread[id=17514, name=zkConnectionManagerCallback-3026-thread-1, state=WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
[junit4] > at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] > 17) Thread[id=17498, name=qtp797491010-17498, state=RUNNABLE, group=TGRP-LeaderInitiatedRecoveryOnCommitTest]
[junit4] > at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method)
[junit4] > at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:317)
[junit4] > at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:207)
[junit4] > at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:213)
[junit4] > at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:219)
[junit4] > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:92)
[junit4] > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
[junit4] > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
[junit4] > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
[junit4] > at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)
[junit4] > at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)
[junit4] > at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)
[junit4] > at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
[junit4] > at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] > 18) Thread[id=17550, name=ScheduledTrigger-5879-thread-1, state=TIMED_WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
[junit4] > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
[junit4] > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] > 19) Thread[id=17517, name=Thread-2506, state=WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest]
[junit4] > at java.lang.Object.wait(Native Method)
[junit4] > at java.lang.Object.wait(Object.java:502)
[junit4] > at org.apache.solr.core.CloserThread.run(CoreContainer.java:1719)
[junit4] > 20) Thread[id=17555, name=ScheduledTrigger-5879-thread-3, state=WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
[junit4] > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
[junit4] > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] > 21) Thread[id=17527, name=zkCallback-3025-thread-2, state=TIMED_WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
[junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
[junit4] > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] > 22) Thread[id=17495, name=qtp797491010-17495, state=RUNNABLE, group=TGRP-LeaderInitiatedRecoveryOnCommitTest]
[junit4] > at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method)
[junit4] > at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:317)
[junit4] > at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:207)
[junit4] > at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:213)
[junit4] > at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:219)
[junit4] > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:92)
[junit4] > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
[junit4] > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
[junit4] > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
[junit4] > at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)
[junit4] > at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)
[junit4] > at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)
[junit4] > at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
[junit4] > at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] > at __randomizedtesting.SeedInfo.seed([1A946A2BB09AB81]:0)
[junit4] Completed [526/764 (5!)] on J0 in 130.48s, 1 test, 3 errors <<< FAILURES!
[...truncated 48812 lines...]
[JENKINS] Lucene-Solr-Tests-7.x - Build # 293 - Still Failing
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-7.x/293/
No tests ran.
Build Log:
[...truncated 11810 lines...]
[junit4] Suite: org.apache.solr.cloud.autoscaling.SystemLogListenerTest
[junit4] 2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.SystemLogListenerTest_B11D509A9403386F-001/init-core-data-001
[junit4] 2> 255947 WARN (SUITE-SystemLogListenerTest-seed#[B11D509A9403386F]-worker) [ ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=61 numCloses=61
[junit4] 2> 255947 INFO (SUITE-SystemLogListenerTest-seed#[B11D509A9403386F]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
[junit4] 2> 256096 INFO (SUITE-SystemLogListenerTest-seed#[B11D509A9403386F]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
[junit4] 2> 256097 INFO (SUITE-SystemLogListenerTest-seed#[B11D509A9403386F]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
[junit4] 2> 256102 INFO (SUITE-SystemLogListenerTest-seed#[B11D509A9403386F]-worker) [ ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 3 servers in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.SystemLogListenerTest_B11D509A9403386F-001/tempDir-001
[junit4] 2> 256102 INFO (SUITE-SystemLogListenerTest-seed#[B11D509A9403386F]-worker) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 256102 INFO (Thread-45) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 256102 INFO (Thread-45) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 256192 ERROR (Thread-45) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 256212 INFO (SUITE-SystemLogListenerTest-seed#[B11D509A9403386F]-worker) [ ] o.a.s.c.ZkTestServer start zk server on port:32962
[junit4] 2> 256256 INFO (zkConnectionManagerCallback-117-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 256303 INFO (jetty-launcher-114-thread-1) [ ] o.e.j.s.Server jetty-9.3.20.v20170531
[junit4] 2> 256346 INFO (jetty-launcher-114-thread-2) [ ] o.e.j.s.Server jetty-9.3.20.v20170531
[junit4] 2> 256549 INFO (jetty-launcher-114-thread-1) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@66132353{/solr,null,AVAILABLE}
[junit4] 2> 256666 INFO (jetty-launcher-114-thread-2) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4f601a11{/solr,null,AVAILABLE}
[junit4] 2> 256667 INFO (jetty-launcher-114-thread-3) [ ] o.e.j.s.Server jetty-9.3.20.v20170531
[junit4] 2> 256864 INFO (jetty-launcher-114-thread-3) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@25dff98d{/solr,null,AVAILABLE}
[junit4] 2> 256667 INFO (jetty-launcher-114-thread-2) [ ] o.e.j.s.AbstractConnector Started ServerConnector@302eac67{HTTP/1.1,[http/1.1]}{127.0.0.1:45421}
[junit4] 2> 256957 INFO (jetty-launcher-114-thread-2) [ ] o.e.j.s.Server Started @281299ms
[junit4] 2> 256957 INFO (jetty-launcher-114-thread-2) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=45421}
[junit4] 2> 256957 ERROR (jetty-launcher-114-thread-2) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 256958 INFO (jetty-launcher-114-thread-2) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 7.3.0
[junit4] 2> 256958 INFO (jetty-launcher-114-thread-2) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 256958 INFO (jetty-launcher-114-thread-2) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 256958 INFO (jetty-launcher-114-thread-2) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-12-23T05:36:10.929Z
[junit4] 2> 256865 INFO (jetty-launcher-114-thread-3) [ ] o.e.j.s.AbstractConnector Started ServerConnector@3754e8ab{HTTP/1.1,[http/1.1]}{127.0.0.1:51580}
[junit4] 2> 256958 INFO (jetty-launcher-114-thread-3) [ ] o.e.j.s.Server Started @281300ms
[junit4] 2> 256958 INFO (jetty-launcher-114-thread-3) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=51580}
[junit4] 2> 257096 INFO (jetty-launcher-114-thread-1) [ ] o.e.j.s.AbstractConnector Started ServerConnector@4eaa8e6b{HTTP/1.1,[http/1.1]}{127.0.0.1:48528}
[junit4] 2> 257096 INFO (jetty-launcher-114-thread-1) [ ] o.e.j.s.Server Started @281438ms
[junit4] 2> 257096 INFO (jetty-launcher-114-thread-1) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=48528}
[junit4] 2> 256959 ERROR (jetty-launcher-114-thread-3) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 257096 ERROR (jetty-launcher-114-thread-1) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 257103 INFO (jetty-launcher-114-thread-3) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 7.3.0
[junit4] 2> 257141 INFO (jetty-launcher-114-thread-3) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 257141 INFO (jetty-launcher-114-thread-3) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 257141 INFO (jetty-launcher-114-thread-3) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-12-23T05:36:11.112Z
[junit4] 2> 257105 INFO (jetty-launcher-114-thread-1) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 7.3.0
[junit4] 2> 257141 INFO (jetty-launcher-114-thread-1) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 257141 INFO (jetty-launcher-114-thread-1) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 257142 INFO (jetty-launcher-114-thread-1) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-12-23T05:36:11.113Z
[junit4] 2> 257142 INFO (zkConnectionManagerCallback-119-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 257195 INFO (zkConnectionManagerCallback-121-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 257366 INFO (jetty-launcher-114-thread-2) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 258446 INFO (jetty-launcher-114-thread-2) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:32962/solr
[junit4] 2> 258457 INFO (zkConnectionManagerCallback-123-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 258526 INFO (jetty-launcher-114-thread-3) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 258578 INFO (jetty-launcher-114-thread-3) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:32962/solr
[junit4] 2> 258606 INFO (jetty-launcher-114-thread-1) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 258612 INFO (zkConnectionManagerCallback-131-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 258645 INFO (zkConnectionManagerCallback-127-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 258681 INFO (jetty-launcher-114-thread-1) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:32962/solr
[junit4] 2> 258846 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x16081df195a0004, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 258847 INFO (zkConnectionManagerCallback-135-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 258967 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x16081df195a0006, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 258970 INFO (zkConnectionManagerCallback-137-thread-1-processing-n:127.0.0.1:51580_solr) [n:127.0.0.1:51580_solr ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 258974 INFO (zkConnectionManagerCallback-139-thread-1-processing-n:127.0.0.1:45421_solr) [n:127.0.0.1:45421_solr ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 259197 INFO (zkConnectionManagerCallback-141-thread-1-processing-n:127.0.0.1:48528_solr) [n:127.0.0.1:48528_solr ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 260711 INFO (jetty-launcher-114-thread-3) [n:127.0.0.1:51580_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 260712 INFO (jetty-launcher-114-thread-1) [n:127.0.0.1:48528_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 260712 INFO (jetty-launcher-114-thread-2) [n:127.0.0.1:45421_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 260714 INFO (jetty-launcher-114-thread-3) [n:127.0.0.1:51580_solr ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:51580_solr
[junit4] 2> 260715 INFO (jetty-launcher-114-thread-1) [n:127.0.0.1:48528_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:48528_solr
[junit4] 2> 260715 INFO (jetty-launcher-114-thread-2) [n:127.0.0.1:45421_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45421_solr
[junit4] 2> 260716 INFO (jetty-launcher-114-thread-3) [n:127.0.0.1:51580_solr ] o.a.s.c.Overseer Overseer (id=99221987005169671-127.0.0.1:51580_solr-n_0000000000) starting
[junit4] 2> 261686 INFO (jetty-launcher-114-thread-1) [n:127.0.0.1:48528_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_48528.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2a2fd3ac
[junit4] 2> 261763 INFO (jetty-launcher-114-thread-1) [n:127.0.0.1:48528_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_48528.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2a2fd3ac
[junit4] 2> 261763 INFO (jetty-launcher-114-thread-1) [n:127.0.0.1:48528_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_48528.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2a2fd3ac
[junit4] 2> 261792 INFO (jetty-launcher-114-thread-1) [n:127.0.0.1:48528_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.SystemLogListenerTest_B11D509A9403386F-001/tempDir-001/node1/.
[junit4] 2> 262077 INFO (zkCallback-136-thread-1-processing-n:127.0.0.1:51580_solr) [n:127.0.0.1:51580_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 262077 INFO (zkCallback-138-thread-1-processing-n:127.0.0.1:45421_solr) [n:127.0.0.1:45421_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 262077 INFO (zkCallback-140-thread-1-processing-n:127.0.0.1:48528_solr) [n:127.0.0.1:48528_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 262326 INFO (jetty-launcher-114-thread-3) [n:127.0.0.1:51580_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51580_solr
[junit4] 2> 262327 INFO (zkCallback-140-thread-1-processing-n:127.0.0.1:48528_solr) [n:127.0.0.1:48528_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 262327 INFO (zkCallback-138-thread-1-processing-n:127.0.0.1:45421_solr) [n:127.0.0.1:45421_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 262345 INFO (zkCallback-136-thread-1-processing-n:127.0.0.1:51580_solr) [n:127.0.0.1:51580_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 262367 DEBUG (OverseerAutoScalingTriggerThread-99221987005169671-127.0.0.1:51580_solr-n_0000000000) [n:127.0.0.1:51580_solr ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
[junit4] 2> 262368 DEBUG (OverseerAutoScalingTriggerThread-99221987005169671-127.0.0.1:51580_solr-n_0000000000) [n:127.0.0.1:51580_solr ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
[junit4] 2> 262380 DEBUG (OverseerAutoScalingTriggerThread-99221987005169671-127.0.0.1:51580_solr-n_0000000000) [n:127.0.0.1:51580_solr ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:51580_solr, 127.0.0.1:48528_solr, 127.0.0.1:45421_solr]
[junit4] 2> 262380 DEBUG (OverseerAutoScalingTriggerThread-99221987005169671-127.0.0.1:51580_solr-n_0000000000) [n:127.0.0.1:51580_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion -1
[junit4] 2> 262380 DEBUG (OverseerAutoScalingTriggerThread-99221987005169671-127.0.0.1:51580_solr-n_0000000000) [n:127.0.0.1:51580_solr ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
[junit4] 2> 262527 DEBUG (OverseerAutoScalingTriggerThread-99221987005169671-127.0.0.1:51580_solr-n_0000000000) [n:127.0.0.1:51580_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 262527 DEBUG (OverseerAutoScalingTriggerThread-99221987005169671-127.0.0.1:51580_solr-n_0000000000) [n:127.0.0.1:51580_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 1
[junit4] 2> 262527 DEBUG (ScheduledTrigger-150-thread-1) [n:127.0.0.1:51580_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 262869 INFO (jetty-launcher-114-thread-2) [n:127.0.0.1:45421_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_45421.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2a2fd3ac
[junit4] 2> 263367 INFO (jetty-launcher-114-thread-2) [n:127.0.0.1:45421_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_45421.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2a2fd3ac
[junit4] 2> 263367 INFO (jetty-launcher-114-thread-2) [n:127.0.0.1:45421_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_45421.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2a2fd3ac
[junit4] 2> 263428 INFO (jetty-launcher-114-thread-2) [n:127.0.0.1:45421_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.SystemLogListenerTest_B11D509A9403386F-001/tempDir-001/node2/.
[junit4] 2> 263767 INFO (jetty-launcher-114-thread-3) [n:127.0.0.1:51580_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_51580.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2a2fd3ac
[junit4] 2> 263807 DEBUG (ScheduledTrigger-150-thread-1) [n:127.0.0.1:51580_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 263991 INFO (jetty-launcher-114-thread-3) [n:127.0.0.1:51580_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_51580.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2a2fd3ac
[junit4] 2> 263991 INFO (jetty-launcher-114-thread-3) [n:127.0.0.1:51580_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_51580.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2a2fd3ac
[junit4] 2> 264064 INFO (jetty-launcher-114-thread-3) [n:127.0.0.1:51580_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.SystemLogListenerTest_B11D509A9403386F-001/tempDir-001/node3/.
[junit4] 2> 264880 DEBUG (ScheduledTrigger-150-thread-2) [n:127.0.0.1:51580_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 265052 INFO (zkConnectionManagerCallback-149-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 265141 INFO (zkConnectionManagerCallback-153-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 265142 INFO (SUITE-SystemLogListenerTest-seed#[B11D509A9403386F]-worker) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
[junit4] 2> 265143 INFO (SUITE-SystemLogListenerTest-seed#[B11D509A9403386F]-worker) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:32962/solr ready
[junit4] 2> 265259 INFO (qtp342458738-377) [n:127.0.0.1:45421_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=3&name=.system&nrtReplicas=3&action=CREATE&numShards=1&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 265312 INFO (OverseerThreadFactory-152-thread-1-processing-n:127.0.0.1:51580_solr) [n:127.0.0.1:51580_solr ] o.a.s.c.CreateCollectionCmd Create collection .system
[junit4] 2> 265485 INFO (OverseerStateUpdate-99221987005169671-127.0.0.1:51580_solr-n_0000000000) [n:127.0.0.1:51580_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":".system",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":".system_shard1_replica_n1",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:48528/solr",
[junit4] 2> "type":"NRT",
[junit4] 2> "waitForFinalState":"false"}
[junit4] 2> 265543 INFO (OverseerStateUpdate-99221987005169671-127.0.0.1:51580_solr-n_0000000000) [n:127.0.0.1:51580_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":".system",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":".system_shard1_replica_n3",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:45421/solr",
[junit4] 2> "type":"NRT",
[junit4] 2> "waitForFinalState":"false"}
[junit4] 2> 265556 INFO (OverseerStateUpdate-99221987005169671-127.0.0.1:51580_solr-n_0000000000) [n:127.0.0.1:51580_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":".system",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":".system_shard1_replica_n5",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:51580/solr",
[junit4] 2> "type":"NRT",
[junit4] 2> "waitForFinalState":"false"}
[junit4] 2> 265800 INFO (qtp342458738-375) [n:127.0.0.1:45421_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node4&collection.configName=.system&newCollection=true&name=.system_shard1_replica_n3&action=CREATE&numShards=1&collection=.system&shard=shard1&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 265802 INFO (qtp342458738-375) [n:127.0.0.1:45421_solr ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 265984 DEBUG (ScheduledTrigger-150-thread-1) [n:127.0.0.1:51580_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 266257 INFO (qtp1167150554-368) [n:127.0.0.1:48528_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node2&collection.configName=.system&newCollection=true&name=.system_shard1_replica_n1&action=CREATE&numShards=1&collection=.system&shard=shard1&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 266344 INFO (qtp1167150554-368) [n:127.0.0.1:48528_solr ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 266527 INFO (zkCallback-138-thread-1-processing-n:127.0.0.1:45421_solr) [n:127.0.0.1:45421_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
[junit4] 2> 266528 INFO (zkCallback-140-thread-1-processing-n:127.0.0.1:48528_solr) [n:127.0.0.1:48528_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
[junit4] 2> 266623 INFO (qtp1753864094-386) [n:127.0.0.1:51580_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node6&collection.configName=.system&newCollection=true&name=.system_shard1_replica_n5&action=CREATE&numShards=1&collection=.system&shard=shard1&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 266623 INFO (qtp1753864094-386) [n:127.0.0.1:51580_solr ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 266686 INFO (zkCallback-138-thread-2-processing-n:127.0.0.1:45421_solr) [n:127.0.0.1:45421_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
[junit4] 2> 266788 INFO (zkCallback-140-thread-1-processing-n:127.0.0.1:48528_solr) [n:127.0.0.1:48528_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
[junit4] 2> 266789 INFO (zkCallback-136-thread-1-processing-n:127.0.0.1:51580_solr) [n:127.0.0.1:51580_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
[junit4] 2> 266847 INFO (zkCallback-136-thread-2-processing-n:127.0.0.1:51580_solr) [n:127.0.0.1:51580_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
[junit4] 2> 266904 INFO (zkCallback-138-thread-2-processing-n:127.0.0.1:45421_solr) [n:127.0.0.1:45421_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
[junit4] 2> 266904 INFO (zkCallback-138-thread-1-processing-n:127.0.0.1:45421_solr) [n:127.0.0.1:45421_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
[junit4] 2> 266905 INFO (zkCallback-140-thread-2-processing-n:127.0.0.1:48528_solr) [n:127.0.0.1:48528_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
[junit4] 2> 266947 INFO (zkCallback-140-thread-1-processing-n:127.0.0.1:48528_solr) [n:127.0.0.1:48528_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
[junit4] 2> 266960 INFO (zkCallback-138-thread-1-processing-n:127.0.0.1:45421_solr) [n:127.0.0.1:45421_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
[junit4] 2> 266961 INFO (zkCallback-138-thread-2-processing-n:127.0.0.1:45421_solr) [n:127.0.0.1:45421_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
[junit4] 2> 266988 INFO (zkCallback-136-thread-2-processing-n:127.0.0.1:51580_solr) [n:127.0.0.1:51580_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
[junit4] 2> 266988 INFO (zkCallback-140-thread-3-processing-n:127.0.0.1:48528_solr) [n:127.0.0.1:48528_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
[junit4] 2> 267000 DEBUG (ScheduledTrigger-150-thread-1) [n:127.0.0.1:51580_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 267000 INFO (zkCallback-136-thread-1-processing-n:127.0.0.1:51580_solr) [n:127.0.0.1:51580_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
[junit4] 2> 267560 WARN (qtp342458738-375) [n:127.0.0.1:45421_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n3] o.a.s.c.Config You should not use LATEST as luceneMatchVersion property: if you use this setting, and then Solr upgrades to a newer release of Lucene, sizable changes may happen. If precise back compatibility is important then you should instead explicitly specify an actual Lucene version.
[junit4] 2> 267578 INFO (qtp342458738-375) [n:127.0.0.1:45421_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n3] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
[junit4] 2> 267610 INFO (qtp342458738-375) [n:127.0.0.1:45421_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n3] o.a.s.s.ManagedIndexSchemaFactory The schema is configured as managed, but managed schema resource managed-schema not found - loading non-managed schema schema.xml instead
[junit4] 2> 267613 INFO (qtp342458738-375) [n:127.0.0.1:45421_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n3] o.a.s.s.IndexSchema [.system_shard1_replica_n3] Schema name=_system collection or core
[junit4] 2> 267673 INFO (qtp342458738-375) [n:127.0.0.1:45421_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n3] o.a.s.s.IndexSchema Loaded schema _system collection or core/1.1 with uniqueid field id
[junit4] 2> 267694 INFO (qtp342458738-375) [n:127.0.0.1:45421_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n3] o.a.s.s.ManagedIndexSchema Created and persisted managed schema znode at /configs/.system/managed-schema
[junit4] 2> 267710 INFO (qtp342458738-375) [n:127.0.0.1:45421_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n3] o.a.s.s.ManagedIndexSchemaFactory After upgrading to managed schema in ZooKeeper, renamed the non-managed schema /configs/.system/schema.xml to /configs/.system/schema.xml.bak
[junit4] 2> 267710 INFO (qtp342458738-375) [n:127.0.0.1:45421_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n3] o.a.s.c.CoreContainer Creating SolrCore '.system_shard1_replica_n3' using configuration from collection .system, trusted=true
[junit4] 2> 267711 INFO (qtp342458738-375) [n:127.0.0.1:45421_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n3] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_45421.solr.core..system.shard1.replica_n3' (registry 'solr.core..system.shard1.replica_n3') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2a2fd3ac
[junit4] 2> 267712 INFO (qtp342458738-375) [n:127.0.0.1:45421_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n3] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 267712 INFO (qtp342458738-375) [n:127.0.0.1:45421_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n3] o.a.s.c.SolrCore [[.system_shard1_replica_n3] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.SystemLogListenerTest_B11D509A9403386F-001/tempDir-001/node2/.system_shard1_replica_n3], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.SystemLogListenerTest_B11D509A9403386F-001/tempDir-001/node2/./.system_shard1_replica_n3/data/]
[junit4] 2> 267894 INFO (qtp1167150554-368) [n:127.0.0.1:48528_solr c:.system s:shard1 r:core_node2 x:.system_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
[junit4] 2> 267895 INFO (qtp1753864094-386) [n:127.0.0.1:51580_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_n5] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
[junit4] 2> 267933 INFO (qtp1753864094-386) [n:127.0.0.1:51580_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_n5] o.a.s.s.IndexSchema [.system_shard1_replica_n5] Schema name=_system collection or core
[junit4] 2> 267954 INFO (qtp1167150554-368) [n:127.0.0.1:48528_solr c:.system s:shard1 r:core_node2 x:.system_shard1_replica_n1] o.a.s.s.IndexSchema [.system_shard1_replica_n1] Schema name=_system collection or core
[junit4] 2> 268012 DEBUG (ScheduledTrigger-150-thread-1) [n:127.0.0.1:51580_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 268480 INFO (qtp1167150554-368) [n:127.0.0.1:48528_solr c:.system s:shard1 r:core_node2 x:.system_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema _system collection or core/1.1 with uniqueid field id
[junit4] 2> 268480 INFO (qtp1167150554-368) [n:127.0.0.1:48528_solr c:.system s:shard1 r:core_node2 x:.system_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore '.system_shard1_replica_n1' using configuration from collection .system, trusted=true
[junit4] 2> 268481 INFO (qtp1167150554-368) [n:127.0.0.1:48528_solr c:.system s:shard1 r:core_node2 x:.system_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_48528.solr.core..system.shard1.replica_n1' (registry 'solr.core..system.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2a2fd3ac
[junit4] 2> 268481 INFO (qtp1167150554-368) [n:127.0.0.1:48528_solr c:.system s:shard1 r:core_node2 x:.system_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 268481 INFO (qtp1167150554-368) [n:127.0.0.1:48528_solr c:.system s:shard1 r:core_node2 x:.system_shard1_replica_n1] o.a.s.c.SolrCore [[.system_shard1_replica_n1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.SystemLogListenerTest_B11D509A9403386F-001/tempDir-001/node1/.system_shard1_replica_n1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.SystemLogListenerTest_B11D509A9403386F-001/tempDir-001/node1/./.system_shard1_replica_n1/data/]
[junit4] 2> 268520 INFO (qtp1753864094-386) [n:127.0.0.1:51580_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_n5] o.a.s.s.IndexSchema Loaded schema _system collection or core/1.1 with uniqueid field id
[junit4] 2> 268520 INFO (qtp1753864094-386) [n:127.0.0.1:51580_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_n5] o.a.s.c.CoreContainer Creating SolrCore '.system_shard1_replica_n5' using configuration from collection .system, trusted=true
[junit4] 2> 268521 INFO (qtp1753864094-386) [n:127.0.0.1:51580_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_n5] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_51580.solr.core..system.shard1.replica_n5' (registry 'solr.core..system.shard1.replica_n5') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2a2fd3ac
[junit4] 2> 268521 INFO (qtp1753864094-386) [n:127.0.0.1:51580_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_n5] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 268521 INFO (qtp1753864094-386) [n:127.0.0.1:51580_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_n5] o.a.s.c.SolrCore [[.system_shard1_replica_n5] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.SystemLogListenerTest_B11D509A9403386F-001/tempDir-001/node3/.system_shard1_replica_n5], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.SystemLogListenerTest_B11D509A9403386F-001/tempDir-001/node3/./.system_shard1_replica_n5/data/]
[junit4] 2> 269827 DEBUG (ScheduledTrigger-150-thread-1) [n:127.0.0.1:51580_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 270836 DEBUG (ScheduledTrigger-150-thread-1) [n:127.0.0.1:51580_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 270919 INFO (qtp342458738-375) [n:127.0.0.1:45421_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n3] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 270920 INFO (qtp342458738-375) [n:127.0.0.1:45421_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n3] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 270921 INFO (qtp342458738-375) [n:127.0.0.1:45421_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n3] o.a.s.u.CommitTracker Hard AutoCommit: if 1 uncommited docs
[junit4] 2> 270921 INFO (qtp342458738-375) [n:127.0.0.1:45421_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n3] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 270941 INFO (qtp342458738-375) [n:127.0.0.1:45421_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n3] o.a.s.s.SolrIndexSearcher Opening [Searcher@6cd14c5e[.system_shard1_replica_n3] main]
[junit4] 2> 270959 INFO (qtp342458738-375) [n:127.0.0.1:45421_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n3] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/.system
[junit4] 2> 270976 INFO (qtp342458738-375) [n:127.0.0.1:45421_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n3] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/.system
[junit4] 2> 270980 INFO (qtp1753864094-386) [n:127.0.0.1:51580_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_n5] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 270980 INFO (qtp1753864094-386) [n:127.0.0.1:51580_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_n5] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 270982 INFO (qtp1753864094-386) [n:127.0.0.1:51580_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_n5] o.a.s.u.CommitTracker Hard AutoCommit: if 1 uncommited docs
[junit4] 2> 270982 INFO (qtp1753864094-386) [n:127.0.0.1:51580_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_n5] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 270999 INFO (qtp1753864094-386) [n:127.0.0.1:51580_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_n5] o.a.s.s.SolrIndexSearcher Opening [Searcher@75db7079[.system_shard1_replica_n5] main]
[junit4] 2> 271000 INFO (qtp1753864094-386) [n:127.0.0.1:51580_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_n5] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/.system
[junit4] 2> 271000 INFO (qtp1753864094-386) [n:127.0.0.1:51580_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_n5] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/.system
[junit4] 2> 271001 INFO (qtp1167150554-368) [n:127.0.0.1:48528_solr c:.system s:shard1 r:core_node2 x:.system_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 271001 INFO (qtp1167150554-368) [n:127.0.0.1:48528_solr c:.system s:shard1 r:core_node2 x:.system_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 270976 INFO (qtp342458738-375) [n:127.0.0.1:45421_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n3] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/.system/managed-schema
[junit4] 2> 271000 INFO (qtp1753864094-386) [n:127.0.0.1:51580_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_n5] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/.system/managed-schema
[junit4] 2> 271156 INFO (qtp342458738-375) [n:127.0.0.1:45421_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n3] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
[junit4] 2> 271157 INFO (qtp342458738-375) [n:127.0.0.1:45421_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n3] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 271164 INFO (qtp1753864094-386) [n:127.0.0.1:51580_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_n5] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
[junit4] 2> 271283 INFO (qtp1167150554-368) [n:127.0.0.1:48528_solr c:.system s:shard1 r:core_node2 x:.system_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: if 1 uncommited docs
[junit4] 2> 271283 INFO (qtp1167150554-368) [n:127.0.0.1:48528_solr c:.system s:shard1 r:core_node2 x:.system_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 271284 INFO (qtp1753864094-386) [n:127.0.0.1:51580_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_n5] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 271285 INFO (qtp1753864094-386) [n:127.0.0.1:51580_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_n5] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1587551808002195456
[junit4] 2> 271288 INFO (qtp342458738-375) [n:127.0.0.1:45421_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n3] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1587551808005341184
[junit4] 2> 271302 INFO (searcherExecutor-161-thread-1-processing-n:127.0.0.1:51580_solr x:.system_shard1_replica_n5 s:shard1 c:.system r:core_node6) [n:127.0.0.1:51580_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_n5] o.a.s.c.SolrCore [.system_shard1_replica_n5] Registered new searcher Searcher@75db7079[.system_shard1_replica_n5] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 271303 INFO (searcherExecutor-159-thread-1-processing-n:127.0.0.1:45421_solr x:.system_shard1_replica_n3 s:shard1 c:.system r:core_node4) [n:127.0.0.1:45421_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n3] o.a.s.c.SolrCore [.system_shard1_replica_n3] Registered new searcher Searcher@6cd14c5e[.system_shard1_replica_n3] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 271327 INFO (qtp342458738-375) [n:127.0.0.1:45421_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n3] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard1: total=3 found=1 timeoutin=9999ms
[junit4] 2> 271352 INFO (qtp1167150554-368) [n:127.0.0.1:48528_solr c:.system s:shard1 r:core_node2 x:.system_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@5ec58dec[.system_shard1_replica_n1] main]
[junit4] 2> 271444 INFO (zkCallback-138-thread-2-processing-n:127.0.0.1:45421_solr) [n:127.0.0.1:45421_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
[junit4] 2> 271444 INFO (zkCallback-138-thread-1-processing-n:127.0.0.1:45421_solr) [n:127.0.0.1:45421_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
[junit4] 2> 271445 INFO (qtp1167150554-368) [n:127.0.0.1:48528_solr c:.system s:shard1 r:core_node2 x:.system_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/.system
[junit4] 2> 271445 INFO (zkCallback-140-thread-3-processing-n:127.0.0.1:48528_solr) [n:127.0.0.1:48528_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
[junit4] 2> 271445 INFO (zkCallback-140-thread-2-processing-n:127.0.0.1:48528_solr) [n:127.0.0.1:48528_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
[junit4] 2> 271445 INFO (zkCallback-136-thread-2-processing-n:127.0.0.1:51580_solr) [n:127.0.0.1:51580_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
[junit4] 2> 271445 INFO (zkCallback-136-thread-3-processing-n:127.0.0.1:51580_solr) [n:127.0.0.1:51580_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
[junit4] 2> 271446 INFO (qtp1167150554-368) [n:127.0.0.1:48528_solr c:.system s:shard1 r:core_node2 x:.system_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/.system
[junit4] 2> 271446 INFO (qtp1167150554-368) [n:127.0.0.1:48528_solr c:.system s:shard1 r:core_node2 x:.system_shard1_replica_n1] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/.system/managed-schema
[junit4] 2> 271465 INFO (qtp1167150554-368) [n:127.0.0.1:48528_solr c:.system s:shard1 r:core_node2 x:.system_shard1_replica_n1] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
[junit4] 2> 271465 INFO (qtp1167150554-368) [n:127.0.0.1:48528_solr c:.system s:shard1 r:core_node2 x:.system_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 271466 INFO (qtp1167150554-368) [n:127.0.0.1:48528_solr c:.system s:shard1 r:core_node2 x:.system_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1587551808191987712
[junit4] 2> 271473 INFO (searcherExecutor-160-thread-1-processing-n:127.0.0.1:48528_solr x:.system_shard1_replica_n1 s:shard1 c:.system r:core_node2) [n:127.0.0.1:48528_solr c:.system s:shard1 r:core_node2 x:.system_shard1_replica_n1] o.a.s.c.SolrCore [.system_shard1_replica_n1] Registered new searcher Searcher@5ec58dec[.system_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 271828 INFO (qtp342458738-375) [n:127.0.0.1:45421_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n3] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 271829 INFO (qtp342458738-375) [n:127.0.0.1:45421_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n3] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 271829 INFO (qtp342458738-375) [n:127.0.0.1:45421_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n3] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:45421/solr/.system_shard1_replica_n3/
[junit4] 2> 271829 INFO (qtp342458738-375) [n:127.0.0.1:45421_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n3] o.a.s.u.PeerSync PeerSync: core=.system_shard1_replica_n3 url=http://127.0.0.1:45421/solr START replicas=[http://127.0.0.1:48528/solr/.system_shard1_replica_n1/, http://127.0.0.1:51580/solr/.system_shard1_replica_n5/] nUpdates=100
[junit4] 2> 271965 INFO (qtp1167150554-366) [n:127.0.0.1:48528_solr c:.system s:shard1 r:core_node2 x:.system_shard1_replica_n1] o.a.s.c.S.Request [.system_shard1_replica_n1] webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=39
[junit4] 2> 272224 INFO (qtp342458738-375) [n:127.0.0.1:45421_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n3] o.a.s.u.PeerSync PeerSync: core=.system_shard1_replica_n3 url=http://127.0.0.1:45421/solr DONE. We have no versions. sync failed.
[junit4] 2> 272437 DEBUG (ScheduledTrigger-150-thread-1) [n:127.0.0.1:51580_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 272477 INFO (qtp1753864094-383) [n:127.0.0.1:51580_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_n5] o.a.s.c.S.Request [.system_shard1_replica_n5] webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=38
[junit4] 2> 272477 INFO (qtp342458738-375) [n:127.0.0.1:45421_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n3] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate
[junit4] 2> 272478 INFO (qtp342458738-375) [n:127.0.0.1:45421_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n3] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
[junit4] 2> 272478 INFO (qtp342458738-375) [n:127.0.0.1:45421_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n3] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 272530 INFO (qtp342458738-375) [n:127.0.0.1:45421_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n3] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:45421/solr/.system_shard1_replica_n3/ shard1
[junit4] 2> 273448 DEBUG (ScheduledTrigger-150-thread-1) [n:127.0.0.1:51580_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 273737 INFO (zkCallback-140-thread-2-processing-n:127.0.0.1:48528_solr) [n:127.0.0.1:48528_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
[junit4] 2> 273737 INFO (zkCallback-140-thread-3-processing-n:127.0.0.1:48528_solr) [n:127.0.0.1:48528_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
[junit4] 2> 273738 INFO (zkCallback-136-thread-1-processing-n:127.0.0.1:51580_solr) [n:127.0.0.1:51580_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
[junit4] 2> 273744 INFO (zkCallback-136-thread-3-processing-n:127.0.0.1:51580_solr) [n:127.0.0.1:51580_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
[junit4] 2> 274456 DEBUG (ScheduledTrigger-150-thread-1) [n:127.0.0.1:51580_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 274491 INFO (zkCallback-138-thread-2-processing-n:127.0.0.1:45421_solr) [n:127.0.0.1:45421_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
[junit4] 2> 274491 INFO (zkCallback-138-thread-1-processing-n:127.0.0.1:45421_solr) [n:127.0.0.1:45421_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
[junit4] 2> 274501 INFO (qtp1753864094-386) [n:127.0.0.1:51580_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_n5] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node6&collection.configName=.system&newCollection=true&name=.system_shard1_replica_n5&action=CREATE&numShards=1&collection=.system&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=7878
[junit4] 2> 274502 INFO (qtp1167150554-368) [n:127.0.0.1:48528_solr c:.system s:shard1 r:core_node2 x:.system_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node2&collection.configName=.system&newCollection=true&name=.system_shard1_replica_n1&action=CREATE&numShards=1&collection=.system&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=8245
[junit4] 2> 274548 INFO (qtp342458738-375) [n:127.0.0.1:45421_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n3] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 274556 INFO (qtp342458738-375) [n:127.0.0.1:45421_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node4&collection.configName=.system&newCollection=true&name=.system_shard1_replica_n3&action=CREATE&numShards=1&collection=.system&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=8756
[junit4] 2> 274617 INFO (qtp342458738-377) [n:127.0.0.1:45421_solr ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
[junit4] 2> 274677 INFO (zkCallback-138-thread-2-processing-n:127.0.0.1:45421_solr) [n:127.0.0.1:45421_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
[junit4] 2> 274677 INFO (zkCallback-138-thread-1-processing-n:127.0.0.1:45421_solr) [n:127.0.0.1:45421_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
[junit4] 2> 274677 INFO (zkCallback-136-thread-3-processing-n:127.0.0.1:51580_solr) [n:127.0.0.1:51580_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
[junit4] 2> 274688 INFO (zkCallback-136-thread-1-processing-n:127.0.0.1:51580_solr) [n:127.0.0.1:51580_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
[junit4] 2> 274688 INFO (zkCallback-140-thread-2-processing-n:127.0.0.1:48528_solr) [n:127.0.0.1:48528_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
[junit4] 2> 274688 INFO (zkCallback-140-thread-3-processing-n:127.0.0.1:48528_solr) [n:127.0.0.1:48528_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
[junit4] 2> 275464 DEBUG (ScheduledTrigger-150-thread-1) [n:127.0.0.1:51580_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 275960 INFO (qtp342458738-377) [n:127.0.0.1:45421_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=3&name=.system&nrtReplicas=3&action=CREATE&numShards=1&wt=javabin&version=2} status=0 QTime=10701
[junit4] 2> 276206 INFO (TEST-SystemLogListenerTest.test-seed#[B11D509A9403386F]) [ ] o.a.s.SolrTestCaseJ4 ###Starting test
[junit4] 2> 276496 DEBUG (ScheduledTrigger-150-thread-1) [n:127.0.0.1:51580_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 276534 INFO (OverseerCollectionConfigSetProcessor-99221987005169671-127.0.0.1:51580_solr-n_0000000000) [n:127.0.0.1:51580_solr ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 276709 INFO (qtp342458738-380) [n:127.0.0.1:45421_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CORE.coreName&wt=javabin&version=2&group=solr.core} status=0 QTime=175
[junit4] 2> 277242 INFO (qtp1753864094-388) [n:127.0.0.1:51580_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CORE.coreName&wt=javabin&version=2&group=solr.core} status=0 QTime=1
[junit4] 2> 277271 INFO (qtp1167150554-371) [n:127.0.0.1:48528_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CORE.coreName&wt=javabin&version=2&group=solr.core} status=0 QTime=1
[junit4] 2> 277289 DEBUG (qtp1167150554-370) [n:127.0.0.1:48528_solr ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 277290 DEBUG (zkCallback-136-thread-1-processing-n:127.0.0.1:51580_solr) [n:127.0.0.1:51580_solr ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 2
[junit4] 2> 277291 DEBUG (zkCallback-136-thread-1-processing-n:127.0.0.1:51580_solr) [n:127.0.0.1:51580_solr ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:51580_solr, 127.0.0.1:48528_solr, 127.0.0.1:45421_solr]
[junit4] 2> 277291 DEBUG (zkCallback-136-thread-1-processing-n:127.0.0.1:51580_solr) [n:127.0.0.1:51580_solr ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:51580_solr, 127.0.0.1:48528_solr, 127.0.0.1:45421_solr]
[junit4] 2> 277291 DEBUG (OverseerAutoScalingTriggerThread-99221987005169671-127.0.0.1:51580_solr-n_0000000000) [n:127.0.0.1:51580_solr ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
[junit4] 2> 277293 DEBUG (OverseerAutoScalingTriggerThread-99221987005169671-127.0.0.1:51580_solr-n_0000000000) [n:127.0.0.1:51580_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 277293 DEBUG (ScheduledTrigger-150-thread-2) [n:127.0.0.1:51580_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 3
[junit4] 2> 277293 DEBUG (OverseerAutoScalingTriggerThread-99221987005169671-127.0.0.1:51580_solr-n_0000000000) [n:127.0.0.1:51580_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2
[junit4] 2> 277308 INFO (qtp1167150554-370) [n:127.0.0.1:48528_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/cluster/autoscaling params={wt=javabin&version=2} status=0 QTime=1096
[junit4] 2> 277312 INFO (qtp342458738-381) [n:127.0.0.1:45421_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CORE.coreName&wt=javabin&version=2&group=solr.core} status=0 QTime=1
[junit4] 2> 277315 INFO (qtp1753864094-388) [n:127.0.0.1:51580_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CORE.coreName&wt=javabin&version=2&group=solr.core} status=0 QTime=1
[junit4] 2> 277361 INFO (qtp1167150554-369) [n:127.0.0.1:48528_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CORE.coreName&wt=javabin&version=2&group=solr.core} status=0 QTime=0
[junit4] 2> 277362 DEBUG (qtp342458738-378) [n:127.0.0.1:45421_solr ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 277381 DEBUG (zkCallback-136-thread-1-processing-n:127.0.0.1:51580_solr) [n:127.0.0.1:51580_solr ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 3
[junit4] 2> 277381 DEBUG (zkCallback-136-thread-1-processing-n:127.0.0.1:51580_solr) [n:127.0.0.1:51580_solr ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:51580_solr, 127.0.0.1:48528_solr, 127.0.0.1:45421_solr]
[junit4] 2> 277381 DEBUG (zkCallback-136-thread-1-processing-n:127.0.0.1:51580_solr) [n:127.0.0.1:51580_solr ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:51580_solr, 127.0.0.1:48528_solr, 127.0.0.1:45421_solr]
[junit4] 2> 277381 DEBUG (OverseerAutoScalingTriggerThread-99221987005169671-127.0.0.1:51580_solr-n_0000000000) [n:127.0.0.1:51580_solr ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
[junit4] 2> 277382 DEBUG (OverseerAutoScalingTriggerThread-99221987005169671-127.0.0.1:51580_solr-n_0000000000) [n:127.0.0.1:51580_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 277382 DEBUG (OverseerAutoScalingTriggerThread-99221987005169671-127.0.0.1:51580_solr-n_0000000000) [n:127.0.0.1:51580_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
[junit4] 2> 277382 INFO (qtp342458738-378) [n:127.0.0.1:45421_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/autoscaling params={wt=javabin&version=2} status=0 QTime=72
[junit4] 2> 277383 INFO (qtp342458738-374) [n:127.0.0.1:45421_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=2&collection.configName=conf&maxShardsPerNode=3&name=test&nrtReplicas=2&action=CREATE&numShards=3&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 277415 INFO (OverseerThreadFactory-152-thread-2-processing-n:127.0.0.1:51580_solr) [n:127.0.0.1:51580_solr ] o.a.s.c.CreateCollectionCmd Create collection test
[junit4] 2> 277582 DEBUG (ScheduledTrigger-150-thread-4) [n:127.0.0.1:51580_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 277587 INFO (OverseerStateUpdate-99221987005169671-127.0.0.1:51580_solr-n_0000000000) [n:127.0.0.1:51580_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"test",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"test_shard1_replica_n1",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:48528/solr",
[junit4] 2> "type":"NRT",
[junit4] 2> "waitForFinalState":"false"}
[junit4] 2> 277590 INFO (OverseerStateUpdate-99221987005169671-127.0.0.1:51580_solr-n_0000000000) [n:127.0.0.1:51580_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"test",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"test_shard1_replica_n2",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:45421/solr",
[junit4] 2> "type":"NRT",
[junit4] 2> "waitForFinalState":"false"}
[junit4] 2> 277593 INFO (OverseerStateUpdate-99221987005169671-127.0.0.1:51580_solr-n_0000000000) [n:127.0.0.1:51580_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"test",
[junit4] 2> "shard":"shard2",
[junit4] 2> "core":"test_shard2_replica_n4",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:51580/solr",
[junit4] 2> "type":"NRT",
[junit4] 2> "waitForFinalState":"false"}
[junit4] 2> 277596 INFO (OverseerStateUpdate-99221987005169671-127.0.0.1:51580_solr-n_0000000000) [n:127.0.0.1:51580_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"test",
[junit4] 2> "shard":"shard2",
[junit4] 2> "core":"test_shard2_replica_n6",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:48528/solr",
[junit4] 2> "type":"NRT",
[junit4] 2> "waitForFinalState":"false"}
[junit4] 2> 277621 INFO (OverseerStateUpdate-99221987005169671-127.0.0.1:51580_solr-n_0000000000) [n:127.0.0.1:51580_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"test",
[junit4] 2> "shard":"shard3",
[junit4] 2> "core":"test_shard3_replica_n8",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:45421/solr",
[junit4] 2> "type":"NRT",
[junit4] 2> "waitForFinalState":"false"}
[junit4] 2> 277624 INFO (OverseerStateUpdate-99221987005169671-127.0.0.1:51580_solr-n_0000000000) [n:127.0.0.1:51580_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"test",
[junit4] 2> "shard":"shard3",
[junit4] 2> "core":"test_shard3_replica_n10",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:51580/solr",
[junit4] 2> "type":"NRT",
[junit4] 2> "waitForFinalState":"false"}
[junit4] 2> 277835 INFO (qtp1753864094-387) [n:127.0.0.1:51580_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node7&collection.configName=conf&newCollection=true&name=test_shard2_replica_n4&action=CREATE&numShards=3&collection=test&shard=shard2&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 277881 INFO (qtp1167150554-371) [n:127.0.0.1:48528_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node9&collection.configName=conf&newCollection=true&name=test_shard2_replica_n6&action=CREATE&numShards=3&collection=test&shard=shard2&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 277964 INFO (qtp342458738-375) [n:127.0.0.1:45421_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node11&collection.configName=conf&newCollection=true&name=test_shard3_replica_n8&action=CREATE&numShards=3&collection=test&shard=shard3&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 277997 INFO (qtp1167150554-369) [n:127.0.0.1:48528_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=test_shard1_replica_n1&action=CREATE&numShards=3&collection=test&shard=shard1&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 278001 INFO (qtp342458738-379) [n:127.0.0.1:45421_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node5&collection.configName=conf&newCollection=true&name=test_shard1_replica_n2&action=CREATE&numShards=3&collection=test&shard=shard1&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 278003 INFO (qtp1753864094-384) [n:127.0.0.1:51580_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node12&collection.configName=conf&newCollection=true&name=test_shard3_replica_n10&action=CREATE&numShards=3&collection=test&shard=shard3&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 278240 INFO (zkCallback-140-thread-2-processing-n:127.0.0.1:48528_solr) [n:127.0.0.1:48528_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test/state.json] for collection [test] has occurred - updating... (live nodes size: [3])
[junit4] 2> 278240 INFO (zkCallback-138-thread-2-processing-n:127.0.0.1:45421_solr) [n:127.0.0.1:45421_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test/state.json] for collection [test] has occurred - updating... (live nodes size: [3])
[junit4] 2> 278240 INFO (zkCallback-140-thread-3-processing-n:127.0.0.1:48528_solr) [n:127.0.0.1:48528_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test/state.json] for collection [test] has occurred - updating... (live nodes size: [3])
[junit4] 2> 278241 INFO (zkCallback-136-thread-2-processing-n:127.0.0.1:51580_solr) [n:127.0.0.1:51580_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test/state.json] for collection [test] has occurred - updating... (live nodes size: [3])
[junit4] 2> 278241 INFO (zkCallback-136-thread-1-processing-n:127.0.0.1:51580_solr) [n:127.0.0.1:51580_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test/state.json] for collection [test] has occurred - updating... (live nodes size: [3])
[junit4] 2> 278337 DEBUG (ScheduledTrigger-150-thread-1) [n:127.0.0.1:51580_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 3
[junit4] 2> 278353 INFO (zkCallback-138-thread-2-processing-n:127.0.0.1:45421_solr) [n:127.0.0.1:45421_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test/state.json] for collection [test] has occurred - updating... (live nodes size: [3])
[junit4] 2> 278353 INFO (zkCallback-138-thread-1-processing-n:127.0.0.1:45421_solr) [n:127.0.0.1:45421_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test/state.json] for collection [test] has occurred - updating... (live nodes size: [3])
[junit4] 2> 278353 INFO (zkCallback-140-thread-3-processing-n:127.0.0.1:48528_solr) [n:127.0.0.1:48528_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test/state.json] for collection [test] has occurred - updating... (live nodes size: [3])
[junit4] 2> 278353 INFO (zkCallback-140-thread-2-processing-n:127.0.0.1:48528_solr) [n:127.0.0.1:48528_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test/state.json] for collection [test] has occurred - updating... (live nodes size: [3])
[junit4] 2> 278353 INFO (zkCallback-136-thread-1-processing-n:127.0.0.1:51580_solr) [n:127.0.0.1:51580_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test/state.json] for collection [test] has occurred - updating... (live nodes size: [3])
[junit4] 2> 278353 INFO (zkCallback-136-thread-2-processing-n:127.0.0.1:51580_solr) [n:127.0.0.1:51580_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test/state.json] for collection [test] has occurred - updating... (live nodes size: [3])
[junit4] 2> 278672 DEBUG (ScheduledTrigger-150-thread-3) [n:127.0.0.1:51580_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 278950 INFO (qtp1167150554-371) [n:127.0.0.1:48528_solr c:test s:shard2 r:core_node9 x:test_shard2_replica_n6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
[junit4] 2> 279015 INFO (qtp1753864094-387) [n:127.0.0.1:51580_solr c:test s:shard2 r:core_node7 x:test_shard2_replica_n4] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
[junit4] 2> 279045 INFO (qtp1753864094-384) [n:127.0.0.1:51580_solr c:test s:shard3 r:core_node12 x:test_shard3_replica_n10] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
[junit4] 2> 279047 INFO (qtp1167150554-369) [n:127.0.0.1:48528_solr c:test s:shard1 r:core_node3 x:test_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
[junit4] 2> 279230 INFO (qtp1167150554-371) [n:127.0.0.1:48528_solr c:test s:shard2 r:core_node9 x:test_shard2_replica_n6] o.a.s.s.IndexSchema [test_shard2_replica_n6] Schema name=minimal
[junit4] 2> 279248 INFO (qtp1167150554-371) [n:127.0.0.1:48528_solr c:test s:shard2 r:core_node9 x:test_shard2_replica_n6] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 279248 INFO (qtp1167150554-371) [n:127.0.0.1:48528_solr c:test s:shard2 r:core_node9 x:test_shard2_replica_n6] o.a.s.c.CoreContainer Creating SolrCore 'test_shard2_replica_n6' using configuration from collection test, trusted=true
[junit4] 2> 279248 INFO (qtp1167150554-371) [n:127.0.0.1:48528_solr c:test s:shard2 r:core_node9 x:test_shard2_replica_n6] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_48528.solr.core.test.shard2.replica_n6' (registry 'solr.core.test.shard2.replica_n6') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2a2fd3ac
[junit4] 2> 279248 INFO (qtp1167150554-371) [n:127.0.0.1:48528_solr c:test s:shard2 r:core_node9 x:test_shard2_replica_n6] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 279249 INFO (qtp1167150554-371) [n:127.0.0.1:48528_solr c:test s:shard2 r:core_node9 x:test_shard2_replica_n6] o.a.s.c.SolrCore [[test_shard2_replica_n6] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.SystemLogListenerTest_B11D509A9403386F-001/tempDir-001/node1/test_shard2_replica_n6], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.SystemLogListenerTest_B11D509A9403386F-001/tempDir-001/node1/./test_shard2_replica_n6/data/]
[junit4] 2> 279283 INFO (qtp1167150554-369) [n:127.0.0.1:48528_solr c:test s:shard1 r:core_node3 x:test_shard1_replica_n1] o.a.s.s.IndexSchema [test_shard1_replica_n1] Schema name=minimal
[junit4] 2> 279285 INFO (qtp1167150554-369) [n:127.0.0.1:48528_solr c:test s:shard1 r:core_node3 x:test_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 279285 INFO (qtp1167150554-369) [n:127.0.0.1:48528_solr c:test s:shard1 r:core_node3 x:test_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'test_shard1_replica_n1' using configuration from collection test, trusted=true
[junit4] 2> 279286 INFO (qtp1167150554-369) [n:127.0.0.1:48528_solr c:test s:shard1 r:core_node3 x:test_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_48528.solr.core.test.shard1.replica_n1' (registry 'solr.core.test.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2a2fd3ac
[junit4] 2> 279286 INFO (qtp1167150554-369) [n:127.0.0.1:48528_solr c:test s:shard1 r:core_node3 x:test_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 279286 INFO (qtp1167150554-369) [n:127.0.0.1:48528_solr c:test s:shard1 r:core_node3 x:test_shard1_replica_n1] o.a.s.c.SolrCore [[test_shard1_replica_n1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.SystemLogListenerTest_B11D509A9403386F-001/tempDir-001/node1/test_shard1_replica_n1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.SystemLogListenerTest_B11D509A9403386F-001/tempDir-001/node1/./test_shard1_replica_n1/data/]
[junit4] 2> 279337 DEBUG (ScheduledTrigger-150-thread-2) [n:127.0.0.1:51580_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 3
[junit4] 2> 279339 INFO (qtp1753864094-387) [n:127.0.0.1:51580_solr c:test s:shard2 r:core_node7 x:test_shard2_replica_n4] o.a.s.s.IndexSchema [test_shard2_replica_n4] Schema name=minimal
[junit4] 2> 279349 INFO (qtp1753864094-384) [n:127.0.0.1:51580_solr c:test s:shard3 r:core_node12 x:test_shard3_replica_n10] o.a.s.s.IndexSchema [test_shard3_replica_n10] Schema name=minimal
[junit4] 2> 279351 INFO (qtp1753864094-384) [n:127.0.0.1:51580_solr c:test s:shard3 r:core_node12 x:test_shard3_replica_n10] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 279351 INFO (qtp1753864094-384) [n:127.0.0.1:51580_solr c:test s:shard3 r:core_node12 x:test_shard3_replica_n10] o.a.s.c.CoreContainer Creating SolrCore 'test_shard3_replica_n10' using configuration from collection test, trusted=true
[junit4] 2> 279352 INFO (qtp1753864094-384) [n:127.0.0.1:51580_solr c:test s:shard3 r:core_node12 x:test_shard3_replica_n10] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_51580.solr.core.test.shard3.replica_n10' (registry 'solr.core.test.shard3.replica_n10') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2a2fd3ac
[junit4] 2> 279352 INFO (qtp1753864094-384) [n:127.0.0.1:51580_solr c:test s:shard3 r:core_node12 x:test_shard3_replica_n10] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 279352 INFO (qtp1753864094-384) [n:127.0.0.1:51580_solr c:test s:shard3 r:core_node12 x:test_shard3_replica_n10] o.a.s.c.SolrCore [[test_shard3_replica_n10] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.SystemLogListenerTest_B11D509A9403386F-001/tempDir-001/node3/test_shard3_replica_n10], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.SystemLogListenerTest_B11D509A9403386F-001/tempDir-001/node3/./test_shard3_replica_n10/data/]
[junit4] 2> 279361 INFO (qtp1753864094-387) [n:127.0.0.1:51580_solr c:test s:shard2 r:core_node7 x:test_shard2_replica_n4] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 279361 INFO (qtp1753864094-387) [n:127.0.0.1:51580_solr c:test s:shard2 r:core_node7 x:test_shard2_replica_n4] o.a.s.c.CoreContainer Creating SolrCore 'test_shard2_replica_n4' using configuration from collection test, trusted=true
[junit4] 2> 279424 INFO (qtp1753864094-387) [n:127.0.0.1:51580_solr c:test s:shard2 r:core_node7 x:test_shard2_replica_n4] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_51580.solr.core.test.shard2.replica_n4' (registry 'solr.core.test.shard2.replica_n4') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2a2fd3ac
[junit4] 2> 279425 INFO (qtp1753864094-387) [n:127.0.0.1:51580_solr c:test s:shard2 r:core_node7 x:test_shard2_replica_n4] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 279425 INFO (qtp1753864094-387) [n:127.0.0.1:51580_solr c:test s:shard2 r:core_node7 x:test_shard2_replica_n4] o.a.s.c.SolrCore [[test_shard2_replica_n4] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.SystemLogListenerTest_B11D509A9403386F-001/tempDir-001/node3/test_shard2_replica_n4], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.SystemLogListenerTest_B11D509A9403386F-001/tempDir-001/node3/./test_shard2_replica_n4/data/]
[junit4] 2> 279442 INFO (qtp342458738-375) [n:127.0.0.1:45421_solr c:test s:shard3 r:core_node11 x:test_shard3_replica_n8] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
[junit4] 2> 279464 INFO (qtp342458738-379) [n:127.0.0.1:45421_solr c:test s:shard1 r:core_node5 x:test_shard1_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
[junit4] 2> 279497 INFO (qtp342458738-379) [n:127.0.0.1:45421_solr c:test s:shard1 r:core_node5 x:test_shard1_replica_n2] o.a.s.s.IndexSchema [test_shard1_replica_n2] Schema name=minimal
[junit4] 2> 279500 INFO (qtp342458738-379) [n:127.0.0.1:45421_solr c:test s:shard1 r:core_node5 x:test_shard1_replica_n2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 279500 INFO (qtp342458738-379) [n:127.0.0.1:45421_solr c:test s:shard1 r:core_node5 x:test_shard1_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'test_shard1_replica_n2' using configuration from collection test, trusted=true
[junit4] 2> 279533 INFO (qtp342458738-375) [n:127.0.0.1:45421_solr c:test s:shard3 r:core_node11 x:test_shard3_replica_n8] o.a.s.s.IndexSchema [test_shard3_replica_n8] Schema name=minimal
[junit4] 2> 279542 INFO (qtp342458738-375) [n:127.0.0.1:45421_solr c:test s:shard3 r:core_node11 x:test_shard3_replica_n8] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 279542 INFO (qtp342458738-375) [n:127.0.0.1:45421_solr c:test s:shard3 r:core_node11 x:test_shard3_replica_n8] o.a.s.c.CoreContainer Creating SolrCore 'test_shard3_replica_n8' using configuration from collection test, trusted=true
[junit4] 2> 279542 INFO (qtp342458738-375) [n:127.0.0.1:45421_solr c:test s:shard3 r:core_node11 x:test_shard3_replica_n8] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_45421.solr.core.test.shard3.replica_n8' (registry 'solr.core.test.shard3.replica_n8') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2a2fd3ac
[junit4] 2> 279542 INFO (qtp342458738-375) [n:127.0.0.1:45421_solr c:test s:shard3 r:core_node11 x:test_shard3_replica_n8] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 279542 INFO (qtp342458738-375) [n:127.0.0.1:45421_solr c:test s:shard3 r:core_node11 x:test_shard3_replica_n8] o.a.s.c.SolrCore [[test_shard3_replica_n8] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.SystemLogListenerTest_B11D509A9403386F-001/tempDir-001/node2/test_shard3_replica_n8], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.SystemLogListenerTest_B11D509A9403386F-001/tempDir-001/node2/./test_shard3_replica_n8/data/]
[junit4] 2> 279652 INFO (qtp342458738-379) [n:127.0.0.1:45421_solr c:test s:shard1 r:core_node5 x:test_shard1_replica_n2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_45421.solr.core.test.shard1.replica_n2' (registry 'solr.core.test.shard1.replica_n2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2a2fd3ac
[junit4] 2> 279653 INFO (qtp342458738-379) [n:127.0.0.1:45421_solr c:test s:shard1 r:core_node5 x:test_shard1_replica_n2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 279653 INFO (qtp342458738-379) [n:127.0.0.1:45421_solr c:test s:shard1 r:core_node5 x:test_shard1_replica_n2] o.a.s.c.SolrCore [[test_shard1_replica_n2] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.SystemLogListenerTest_B11D509A9403386F-001/tempDir-001/node2/test_shard1_replica_n2], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.SystemLogListenerTest_B11D509A9403386F-001/tempDir-001/node2/./test_shard1_replica_n2/data/]
[junit4] 2> 279673 DEBUG (ScheduledTrigger-150-thread-2) [n:127.0.0.1:51580_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 280620 DEBUG (ScheduledTrigger-150-thread-2) [n:127.0.0.1:51580_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 3
[junit4] 2> 281392 DEBUG (ScheduledTrigger-150-thread-3) [n:127.0.0.1:51580_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 281520 INFO (qtp1167150554-371) [n:127.0.0.1:48528_solr c:test s:shard2 r:core_node9 x:test_shard2_replica_n6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 281520 INFO (qtp1167150554-371) [n:127.0.0.1:48528_solr c:test s:shard2 r:core_node9 x:test_shard2_replica_n6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 281533 INFO (qtp1167150554-371) [n:127.0.0.1:48528_solr c:test s:shard2 r:core_node9 x:test_shard2_replica_n6] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 281533 INFO (qtp1167150554-371) [n:127.0.0.1:48528_solr c:test s:shard2 r:core_node9 x:test_shard2_replica_n6] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 281556 INFO (qtp1753864094-384) [n:127.0.0.1:51580_solr c:test s:shard3 r:core_node12 x:test_shard3_replica_n10] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 281556 INFO (qtp1753864094-384) [n:127.0.0.1:51580_solr c:test s:shard3 r:core_node12 x:test_shard3_replica_n10] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 281557 INFO (qtp1753864094-384) [n:127.0.0.1:51580_solr c:test s:shard3 r:core_node12 x:test_shard3_replica_n10] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 281557 INFO (qtp1753864094-384) [n:127.0.0.1:51580_solr c:test s:shard3 r:core_node12 x:test_shard3_replica_n10] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 281588 INFO (qtp1167150554-371) [n:127.0.0.1:48528_solr c:test s:shard2 r:core_node9 x:test_shard2_replica_n6] o.a.s.s.SolrIndexSearcher Opening [Searcher@714923b5[test_shard2_replica_n6] main]
[junit4] 2> 281596 INFO (qtp1167150554-371) [n:127.0.0.1:48528_solr c:test s:shard2 r:core_node9 x:test_shard2_replica_n6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
[junit4] 2> 281596 INFO (qtp1167150554-371) [n:127.0.0.1:48528_solr c:test s:shard2 r:core_node9 x:test_shard2_replica_n6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
[junit4] 2> 281597 INFO (qtp1167150554-371) [n:127.0.0.1:48528_solr c:test s:shard2 r:core_node9 x:test_shard2_replica_n6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 281597 INFO (qtp1167150554-371) [n:127.0.0.1:48528_solr c:test s:shard2 r:core_node9 x:test_shard2_replica_n6] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1587551818815111168
[junit4] 2> 281713 INFO (zkCallback-138-thread-2-processing-n:127.0.0.1:45421_solr) [n:127.0.0.1:45421_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test/state.json] for collection [test] has occurred - updating... (live nodes size: [3])
[junit4] 2> 281714 INFO (zkCallback-138-thread-1-processing-n:127.0.0.1:45421_solr) [n:127.0.0.1:45421_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test/state.json] for collection [test] has occurred - updating... (live nodes size: [3])
[junit4] 2> 281714 INFO (zkCallback-136-thread-2-processing-n:127.0.0.1:51580_solr) [n:127.0.0.1:51580_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test/state.json] for collection [test] has occurred - updating... (live nodes size: [3])
[junit4] 2> 281714 INFO (zkCallback-136-thread-1-processing-n:127.0.0.1:51580_solr) [n:127.0.0.1:51580_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test/state.json] for collection [test] has occurred - updating... (live nodes size: [3])
[junit4] 2> 281714 INFO (zkCallback-140-thread-3-processing-n:127.0.0.1:48528_solr) [n:127.0.0.1:48528_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test/state.json] for collection [test] has occurred - updating... (live nodes size: [3])
[junit4] 2> 281714 INFO (zkCallback-140-thread-2-processing-n:127.0.0.1:48528_solr) [n:127.0.0.1:48528_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test/state.json] for collection [test] has occurred - updating... (live nodes size: [3])
[junit4] 2> 281716 INFO (qtp1167150554-371) [n:127.0.0.1:48528_solr c:test s:shard2 r:core_node9 x:test_shard2_replica_n6] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard2: total=2 found=1 timeoutin=9998ms
[junit4] 2> 281716 INFO (qtp1753864094-384) [n:127.0.0.1:51580_solr c:test s:shard3 r:core_node12 x:test_shard3_replica_n10] o.a.s.s.SolrIndexSearcher Opening [Searcher@696ae4f8[test_shard3_replica_n10] main]
[junit4] 2> 281717 DEBUG (ScheduledTrigger-150-thread-2) [n:127.0.0.1:51580_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 3
[junit4] 2> 281767 INFO (searcherExecutor-174-thread-1-processing-n:127.0.0.1:48528_solr x:test_shard2_replica_n6 s:shard2 c:test r:core_node9) [n:127.0.0.1:48528_solr c:test s:shard2 r:core_node9 x:test_shard2_replica_n6] o.a.s.c.SolrCore [test_shard2_replica_n6] Registered new searcher Searcher@714923b5[test_shard2_replica_n6] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 281812 INFO (qtp1753864094-384) [n:127.0.0.1:51580_solr c:test s:shard3 r:core_node12 x:test_shard3_replica_n10] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
[junit4] 2> 281813 INFO (qtp1753864094-384) [n:127.0.0.1:51580_solr c:test s:shard3 r:core_node12 x:test_shard3_replica_n10] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
[junit4] 2> 281813 INFO (qtp1753864094-384) [n:127.0.0.1:51580_solr c:test s:shard3 r:core_node12 x:test_shard3_replica_n10]
[...truncated too long message...]
:
ivy-fail:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/top-level-ivy-settings.xml
resolve:
-check-git-state:
-git-cleanroot:
-copy-git-state:
git-autoclean:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/top-level-ivy-settings.xml
resolve:
common.init:
compile-lucene-core:
-check-git-state:
-git-cleanroot:
-copy-git-state:
git-autoclean:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/top-level-ivy-settings.xml
resolve:
init:
-clover.disable:
-clover.load:
-clover.classpath:
-clover.setup:
clover:
compile-core:
init:
-clover.disable:
-clover.load:
-clover.classpath:
-clover.setup:
clover:
common.compile-core:
common-solr.compile-core:
compile-core:
resolve-groovy:
define-lucene-javadoc-url:
check-lucene-core-javadocs-uptodate:
javadocs-lucene-core:
check-analyzers-common-javadocs-uptodate:
javadocs-analyzers-common:
check-analyzers-icu-javadocs-uptodate:
javadocs-analyzers-icu:
check-analyzers-kuromoji-javadocs-uptodate:
javadocs-analyzers-kuromoji:
check-analyzers-phonetic-javadocs-uptodate:
javadocs-analyzers-phonetic:
check-analyzers-smartcn-javadocs-uptodate:
javadocs-analyzers-smartcn:
check-analyzers-morfologik-javadocs-uptodate:
javadocs-analyzers-morfologik:
check-analyzers-stempel-javadocs-uptodate:
javadocs-analyzers-stempel:
check-analyzers-uima-javadocs-uptodate:
javadocs-analyzers-uima:
check-backward-codecs-javadocs-uptodate:
javadocs-backward-codecs:
check-codecs-javadocs-uptodate:
javadocs-codecs:
check-expressions-javadocs-uptodate:
javadocs-expressions:
check-suggest-javadocs-uptodate:
javadocs-suggest:
check-grouping-javadocs-uptodate:
javadocs-grouping:
check-queries-javadocs-uptodate:
javadocs-queries:
check-queryparser-javadocs-uptodate:
javadocs-queryparser:
check-highlighter-javadocs-uptodate:
javadocs-highlighter:
check-memory-javadocs-uptodate:
javadocs-memory:
check-misc-javadocs-uptodate:
javadocs-misc:
check-spatial-extras-javadocs-uptodate:
javadocs-spatial-extras:
check-join-javadocs-uptodate:
javadocs-join:
check-test-framework-javadocs-uptodate:
javadocs-test-framework:
lucene-javadocs:
check-solr-core-javadocs-uptodate:
javadocs-solr-core:
check-javadocs-uptodate:
javadocs:
[mkdir] Created dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/docs/solr-velocity
download-java8-javadoc-packagelist:
[javadoc] Generating Javadoc
[javadoc] Javadoc execution
[javadoc] Loading source files for package org.apache.solr.response...
[javadoc] Constructing Javadoc information...
[javadoc] Standard Doclet version 1.8.0_144
[javadoc] Building tree for all the packages and classes...
[javadoc] Building index for all the packages and classes...
[javadoc] Building index for all classes...
[javadoc] Generating /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/docs/solr-velocity/help-doc.html...
[javadoc] Note: Custom tags that were not seen: @lucene.internal, @lucene.experimental
[jar] Building jar: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/contrib/solr-velocity/solr-velocity-7.3.0-SNAPSHOT-javadoc.jar
javadocs:
changes-to-html:
[mkdir] Created dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/docs/changes
[delete] Deleting: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/doap.solr.version.dates.csv
[copy] Copying 3 files to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/docs/changes
resolve-markdown:
process-webpages:
[xslt] Processing /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build.xml to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/docs/index.html
[xslt] Loading stylesheet /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/site/index.xsl
[markdown] Copying 1 file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/docs
[markdown] Copying /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/site/SYSTEM_REQUIREMENTS.mdtext to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/docs/SYSTEM_REQUIREMENTS.html
[copy] Copying 7 files to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/docs
documentation:
-documentation-lint:
[jtidy] Checking for broken html (such as invalid tags)...
[delete] Deleting directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/jtidy_tmp
[echo] Checking for broken links...
[exec]
[exec] Crawl/parse...
[exec]
[exec] Verify...
[echo] Checking for malformed docs...
jar-checksums:
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/top-level-ivy-settings.xml
resolve:
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[JENKINS] Lucene-Solr-Tests-7.x - Build # 292 - Still Failing
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-7.x/292/
3 tests failed.
FAILED: org.apache.solr.TestDistributedSearch.test
Error Message:
Expected to find shardAddress in the up shard info: {error=org.apache.solr.client.solrj.SolrServerException: Time allowed to handle this request exceeded,trace=org.apache.solr.client.solrj.SolrServerException: Time allowed to handle this request exceeded at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:460) at org.apache.solr.handler.component.HttpShardHandlerFactory.makeLoadBalancedRequest(HttpShardHandlerFactory.java:273) at org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0(HttpShardHandler.java:175) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) ,time=2}
Stack Trace:
java.lang.AssertionError: Expected to find shardAddress in the up shard info: {error=org.apache.solr.client.solrj.SolrServerException: Time allowed to handle this request exceeded,trace=org.apache.solr.client.solrj.SolrServerException: Time allowed to handle this request exceeded
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:460)
at org.apache.solr.handler.component.HttpShardHandlerFactory.makeLoadBalancedRequest(HttpShardHandlerFactory.java:273)
at org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0(HttpShardHandler.java:175)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
,time=2}
at __randomizedtesting.SeedInfo.seed([8B8823E327AD657E:3DC1C3989510886]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.apache.solr.TestDistributedSearch.comparePartialResponses(TestDistributedSearch.java:1191)
at org.apache.solr.TestDistributedSearch.queryPartialResults(TestDistributedSearch.java:1132)
at org.apache.solr.TestDistributedSearch.test(TestDistributedSearch.java:992)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsRepeatStatement.callStatement(BaseDistributedSearchTestCase.java:1019)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: org.apache.solr.cloud.TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete
Error Message:
Error from server at http://127.0.0.1:36530/solr/testcollection_shard1_replica_n3: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 404 </title> </head> <body> <h2>HTTP ERROR: 404</h2> <p>Problem accessing /solr/testcollection_shard1_replica_n3/update. Reason: <pre> Can not find: /solr/testcollection_shard1_replica_n3/update</pre></p> <hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.20.v20170531</a><hr/> </body> </html>
Stack Trace:
org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error from server at http://127.0.0.1:36530/solr/testcollection_shard1_replica_n3: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 404 </title>
</head>
<body>
<h2>HTTP ERROR: 404</h2>
<p>Problem accessing /solr/testcollection_shard1_replica_n3/update. Reason:
<pre> Can not find: /solr/testcollection_shard1_replica_n3/update</pre></p>
<hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.20.v20170531</a><hr/>
</body>
</html>
at __randomizedtesting.SeedInfo.seed([8B8823E327AD657E:28728D46A0458FDB]:0)
at org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:550)
at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1013)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:946)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:946)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:946)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:946)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:946)
at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
at org.apache.solr.client.solrj.request.UpdateRequest.commit(UpdateRequest.java:233)
at org.apache.solr.cloud.TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete(TestCollectionsAPIViaSolrCloudCluster.java:167)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:36530/solr/testcollection_shard1_replica_n3: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 404 </title>
</head>
<body>
<h2>HTTP ERROR: 404</h2>
<p>Problem accessing /solr/testcollection_shard1_replica_n3/update. Reason:
<pre> Can not find: /solr/testcollection_shard1_replica_n3/update</pre></p>
<hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.20.v20170531</a><hr/>
</body>
</html>
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:607)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
at org.apache.solr.client.solrj.impl.CloudSolrClient.lambda$directUpdate$0(CloudSolrClient.java:527)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more
FAILED: org.apache.solr.cloud.autoscaling.sim.TestDistribStateManager.testGetSetRemoveData
Error Message:
Node watch should have fired!
Stack Trace:
java.lang.AssertionError: Node watch should have fired!
at __randomizedtesting.SeedInfo.seed([8B8823E327AD657E:AD18CCB17323A374]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.autoscaling.sim.TestDistribStateManager.testGetSetRemoveData(TestDistribStateManager.java:256)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
Build Log:
[...truncated 12239 lines...]
[junit4] Suite: org.apache.solr.TestDistributedSearch
[junit4] 2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_8B8823E327AD657E-001/init-core-data-001
[junit4] 2> 638035 WARN (SUITE-TestDistributedSearch-seed#[8B8823E327AD657E]-worker) [ ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=30 numCloses=30
[junit4] 2> 638035 INFO (SUITE-TestDistributedSearch-seed#[8B8823E327AD657E]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
[junit4] 2> 638037 INFO (SUITE-TestDistributedSearch-seed#[8B8823E327AD657E]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-9061)
[junit4] 2> 638037 INFO (SUITE-TestDistributedSearch-seed#[8B8823E327AD657E]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
[junit4] 2> 638037 INFO (SUITE-TestDistributedSearch-seed#[8B8823E327AD657E]-worker) [ ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
[junit4] 2> 638167 INFO (TEST-TestDistributedSearch.test-seed#[8B8823E327AD657E]) [ ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_8B8823E327AD657E-001/tempDir-001/control/cores/collection1
[junit4] 2> 638168 INFO (TEST-TestDistributedSearch.test-seed#[8B8823E327AD657E]) [ ] o.e.j.s.Server jetty-9.3.20.v20170531
[junit4] 2> 638169 INFO (TEST-TestDistributedSearch.test-seed#[8B8823E327AD657E]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@2d1cabd1{/,null,AVAILABLE}
[junit4] 2> 638170 INFO (TEST-TestDistributedSearch.test-seed#[8B8823E327AD657E]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@3e73f7b7{HTTP/1.1,[http/1.1]}{127.0.0.1:45915}
[junit4] 2> 638171 INFO (TEST-TestDistributedSearch.test-seed#[8B8823E327AD657E]) [ ] o.e.j.s.Server Started @641172ms
[junit4] 2> 638171 INFO (TEST-TestDistributedSearch.test-seed#[8B8823E327AD657E]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/, hostPort=45915, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_8B8823E327AD657E-001/tempDir-001/control/cores}
[junit4] 2> 638171 ERROR (TEST-TestDistributedSearch.test-seed#[8B8823E327AD657E]) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 638171 INFO (TEST-TestDistributedSearch.test-seed#[8B8823E327AD657E]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 7.3.0
[junit4] 2> 638171 INFO (TEST-TestDistributedSearch.test-seed#[8B8823E327AD657E]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in standalone mode on port null
[junit4] 2> 638171 INFO (TEST-TestDistributedSearch.test-seed#[8B8823E327AD657E]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 638171 INFO (TEST-TestDistributedSearch.test-seed#[8B8823E327AD657E]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-12-22T09:55:29.056Z
[junit4] 2> 638171 INFO (TEST-TestDistributedSearch.test-seed#[8B8823E327AD657E]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_8B8823E327AD657E-001/tempDir-001/control/solr.xml
[junit4] 2> 638175 INFO (TEST-TestDistributedSearch.test-seed#[8B8823E327AD657E]) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
[junit4] 2> 638175 INFO (TEST-TestDistributedSearch.test-seed#[8B8823E327AD657E]) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
[junit4] 2> 638177 INFO (TEST-TestDistributedSearch.test-seed#[8B8823E327AD657E]) [ ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@2cd66793, but no JMX reporters were configured - adding default JMX reporter.
[junit4] 2> 638218 INFO (TEST-TestDistributedSearch.test-seed#[8B8823E327AD657E]) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2cd66793
[junit4] 2> 638225 INFO (TEST-TestDistributedSearch.test-seed#[8B8823E327AD657E]) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2cd66793
[junit4] 2> 638225 INFO (TEST-TestDistributedSearch.test-seed#[8B8823E327AD657E]) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2cd66793
[junit4] 2> 638227 INFO (TEST-TestDistributedSearch.test-seed#[8B8823E327AD657E]) [ ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_8B8823E327AD657E-001/tempDir-001/control/cores
[junit4] 2> 638227 INFO (TEST-TestDistributedSearch.test-seed#[8B8823E327AD657E]) [ ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
[junit4] 2> 638227 INFO (coreLoadExecutor-2811-thread-1) [ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 638228 INFO (coreLoadExecutor-2811-thread-1) [ x:collection1] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/lib, /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/lib/classes]
[junit4] 2> 638276 INFO (coreLoadExecutor-2811-thread-1) [ x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
[junit4] 2> 638299 INFO (coreLoadExecutor-2811-thread-1) [ x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
[junit4] 2> 638419 INFO (coreLoadExecutor-2811-thread-1) [ x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
[junit4] 2> 638425 INFO (coreLoadExecutor-2811-thread-1) [ x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from configset /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1, trusted=true
[junit4] 2> 638426 INFO (coreLoadExecutor-2811-thread-1) [ x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2cd66793
[junit4] 2> 638426 INFO (coreLoadExecutor-2811-thread-1) [ x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 638426 INFO (coreLoadExecutor-2811-thread-1) [ x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_8B8823E327AD657E-001/tempDir-001/control/cores/collection1/data/]
[junit4] 2> 638427 INFO (coreLoadExecutor-2811-thread-1) [ x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=19, maxMergeAtOnceExplicit=31, maxMergedSegmentMB=48.1015625, floorSegmentMB=2.001953125, forceMergeDeletesPctAllowed=10.006900497219839, segmentsPerTier=11.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5887011307659544
[junit4] 2> 638468 INFO (coreLoadExecutor-2811-thread-1) [ x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 638468 INFO (coreLoadExecutor-2811-thread-1) [ x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 638469 INFO (coreLoadExecutor-2811-thread-1) [ x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 638469 INFO (coreLoadExecutor-2811-thread-1) [ x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 638470 INFO (coreLoadExecutor-2811-thread-1) [ x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=13, maxMergeAtOnceExplicit=18, maxMergedSegmentMB=54.5966796875, floorSegmentMB=1.927734375, forceMergeDeletesPctAllowed=24.016736627113637, segmentsPerTier=15.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
[junit4] 2> 638470 INFO (coreLoadExecutor-2811-thread-1) [ x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@7c1adbb3[collection1] main]
[junit4] 2> 638470 INFO (coreLoadExecutor-2811-thread-1) [ x:collection1] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
[junit4] 2> 638479 INFO (coreLoadExecutor-2811-thread-1) [ x:collection1] o.a.s.s.DirectSolrSpellChecker init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4] 2> 638516 INFO (coreLoadExecutor-2811-thread-1) [ x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 638517 INFO (searcherExecutor-2812-thread-1-processing-x:collection1) [ x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
[junit4] 2> 638517 INFO (coreLoadExecutor-2811-thread-1) [ x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1587477510254231552
[junit4] 2> 638517 INFO (searcherExecutor-2812-thread-1-processing-x:collection1) [ x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: direct
[junit4] 2> 638517 INFO (searcherExecutor-2812-thread-1-processing-x:collection1) [ x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: wordbreak
[junit4] 2> 638517 INFO (searcherExecutor-2812-thread-1-processing-x:collection1) [ x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: multipleFields
[junit4] 2> 638517 INFO (searcherExecutor-2812-thread-1-processing-x:collection1) [ x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: jarowinkler
[junit4] 2> 638517 INFO (searcherExecutor-2812-thread-1-processing-x:collection1) [ x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: external
[junit4] 2> 638534 INFO (searcherExecutor-2812-thread-1-processing-x:collection1) [ x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: freq
[junit4] 2> 638534 INFO (searcherExecutor-2812-thread-1-processing-x:collection1) [ x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: fqcn
[junit4] 2> 638534 INFO (searcherExecutor-2812-thread-1-processing-x:collection1) [ x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: perDict
[junit4] 2> 638535 INFO (searcherExecutor-2812-thread-1-processing-x:collection1) [ x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@7c1adbb3[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 638623 INFO (TEST-TestDistributedSearch.test-seed#[8B8823E327AD657E]) [ ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_8B8823E327AD657E-001/tempDir-001/shard0/cores/collection1
[junit4] 2> 638624 INFO (TEST-TestDistributedSearch.test-seed#[8B8823E327AD657E]) [ ] o.e.j.s.Server jetty-9.3.20.v20170531
[junit4] 2> 638628 INFO (TEST-TestDistributedSearch.test-seed#[8B8823E327AD657E]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@c99c2cc{/,null,AVAILABLE}
[junit4] 2> 638632 INFO (TEST-TestDistributedSearch.test-seed#[8B8823E327AD657E]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@4dd1dc85{HTTP/1.1,[http/1.1]}{127.0.0.1:46478}
[junit4] 2> 638632 INFO (TEST-TestDistributedSearch.test-seed#[8B8823E327AD657E]) [ ] o.e.j.s.Server Started @641634ms
[junit4] 2> 638632 INFO (TEST-TestDistributedSearch.test-seed#[8B8823E327AD657E]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/, hostPort=46478, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_8B8823E327AD657E-001/tempDir-001/shard0/cores}
[junit4] 2> 638633 ERROR (TEST-TestDistributedSearch.test-seed#[8B8823E327AD657E]) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 638633 INFO (TEST-TestDistributedSearch.test-seed#[8B8823E327AD657E]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 7.3.0
[junit4] 2> 638633 INFO (TEST-TestDistributedSearch.test-seed#[8B8823E327AD657E]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in standalone mode on port null
[junit4] 2> 638633 INFO (TEST-TestDistributedSearch.test-seed#[8B8823E327AD657E]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 638633 INFO (TEST-TestDistributedSearch.test-seed#[8B8823E327AD657E]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-12-22T09:55:29.518Z
[junit4] 2> 638633 INFO (TEST-TestDistributedSearch.test-seed#[8B8823E327AD657E]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_8B8823E327AD657E-001/tempDir-001/shard0/solr.xml
[junit4] 2> 638636 INFO (TEST-TestDistributedSearch.test-seed#[8B8823E327AD657E]) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
[junit4] 2> 638636 INFO (TEST-TestDistributedSearch.test-seed#[8B8823E327AD657E]) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
[junit4] 2> 638637 INFO (TEST-TestDistributedSearch.test-seed#[8B8823E327AD657E]) [ ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@2cd66793, but no JMX reporters were configured - adding default JMX reporter.
[junit4] 2> 638665 INFO (TEST-TestDistributedSearch.test-seed#[8B8823E327AD657E]) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2cd66793
[junit4] 2> 638673 INFO (TEST-TestDistributedSearch.test-seed#[8B8823E327AD657E]) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2cd66793
[junit4] 2> 638673 INFO (TEST-TestDistributedSearch.test-seed#[8B8823E327AD657E]) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2cd66793
[junit4] 2> 638675 INFO (TEST-TestDistributedSearch.test-seed#[8B8823E327AD657E]) [ ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_8B8823E327AD657E-001/tempDir-001/shard0/cores
[junit4] 2> 638675 INFO (TEST-TestDistributedSearch.test-seed#[8B8823E327AD657E]) [ ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
[junit4] 2> 638675 INFO (coreLoadExecutor-2822-thread-1) [ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 638676 INFO (coreLoadExecutor-2822-thread-1) [ x:collection1] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/lib, /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/lib/classes]
[junit4] 2> 638705 INFO (coreLoadExecutor-2822-thread-1) [ x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
[junit4] 2> 638726 INFO (coreLoadExecutor-2822-thread-1) [ x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
[junit4] 2> 638829 INFO (coreLoadExecutor-2822-thread-1) [ x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
[junit4] 2> 638836 INFO (coreLoadExecutor-2822-thread-1) [ x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from configset /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1, trusted=true
[junit4] 2> 638836 INFO (coreLoadExecutor-2822-thread-1) [ x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2cd66793
[junit4] 2> 638836 INFO (coreLoadExecutor-2822-thread-1) [ x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 638836 INFO (coreLoadExecutor-2822-thread-1) [ x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_8B8823E327AD657E-001/tempDir-001/shard0/cores/collection1/data/]
[junit4] 2> 638838 INFO (coreLoadExecutor-2822-thread-1) [ x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=19, maxMergeAtOnceExplicit=31, maxMergedSegmentMB=48.1015625, floorSegmentMB=2.001953125, forceMergeDeletesPctAllowed=10.006900497219839, segmentsPerTier=11.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5887011307659544
[junit4] 2> 638886 INFO (coreLoadExecutor-2822-thread-1) [ x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 638886 INFO (coreLoadExecutor-2822-thread-1) [ x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 638887 INFO (coreLoadExecutor-2822-thread-1) [ x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 638887 INFO (coreLoadExecutor-2822-thread-1) [ x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 638888 INFO (coreLoadExecutor-2822-thread-1) [ x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=13, maxMergeAtOnceExplicit=18, maxMergedSegmentMB=54.5966796875, floorSegmentMB=1.927734375, forceMergeDeletesPctAllowed=24.016736627113637, segmentsPerTier=15.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
[junit4] 2> 638888 INFO (coreLoadExecutor-2822-thread-1) [ x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@7a8a95d3[collection1] main]
[junit4] 2> 638889 INFO (coreLoadExecutor-2822-thread-1) [ x:collection1] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
[junit4] 2> 638898 INFO (coreLoadExecutor-2822-thread-1) [ x:collection1] o.a.s.s.DirectSolrSpellChecker init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4] 2> 638942 INFO (coreLoadExecutor-2822-thread-1) [ x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 638942 INFO (searcherExecutor-2823-thread-1-processing-x:collection1) [ x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
[junit4] 2> 638942 INFO (coreLoadExecutor-2822-thread-1) [ x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1587477510699876352
[junit4] 2> 638942 INFO (searcherExecutor-2823-thread-1-processing-x:collection1) [ x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: direct
[junit4] 2> 638942 INFO (searcherExecutor-2823-thread-1-processing-x:collection1) [ x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: wordbreak
[junit4] 2> 638942 INFO (searcherExecutor-2823-thread-1-processing-x:collection1) [ x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: multipleFields
[junit4] 2> 638943 INFO (searcherExecutor-2823-thread-1-processing-x:collection1) [ x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: jarowinkler
[junit4] 2> 638943 INFO (searcherExecutor-2823-thread-1-processing-x:collection1) [ x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: external
[junit4] 2> 638943 INFO (searcherExecutor-2823-thread-1-processing-x:collection1) [ x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: freq
[junit4] 2> 638943 INFO (searcherExecutor-2823-thread-1-processing-x:collection1) [ x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: fqcn
[junit4] 2> 638943 INFO (searcherExecutor-2823-thread-1-processing-x:collection1) [ x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: perDict
[junit4] 2> 638945 INFO (TEST-TestDistributedSearch.test-seed#[8B8823E327AD657E]) [ ] o.a.s.SolrTestCaseJ4 ###Starting test
[junit4] 2> 638947 INFO (searcherExecutor-2823-thread-1-processing-x:collection1) [ x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@7a8a95d3[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 638949 INFO (qtp1414754303-5565) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{deleteByQuery=*:* (-1587477510705119232)} 0 1
[junit4] 2> 638952 INFO (qtp1254730362-5582) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{deleteByQuery=*:* (-1587477510709313536)} 0 0
[junit4] 2> 638957 INFO (qtp1414754303-5566) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[1 (1587477510711410688)]} 0 4
[junit4] 2> 638958 INFO (qtp1254730362-5583) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[1 (1587477510715604992)]} 0 0
[junit4] 2> 638959 INFO (qtp1414754303-5567) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[2 (1587477510716653568)]} 0 0
[junit4] 2> 638960 INFO (qtp1254730362-5584) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[2 (1587477510717702144)]} 0 0
[junit4] 2> 638961 INFO (qtp1414754303-5568) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[3 (1587477510718750720)]} 0 0
[junit4] 2> 638961 INFO (qtp1254730362-5585) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[3 (1587477510719799296)]} 0 0
[junit4] 2> 638962 INFO (qtp1414754303-5569) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[4 (1587477510720847872)]} 0 0
[junit4] 2> 638963 INFO (qtp1254730362-5586) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[4 (1587477510721896448)]} 0 0
[junit4] 2> 638969 INFO (qtp1414754303-5569) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[5 (1587477510727139328)]} 0 0
[junit4] 2> 638970 INFO (qtp1254730362-5579) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[5 (1587477510728187904)]} 0 0
[junit4] 2> 638974 INFO (qtp1414754303-5569) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[6 (1587477510732382208)]} 0 0
[junit4] 2> 638974 INFO (qtp1254730362-5579) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[6 (1587477510733430784)]} 0 0
[junit4] 2> 638975 INFO (qtp1414754303-5569) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[7 (1587477510734479360)]} 0 0
[junit4] 2> 638976 INFO (qtp1254730362-5579) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[7 (1587477510734479360)]} 0 0
[junit4] 2> 638976 INFO (qtp1414754303-5569) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[8 (1587477510735527936)]} 0 0
[junit4] 2> 638977 INFO (qtp1254730362-5579) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[8 (1587477510736576512)]} 0 0
[junit4] 2> 638978 INFO (qtp1414754303-5568) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[9 (1587477510737625088)]} 0 0
[junit4] 2> 638979 INFO (qtp1254730362-5579) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[9 (1587477510737625088)]} 0 0
[junit4] 2> 638979 INFO (qtp1414754303-5568) [ x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1587477510738673664,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 638979 INFO (qtp1414754303-5568) [ x:collection1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@6a3a88f7 commitCommandVersion:1587477510738673664
[junit4] 2> 638990 INFO (qtp1414754303-5568) [ x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@3fadd06c[collection1] main]
[junit4] 2> 638990 INFO (qtp1414754303-5568) [ x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 638992 INFO (searcherExecutor-2812-thread-1-processing-x:collection1) [ x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@3fadd06c[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.3.0):c9)))}
[junit4] 2> 638992 INFO (qtp1414754303-5568) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 13
[junit4] 2> 638992 INFO (qtp1254730362-5579) [ x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1587477510752305152,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 638992 INFO (qtp1254730362-5579) [ x:collection1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@6b0eb7af commitCommandVersion:1587477510752305152
[junit4] 2> 639017 INFO (qtp1254730362-5579) [ x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@19908917[collection1] main]
[junit4] 2> 639019 INFO (searcherExecutor-2823-thread-1-processing-x:collection1) [ x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@19908917[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.3.0):c9)))}
[junit4] 2> 639019 INFO (qtp1254730362-5579) [ x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 639019 INFO (qtp1254730362-5579) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 27
[junit4] 2> 639022 INFO (qtp1414754303-5565) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[10 (1587477510781665280)]} 0 1
[junit4] 2> 639023 INFO (qtp1254730362-5582) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[10 (1587477510783762432)]} 0 1
[junit4] 2> 639025 INFO (qtp1414754303-5566) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[11 (1587477510785859584)]} 0 0
[junit4] 2> 639025 INFO (qtp1254730362-5583) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[11 (1587477510786908160)]} 0 0
[junit4] 2> 639026 INFO (qtp1414754303-5567) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[12 (1587477510787956736)]} 0 0
[junit4] 2> 639027 INFO (qtp1254730362-5583) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[12 (1587477510787956736)]} 0 0
[junit4] 2> 639027 INFO (qtp1414754303-5567) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[13 (1587477510789005312)]} 0 0
[junit4] 2> 639028 INFO (qtp1254730362-5583) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[13 (1587477510790053888)]} 0 0
[junit4] 2> 639029 INFO (qtp1414754303-5567) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[1001 (1587477510790053888)]} 0 0
[junit4] 2> 639030 INFO (qtp1254730362-5583) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[1001 (1587477510791102464)]} 0 0
[junit4] 2> 639030 INFO (qtp1414754303-5567) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[14 (1587477510792151040)]} 0 0
[junit4] 2> 639031 INFO (qtp1254730362-5583) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[14 (1587477510793199616)]} 0 0
[junit4] 2> 639032 INFO (qtp1414754303-5567) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[15 (1587477510794248192)]} 0 0
[junit4] 2> 639032 INFO (qtp1254730362-5583) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[15 (1587477510794248192)]} 0 0
[junit4] 2> 639033 INFO (qtp1414754303-5567) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[16 (1587477510795296768)]} 0 0
[junit4] 2> 639034 INFO (qtp1254730362-5583) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[16 (1587477510795296768)]} 0 0
[junit4] 2> 639034 INFO (qtp1414754303-5567) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[17 (1587477510796345344)]} 0 0
[junit4] 2> 639035 INFO (qtp1254730362-5585) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[17 (1587477510797393920)]} 0 0
[junit4] 2> 639036 INFO (qtp1414754303-5562) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[100 (1587477510798442496)]} 0 0
[junit4] 2> 639037 INFO (qtp1254730362-5585) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[100 (1587477510798442496)]} 0 0
[junit4] 2> 639037 INFO (qtp1414754303-5562) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[101 (1587477510799491072)]} 0 0
[junit4] 2> 639038 INFO (qtp1254730362-5585) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[101 (1587477510799491072)]} 0 0
[junit4] 2> 639038 INFO (qtp1414754303-5565) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[102 (1587477510800539648)]} 0 0
[junit4] 2> 639039 INFO (qtp1254730362-5585) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[102 (1587477510801588224)]} 0 0
[junit4] 2> 639040 INFO (qtp1414754303-5565) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[103 (1587477510801588224)]} 0 0
[junit4] 2> 639040 INFO (qtp1254730362-5584) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[103 (1587477510802636800)]} 0 0
[junit4] 2> 639041 INFO (qtp1414754303-5565) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[104 (1587477510803685376)]} 0 0
[junit4] 2> 639041 INFO (qtp1254730362-5584) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[104 (1587477510803685376)]} 0 0
[junit4] 2> 639042 INFO (qtp1414754303-5565) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[105 (1587477510804733952)]} 0 0
[junit4] 2> 639043 INFO (qtp1254730362-5584) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[105 (1587477510804733952)]} 0 0
[junit4] 2> 639043 INFO (qtp1414754303-5565) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[106 (1587477510805782528)]} 0 0
[junit4] 2> 639044 INFO (qtp1254730362-5579) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[106 (1587477510805782528)]} 0 0
[junit4] 2> 639044 INFO (qtp1414754303-5565) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[107 (1587477510806831104)]} 0 0
[junit4] 2> 639045 INFO (qtp1254730362-5579) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[107 (1587477510807879680)]} 0 0
[junit4] 2> 639046 INFO (qtp1414754303-5568) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[108 (1587477510807879680)]} 0 0
[junit4] 2> 639046 INFO (qtp1254730362-5579) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[108 (1587477510808928256)]} 0 0
[junit4] 2> 639047 INFO (qtp1414754303-5568) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[109 (1587477510809976832)]} 0 0
[junit4] 2> 639047 INFO (qtp1254730362-5579) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[109 (1587477510809976832)]} 0 0
[junit4] 2> 639048 INFO (qtp1414754303-5568) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[110 (1587477510811025408)]} 0 0
[junit4] 2> 639049 INFO (qtp1254730362-5579) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[110 (1587477510811025408)]} 0 0
[junit4] 2> 639049 INFO (qtp1414754303-5569) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[111 (1587477510812073984)]} 0 0
[junit4] 2> 639050 INFO (qtp1254730362-5584) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[111 (1587477510813122560)]} 0 0
[junit4] 2> 639051 INFO (qtp1414754303-5569) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[112 (1587477510813122560)]} 0 0
[junit4] 2> 639051 INFO (qtp1254730362-5584) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[112 (1587477510814171136)]} 0 0
[junit4] 2> 639052 INFO (qtp1414754303-5569) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[113 (1587477510815219712)]} 0 0
[junit4] 2> 639053 INFO (qtp1254730362-5585) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[113 (1587477510815219712)]} 0 0
[junit4] 2> 639053 INFO (qtp1414754303-5562) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[114 (1587477510816268288)]} 0 0
[junit4] 2> 639054 INFO (qtp1254730362-5585) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[114 (1587477510817316864)]} 0 0
[junit4] 2> 639054 INFO (qtp1414754303-5562) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[115 (1587477510817316864)]} 0 0
[junit4] 2> 639055 INFO (qtp1254730362-5585) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[115 (1587477510818365440)]} 0 0
[junit4] 2> 639056 INFO (qtp1414754303-5562) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[116 (1587477510818365440)]} 0 0
[junit4] 2> 639056 INFO (qtp1254730362-5585) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[116 (1587477510819414016)]} 0 0
[junit4] 2> 639057 INFO (qtp1414754303-5562) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[117 (1587477510820462592)]} 0 0
[junit4] 2> 639059 INFO (qtp1254730362-5582) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[117 (1587477510822559744)]} 0 0
[junit4] 2> 639060 INFO (qtp1414754303-5565) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[118 (1587477510822559744)]} 0 0
[junit4] 2> 639060 INFO (qtp1254730362-5582) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[118 (1587477510823608320)]} 0 0
[junit4] 2> 639061 INFO (qtp1414754303-5565) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[119 (1587477510824656896)]} 0 0
[junit4] 2> 639061 INFO (qtp1254730362-5583) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[119 (1587477510824656896)]} 0 0
[junit4] 2> 639062 INFO (qtp1414754303-5566) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[120 (1587477510825705472)]} 0 0
[junit4] 2> 639063 INFO (qtp1254730362-5583) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[120 (1587477510825705472)]} 0 0
[junit4] 2> 639063 INFO (qtp1414754303-5566) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[121 (1587477510826754048)]} 0 0
[junit4] 2> 639064 INFO (qtp1254730362-5583) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[121 (1587477510827802624)]} 0 0
[junit4] 2> 639064 INFO (qtp1414754303-5566) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[122 (1587477510827802624)]} 0 0
[junit4] 2> 639065 INFO (qtp1254730362-5583) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[122 (1587477510828851200)]} 0 0
[junit4] 2> 639066 INFO (qtp1414754303-5566) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[123 (1587477510828851200)]} 0 0
[junit4] 2> 639066 INFO (qtp1254730362-5584) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[123 (1587477510829899776)]} 0 0
[junit4] 2> 639067 INFO (qtp1414754303-5569) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[124 (1587477510830948352)]} 0 0
[junit4] 2> 639067 INFO (qtp1254730362-5584) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[124 (1587477510830948352)]} 0 0
[junit4] 2> 639068 INFO (qtp1414754303-5569) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[125 (1587477510831996928)]} 0 0
[junit4] 2> 639069 INFO (qtp1254730362-5584) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[125 (1587477510831996928)]} 0 0
[junit4] 2> 639069 INFO (qtp1414754303-5568) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[126 (1587477510833045504)]} 0 0
[junit4] 2> 639070 INFO (qtp1254730362-5579) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[126 (1587477510834094080)]} 0 0
[junit4] 2> 639070 INFO (qtp1414754303-5568) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[127 (1587477510834094080)]} 0 0
[junit4] 2> 639071 INFO (qtp1254730362-5579) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[127 (1587477510835142656)]} 0 0
[junit4] 2> 639072 INFO (qtp1414754303-5568) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[128 (1587477510835142656)]} 0 0
[junit4] 2> 639072 INFO (qtp1254730362-5579) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[128 (1587477510836191232)]} 0 0
[junit4] 2> 639073 INFO (qtp1414754303-5568) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[129 (1587477510837239808)]} 0 0
[junit4] 2> 639073 INFO (qtp1254730362-5579) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[129 (1587477510837239808)]} 0 0
[junit4] 2> 639074 INFO (qtp1414754303-5565) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[130 (1587477510838288384)]} 0 0
[junit4] 2> 639081 INFO (qtp1254730362-5586) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[130 (1587477510845628416)]} 0 0
[junit4] 2> 639082 INFO (qtp1414754303-5565) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[131 (1587477510845628416)]} 0 0
[junit4] 2> 639082 INFO (qtp1254730362-5586) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[131 (1587477510846676992)]} 0 0
[junit4] 2> 639083 INFO (qtp1414754303-5567) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[132 (1587477510847725568)]} 0 0
[junit4] 2> 639085 INFO (qtp1254730362-5585) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[132 (1587477510848774144)]} 0 0
[junit4] 2> 639085 INFO (qtp1414754303-5567) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[133 (1587477510849822720)]} 0 0
[junit4] 2> 639086 INFO (qtp1254730362-5585) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[133 (1587477510850871296)]} 0 0
[junit4] 2> 639087 INFO (qtp1414754303-5567) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[134 (1587477510850871296)]} 0 0
[junit4] 2> 639087 INFO (qtp1254730362-5585) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[134 (1587477510851919872)]} 0 0
[junit4] 2> 639088 INFO (qtp1414754303-5567) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[135 (1587477510851919872)]} 0 0
[junit4] 2> 639088 INFO (qtp1254730362-5585) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[135 (1587477510852968448)]} 0 0
[junit4] 2> 639089 INFO (qtp1414754303-5569) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[136 (1587477510854017024)]} 0 0
[junit4] 2> 639090 INFO (qtp1254730362-5585) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[136 (1587477510854017024)]} 0 0
[junit4] 2> 639090 INFO (qtp1414754303-5569) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[137 (1587477510855065600)]} 0 0
[junit4] 2> 639091 INFO (qtp1254730362-5586) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[137 (1587477510855065600)]} 0 0
[junit4] 2> 639091 INFO (qtp1414754303-5562) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[138 (1587477510856114176)]} 0 0
[junit4] 2> 639092 INFO (qtp1254730362-5583) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[138 (1587477510857162752)]} 0 0
[junit4] 2> 639092 INFO (qtp1414754303-5562) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[139 (1587477510857162752)]} 0 0
[junit4] 2> 639093 INFO (qtp1254730362-5583) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[139 (1587477510858211328)]} 0 0
[junit4] 2> 639094 INFO (qtp1414754303-5562) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[140 (1587477510858211328)]} 0 0
[junit4] 2> 639094 INFO (qtp1254730362-5583) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[140 (1587477510859259904)]} 0 0
[junit4] 2> 639095 INFO (qtp1414754303-5562) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[141 (1587477510860308480)]} 0 0
[junit4] 2> 639095 INFO (qtp1254730362-5583) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[141 (1587477510860308480)]} 0 0
[junit4] 2> 639096 INFO (qtp1414754303-5562) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[142 (1587477510861357056)]} 0 0
[junit4] 2> 639097 INFO (qtp1254730362-5583) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[142 (1587477510861357056)]} 0 0
[junit4] 2> 639097 INFO (qtp1414754303-5562) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[143 (1587477510862405632)]} 0 0
[junit4] 2> 639098 INFO (qtp1254730362-5583) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[143 (1587477510863454208)]} 0 0
[junit4] 2> 639098 INFO (qtp1414754303-5562) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[144 (1587477510863454208)]} 0 0
[junit4] 2> 639099 INFO (qtp1254730362-5583) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[144 (1587477510864502784)]} 0 0
[junit4] 2> 639100 INFO (qtp1414754303-5562) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[145 (1587477510864502784)]} 0 0
[junit4] 2> 639100 INFO (qtp1254730362-5583) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[145 (1587477510865551360)]} 0 0
[junit4] 2> 639101 INFO (qtp1414754303-5567) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[146 (1587477510866599936)]} 0 0
[junit4] 2> 639101 INFO (qtp1254730362-5583) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[146 (1587477510866599936)]} 0 0
[junit4] 2> 639102 INFO (qtp1414754303-5567) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[147 (1587477510867648512)]} 0 0
[junit4] 2> 639103 INFO (qtp1254730362-5583) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[147 (1587477510867648512)]} 0 0
[junit4] 2> 639103 INFO (qtp1414754303-5567) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[148 (1587477510868697088)]} 0 0
[junit4] 2> 639104 INFO (qtp1254730362-5583) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[148 (1587477510868697088)]} 0 0
[junit4] 2> 639104 INFO (qtp1414754303-5567) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[149 (1587477510869745664)]} 0 0
[junit4] 2> 639105 INFO (qtp1254730362-5583) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[149 (1587477510870794240)]} 0 0
[junit4] 2> 639105 INFO (qtp1414754303-5567) [ x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1587477510870794240,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 639105 INFO (qtp1414754303-5567) [ x:collection1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@6a3a88f7 commitCommandVersion:1587477510870794240
[junit4] 2> 639121 INFO (qtp1414754303-5567) [ x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@4c07b3a7[collection1] main]
[junit4] 2> 639122 INFO (qtp1414754303-5567) [ x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 639123 INFO (searcherExecutor-2812-thread-1-processing-x:collection1) [ x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@4c07b3a7[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.3.0):c9) Uninverting(_1(7.3.0):c59)))}
[junit4] 2> 639123 INFO (qtp1414754303-5567) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 18
[junit4] 2> 639124 INFO (qtp1254730362-5583) [ x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1587477510890717184,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 639124 INFO (qtp1254730362-5583) [ x:collection1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@6b0eb7af commitCommandVersion:1587477510890717184
[junit4] 2> 639130 INFO (qtp1254730362-5583) [ x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@224558d2[collection1] main]
[junit4] 2> 639131 INFO (qtp1254730362-5583) [ x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 639132 INFO (searcherExecutor-2823-thread-1-processing-x:collection1) [ x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@224558d2[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.3.0):c9) Uninverting(_1(7.3.0):c59)))}
[junit4] 2> 639132 INFO (qtp1254730362-5583) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 8
[junit4] 2> 639133 INFO (qtp1414754303-5562) [ x:collection1] o.a.s.h.RequestHandlerBase Ignoring exception matching parameter cannot be negative
[junit4] 2> 639133 INFO (qtp1414754303-5562) [ x:collection1] o.a.s.c.S.Request [collection1] webapp= path=/select params={q=*&distrib=false&start=-1&wt=javabin&version=2} status=400 QTime=0
[junit4] 2> 639134 INFO (qtp1414754303-5562) [ x:collection1] o.a.s.h.RequestHandlerBase Ignoring exception matching parameter cannot be negative
[junit4] 2> 639134 INFO (qtp1414754303-5562) [ x:collection1] o.a.s.c.S.Request [collection1] webapp= path=/select params={q=*&distrib=false&start=0&rows=-1&wt=javabin&version=2} status=400 QTime=0
[junit4] 2> 639139 INFO (qtp1414754303-5562) [ x:collection1] o.a.s.c.S.Request [collection1] webapp= path=/select params={q=*:*&distrib=false&sort=n_ti1+desc&wt=javabin&version=2} hits=68 status=0 QTime=0
[junit4] 2> 639340 INFO (qtp1254730362-5580) [ x:collection1] o.a.s.c.S.Request [collection1] webapp= path=/select params={df=text&distrib=false&fl=id&shards.purpose=4&start=0&fsv=true&sort=n_ti1+desc&shard.url=[ff01::083]:33332/|http://127.0.0.1:46478//collection1&rows=10&version=2&q=*:*&NOW=1513936530025&isShard=true&wt=javabin} hits=68 status=0 QTime=0
[junit4] 2> 639342 INFO (qtp1254730362-5580) [ x:collection1] o.a.s.c.S.Request [collection1] webapp= path=/select params={q=*:*&df=text&distrib=false&shards.purpose=64&NOW=1513936530025&ids=144,3,126,148,127,116,128,140,141,120&isShard=true&shard.url=[ff01::083]:33332/|http://127.0.0.1:46478//collection1&wt=javabin&version=2} status=0 QTime=1
[junit4] 2> 639343 INFO (qtp1254730362-5584) [ x:collection1] o.a.s.c.S.Request [collection1] webapp= path=/select params={q=*:*&shards=[ff01::083]:33332/|http://127.0.0.1:46478//collection1&sort=n_ti1+desc&wt=javabin&version=2} hits=68 status=0 QTime=203
[junit4] 2> 639346 INFO (qtp1414754303-5562) [ x:collection1] o.a.s.c.S.Request [collection1] webapp= path=/select params={q=*:*&distrib=false&sort=n_ti1+asc&wt=javabin&version=2} hits=68 status=0 QTime=0
[junit4] 2> 639347 INFO (qtp1254730362-5582) [ x:collection1] o.a.s.c.S.Request [collection1] webapp= path=/select params={df=text&distrib=false&fl=id&shards.purpose=4&start=0&fsv=true&sort=n_ti1+asc&shard.url=[ff01::083]:33332/|http://127.0.0.1:46478//collection1&rows=10&version=2&q=*:*&NOW=1513936530231&isShard=true&wt=javabin} hits=68 status=0 QTime=0
[junit4] 2> 639349 INFO (qtp1254730362-5583) [ x:collection1] o.a.s.c.S.Request [collection1] webapp= path=/select params={q=*:*&df=text&distrib=false&shards.purpose=64&NOW=1513936530231&ids=122,112,146,114,137,149,17,139,129,131&isShard=true&shard.url=[ff01::083]:33332/|http://127.0.0.1:46478//collection1&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 639351 INFO (qtp1254730362-5585) [ x:collection1] o.a.s.c.S.Request [collection1] webapp= path=/select params={q=*:*&shards=[ff01::083]:33332/|http://127.0.0.1:46478//collection1&sort=n_ti1+asc&wt=javabin&version=2} hits=68 status=0 QTime=4
[junit4] 2> 639352 INFO (qtp1414754303-5568) [ x:collection1] o.a.s.c.S.Request [collection1] webapp= path=/select params={q=*:*&distrib=false&sort=n_f1+desc&wt=javabin&version=2} hits=68 status=0 QTime=0
[junit4] 2> 639360 INFO (qtp1254730362-5586) [ x:collection1] o.a.s.c.S.Request [collection1] webapp= path=/select params={df=text&distrib=false&fl=id&shards.purpose=4&start=0&fsv=true&sort=n_f1+desc&shard.url=http://127.0.0.1:46478//collection1&rows=10&version=2&q=*:*&NOW=1513936530244&isShard=true&wt=javabin} hits=68 status=0 QTime=0
[junit4] 2> 639361 INFO (qtp1254730362-5586) [ x:collection1] o.a.s.c.S.Request [collection1] webapp= path=/select params={q=*:*&df=text&distrib=false&shards.purpose=64&NOW=1513936530244&ids=11,110,132,12,102,146,6,138,119,10&isShard=true&shard.url=http://127.0.0.1:46478//collection1&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 639362 INFO (qtp1254730362-5579) [ x:collection1] o.a.s.c.S.Request [collection1] webapp= path=/select params={q=*:*&shards=http://127.0.0.1:46478//collection1&sort=n_f1+desc&wt=javabin&version=2} hits=68 status=0 QTime=2
[junit4] 2> 639363 INFO (qtp1414754303-5567) [ x:collection1] o.a.s.c.S.Request [collection1] webapp= path=/select params={q=*:*&distrib=false&sort=n_f1+asc&wt=javabin&version=2} hits=68 status=0 QTime=0
[junit4] 2> 639532 INFO (qtp1254730362-5586) [ x:collection1] o.a.s.c.S.Request [collection1] webapp= path=/select params={df=text&distrib=false&fl=id&shards.purpose=4&start=0&fsv=true&sort=n_f1+asc&shard.url=[ff01::083]:33332/|[ff01::213]:33332/|http://127.0.0.1:46478//collection1&rows=10&version=2&q=*:*&NOW=1513936530249&isShard=true&wt=javabin} hits=68 status=0 QTime=0
[junit4] 2> 639539 INFO (qtp1254730362-5585) [ x:collection1] o.a.s.c.S.Request [collection1] webapp= path=/select params={q=*:*&df=text&distrib=false&shards.purpose=64&NOW=1513936530249&ids=122,113,114,103,116,106,117,8,108,120&isShard=true&shard.url=[ff01::083]:33332/|[ff01::213]:33332/|http://127.0.0.1:46478//collection1&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 639540 INFO (qtp1254730362-5580) [ x:collection1] o.a.s.c.S.Request [collection1] webapp= path=/select params={q=*:*&shards=[ff01::083]:33332/|[ff01::213]:33332/|http://127.0.0.1:46478//collection1&sort=n_f1+asc&wt=javabin&version=2} hits=68 status=0 QTime=176
[junit4] 2> 639541 INFO (qtp1414754303-5567) [ x:collection1] o.a.s.c.S.Request [collection1] webapp= path=/select params={q=*:*&distrib=false&sort=n_tf1+desc&wt=javabin&version=2} hits=68 status=0 QTime=0
[junit4] 2> 639728 INFO (qtp1254730362-5584) [ x:collection1] o.a.s.c.S.Request [collection1] webapp= path=/select params={df=text&distrib=false&fl=id&shards.purpose=4&start=0&fsv=true&sort=n_tf1+desc&shard.url=[ff01::114]:33332/|http://127.0.0.1:46478//collection1&rows=10&version=2&q=*:*&NOW=1513936530427&isShard=true&wt=javabin} hits=68 status=0 QTime=0
[junit4] 2> 639736 INFO (qtp1254730362-5579) [ x:collection1] o.a.s.c.S.Request [collection1] webapp= path=/select params={q=*:*&df=text&distrib=false&shards.purpose=64&NOW=1513936530427&ids=110,121,100,124,125,137,6,7,8,10&isShard=true&shard.url=[ff01::114]:33332/|http://127.0.0.1:46478//collection1&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 639737 INFO (qtp1254730362-5583) [ x:collection1] o.a.s.c.S.Request [collection1] webapp= path=/select params={q=*:*&shards=[ff01::114]:33332/|http://127.0.0.1:46478//collection1&sort=n_tf1+desc&wt=javabin&version=2} hits=68 status=0 QTime=195
[junit4] 2> 639738 INFO (qtp1414754303-5567) [ x:collection1] o.a.s.c.S.Request [collection1] webapp= path=/select params={q=*:*&distrib=false&sort=n_tf1+asc&wt=javabin&version=2} hits=68 status=0 QTime=0
[junit4] 2> 639741 INFO (qtp1254730362-5579) [ x:collection1] o.a.s.c.S.Request [collection1] webapp= path=/select params={df=text&distrib=false&fl=id&shards.purpose=4&start=0&fsv=true&sort=n_tf1+asc&shard.url=[ff01::083]:33332/|[ff01::213]:33332/|http://127.0.0.1:46478//collection1&rows=10&version=2&q=*:*&NOW=1513936530624&isShard=true&wt=javabin} hits=68 status=0 QTime=1
[junit4] 2> 639743 INFO (qtp1254730362-5585) [ x:collection1] o.a.s.c.S.Request [collection1] webapp= path=/select params={q=*:*&df=text&distrib=false&shards.purpose=64&NOW=1513936530624&ids=143,132,134,2,102,113,135,14,126,105&isShard=true&shard.url=[ff01::083]:33332/|[ff01::213]:33332/|http://127.0.0.1:46478//collection1&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 639743 INFO (qtp1254730362-5582) [ x:collection1] o.a.s.c.S.Request [collection1] webapp= path=/select params={q=*:*&shards=[ff01::083]:33332/|[ff01::213]:33332/|http://127.0.0.1:46478//collection1&sort=n_tf1+asc&wt=javabin&version=2} hits=68 status=0 QTime=4
[junit4] 2> 639744 INFO (qtp1414754303-5567) [ x:collection1] o.a.s.c.S.Request [collection1] webapp= path=/select params={q=*:*&distrib=false&sort=n_d1+desc&wt=javabin&version=2} hits=68 status=0 QTime=0
[junit4] 2> 639746 INFO (qtp1254730362-5585) [ x:collection1] o.a.s.c.S.Request [collection1] webapp= path=/select params={df=text&distrib=false&fl=id&shards.purpose=4&start=0&fsv=true&sort=n_d1+desc&shard.url=http://127.0.0.1:46478//collection1&rows=10&version=2&q=*:*&NOW=1513936530630&isShard=true&wt=javabin} hits=68 status=0 QTime=0
[junit4] 2> 639746 INFO (qtp1254730362-5580) [ x:collection1] o.a.s.c.S.Request [collection1] webapp= path=/select params={q=*:*&df=text&distrib=false&shards.purpose=64&NOW=1513936530630&ids=11,122,123,147,103,6,138,106,119,120&isShard=true&shard.url=http://127.0.0.1:46478//collection1&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 639747 INFO (qtp1254730362-5586) [ x:collection1] o.a.s.c.S.Request [collection1] webapp= path=/select params={q=*:*&shards=http://127.0.0.1:46478//collection1&sort=n_d1+desc&wt=javabin&version=2} hits=68 status=0 QTime=2
[junit4] 2> 639753 INFO (qtp1414754303-5562) [ x:collection1] o.a.s.c.S.Request [collection1] webapp= path=/select params={q=*:*&distrib=false&sort=n_d1+asc&wt=javabin&version=2} hits=68 status=0 QTime=4
[junit4] 2> 639754 INFO (qtp1254730362-5580) [ x:collection1] o.a.s.c.S.Request [collection1] webapp= path=/select params={df=text&distrib=false&fl=id&shards.purpose=4&start=0&fsv=true&sort=n_d1+asc&shard.url=http://127.0.0.1:46478//collection1|[ff01::213]:33332/|[ff01::083]:33332/&rows=10&version=2&q=*:*&NOW=1513936530639&isShard=true&wt=javabin} hits=68 status=0 QTime=0
[junit4] 2> 639755 INFO (qtp1254730362-5582) [ x:collection1] o.a.s.c.S.Request [collection1] webapp= path=/select params={q=*:*&df=text&distrib=false&shards.purpose=64&NOW=1513936530639&ids=13,2,146,124,148,137,5,115,116,8&isShard=true&shard.url=http://127.0.0.1:46478//collection1|[ff01::213]:33332/|[ff01::083]:33332/&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 639756 INFO (qtp1254730362-5583) [ x:collection1] o.a.s.c.S.Request [collection1] webapp= path=/select params={q=*:*&shards=http://127.0.0.1:46478//collection1|[ff01::213]:33332/|[ff01::083]:33332/&sort=n_d1+asc&wt=javabin&version=2} hits=68 status=0 QTime=2
[junit4] 2> 639757 INFO (qtp1414754303-5568) [ x:collection1] o.a.s.c.S.Request [collection1] webapp= path=/select params={q=*:*&distrib=false&sort=n_td1+desc&wt=javabin&version=2} hits=68 status=0 QTime=0
[junit4] 2> 639758 INFO (qtp1254730362-5582) [ x:collection1] o.a.s.c.S.Request [collection1] webapp= path=/select params={df=text&distrib=false&fl=id&shards.purpose=4&start=0&fsv=true&sort=n_td1+desc&shard.url=[ff01::114]:33332/|http://127.0.0.1:46478//collection1|[ff01::213]:33332/|[ff01::083]:33332/&rows=10&version=2&q=*:*&NOW=1513936530643&isShard=true&wt=javabin} hits=68 status=0 QTime=0
[junit4] 2> 639759 INFO (qtp1254730362-5582) [ x:collection1] o.a.s.c.S.Request [collection1] webapp= path=/select params={q=*:*&df=text&distrib=false&shards.purpose=64&NOW=1513936530643&ids=14,103,115,6,149,116,138,119,9,10&isShard=true&shard.url=[ff01::114]:33332/|http://127.0.0.1:46478//collection1|[ff01::213]:33332/|[ff01::083]:33332/&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 639760 INFO (qtp1254730362-5584) [ x:collection1] o.a.s.c.S.Request [collection1] webapp= path=/select params={q=*:*&shards=[ff01::114]:33332/|http://127.0.0.1:46478//collection1|[ff01::213]:33332/|[ff01::083]:33332/&sort=n_td1+desc&wt=javabin&version=2} hits=68 status=0 QTime=2
[junit4] 2> 639760 INFO (qtp1414754303-5565) [ x:collection1] o.a.s.c.S.Request [collection1] webapp= path=/select params={q=*:*&distrib=false&sort=n_td1+asc&wt=javabin&version=2} hits=68 status=0 QTime=0
[junit4] 2> 639762 INFO (qtp1254730362-5582) [ x:collection1] o.a.s.c.S.Request [collection1] webapp= path=/select params={df=text&distrib=false&fl=id&shards.purpose=4&start=0&fsv=true&sort=n_td1+asc&shard.url=[ff01::083]:33332/|[ff01::114]:33332/|http://127.0.0.1:46478//collection1&rows=10&version=2&q=*:*&NOW=1513936530646&isShard=true&wt=javabin} hits=68 status=0 QTime=0
[junit4] 2> 639763 INFO (qtp1254730362-5582) [ x:collection1] o.a.s.c.S.Request [collection1] webapp= path=/select params={q=*:*&df=text&distrib=false&shards.purpose=64&NOW=1513936530646&ids=133,112,101,145,126,137,128,118,108,109&isShard=true&shard.url=[ff01::083]:33332/|[ff01::114]:33332/|http://127.0.0.1:46478//collection1&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 639763 INFO (qtp1254730362-5579) [ x:collection1] o.a.s.c.S.Request [collection1] webapp= path=/select params={q=*:*&shards=[ff01::083]:33332/|[ff01::114]:33332/|http://127.0.0.1:46478//collection1&sort=n_td1+asc&wt=javabin&version=2} hits=68 status=0 QTime=2
[junit4] 2> 639764 INFO (qtp1414754303-5565) [ x:collection1] o.a.s.c.S.Request [collection1] webapp= path=/select params={q=*:*&distrib=false&sort=n_l1+desc&wt=javabin&version=2} hits=68 status=0 QTime=0
[junit4] 2> 639766 INFO (qtp1254730362-5582) [ x:collection1] o.a.s.c.S.Request [collection1] webapp= path=/select params={df=text&distrib=false&fl=id&shards.purpose=4&start=0&fsv=true&sort=n_l1+desc&shard.url=http://127.0.0.1:46478//collection1&rows=10&version=2&q=*:*&NOW=1513936530650&isShard=true&wt=javabin} hits=68 status=0 QTime=0
[junit4] 2> 639766 INFO (qtp1254730362-5582) [ x:collection1] o.a.s.c.S.Request [collection1] webapp= path=/select params={q=*:*&df=text&distrib=false&shards.purpose=64&NOW=1513936530650&ids=1,101,135,147,115,17,149,8,108,10&isShard=true&shard.url=http://127.0.0.1:46478//collection1&wt=javabin&version=2} status=0 QTim
[...truncated too long message...]
updateStream = new UpdateStream(expression, factory);
[ecj-lint] ^^^^^^^^^^^^
[ecj-lint] Resource leak: 'updateStream' is never closed
[ecj-lint] ----------
[ecj-lint] 31. WARNING in /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionToExplanationTest.java (at line 224)
[ecj-lint] stream = new FacetStream(StreamExpressionParser.parse("facet("
[ecj-lint] + "collection1, "
[ecj-lint] + "q=\"*:*\", "
[ecj-lint] + "buckets=\"a_s\", "
[ecj-lint] + "bucketSorts=\"sum(a_i) asc\", "
[ecj-lint] + "bucketSizeLimit=100, "
[ecj-lint] + "sum(a_i), sum(a_f), "
[ecj-lint] + "min(a_i), min(a_f), "
[ecj-lint] + "max(a_i), max(a_f), "
[ecj-lint] + "avg(a_i), avg(a_f), "
[ecj-lint] + "count(*)"
[ecj-lint] + ")"), factory);
[ecj-lint] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[ecj-lint] Resource leak: 'stream' is never closed
[ecj-lint] ----------
[ecj-lint] 32. WARNING in /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionToExplanationTest.java (at line 250)
[ecj-lint] stream = new JDBCStream(StreamExpressionParser.parse("jdbc(connection=\"jdbc:hsqldb:mem:.\", sql=\"select PEOPLE.ID, PEOPLE.NAME, COUNTRIES.COUNTRY_NAME from PEOPLE inner join COUNTRIES on PEOPLE.COUNTRY_CODE = COUNTRIES.CODE order by PEOPLE.ID\", sort=\"ID asc\")"), factory);
[ecj-lint] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[ecj-lint] Resource leak: 'stream' is never closed
[ecj-lint] ----------
[ecj-lint] 33. WARNING in /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionToExplanationTest.java (at line 263)
[ecj-lint] stream = new IntersectStream(StreamExpressionParser.parse("intersect("
[ecj-lint] + "search(collection1, q=\"id:(0 3 4)\", fl=\"id,a_s,a_i,a_f\", sort=\"a_f asc, a_s asc\"),"
[ecj-lint] + "search(collection1, q=\"id:(1 2)\", fl=\"id,a_s,a_i,a_f\", sort=\"a_f asc, a_s asc\"),"
[ecj-lint] + "on=\"a_f, a_s\")"), factory);
[ecj-lint] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[ecj-lint] Resource leak: 'stream' is never closed
[ecj-lint] ----------
[ecj-lint] 34. WARNING in /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionToExplanationTest.java (at line 279)
[ecj-lint] stream = new ComplementStream(StreamExpressionParser.parse("complement("
[ecj-lint] + "search(collection1, q=\"id:(0 3 4)\", fl=\"id,a_s,a_i,a_f\", sort=\"a_f asc, a_s asc\"),"
[ecj-lint] + "search(collection1, q=\"id:(1 2)\", fl=\"id,a_s,a_i,a_f\", sort=\"a_f asc, a_s asc\"),"
[ecj-lint] + "on=\"a_f, a_s\")"), factory);
[ecj-lint] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[ecj-lint] Resource leak: 'stream' is never closed
[ecj-lint] ----------
[ecj-lint] ----------
[ecj-lint] 35. WARNING in /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamingTest.java (at line 159)
[ecj-lint] CloudSolrStream stream = new CloudSolrStream("", "collection1", sParams);
[ecj-lint] ^^^^^^
[ecj-lint] Resource leak: 'stream' is never closed
[ecj-lint] ----------
[ecj-lint] ----------
[ecj-lint] 36. WARNING in /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/solrj/src/test/org/apache/solr/client/solrj/response/NoOpResponseParserTest.java (at line 52)
[ecj-lint] return new SolrResourceLoader().openResource("solrj/sampleRangeFacetResponse.xml");
[ecj-lint] ^^^^^^^^^^^^^^^^^^^^^^^^
[ecj-lint] Resource leak: '<unassigned Closeable value>' is never closed
[ecj-lint] ----------
[ecj-lint] ----------
[ecj-lint] 37. WARNING in /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/solrj/src/test/org/apache/solr/client/solrj/response/QueryResponseTest.java (at line 46)
[ecj-lint] InputStream is = new SolrResourceLoader().openResource("solrj/sampleRangeFacetResponse.xml");
[ecj-lint] ^^^^^^^^^^^^^^^^^^^^^^^^
[ecj-lint] Resource leak: '<unassigned Closeable value>' is never closed
[ecj-lint] ----------
[ecj-lint] 38. WARNING in /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/solrj/src/test/org/apache/solr/client/solrj/response/QueryResponseTest.java (at line 103)
[ecj-lint] InputStream is = new SolrResourceLoader().openResource("solrj/sampleGroupResponse.xml");
[ecj-lint] ^^^^^^^^^^^^^^^^^^^^^^^^
[ecj-lint] Resource leak: '<unassigned Closeable value>' is never closed
[ecj-lint] ----------
[ecj-lint] 39. WARNING in /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/solrj/src/test/org/apache/solr/client/solrj/response/QueryResponseTest.java (at line 205)
[ecj-lint] InputStream is = new SolrResourceLoader().openResource("solrj/sampleSimpleGroupResponse.xml");
[ecj-lint] ^^^^^^^^^^^^^^^^^^^^^^^^
[ecj-lint] Resource leak: '<unassigned Closeable value>' is never closed
[ecj-lint] ----------
[ecj-lint] ----------
[ecj-lint] 40. WARNING in /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/solrj/src/test/org/apache/solr/client/solrj/response/TestClusteringResponse.java (at line 41)
[ecj-lint] InputStream is = new SolrResourceLoader().openResource("solrj/sampleClusteringResponse.xml");
[ecj-lint] ^^^^^^^^^^^^^^^^^^^^^^^^
[ecj-lint] Resource leak: '<unassigned Closeable value>' is never closed
[ecj-lint] ----------
[ecj-lint] ----------
[ecj-lint] 41. WARNING in /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/solrj/src/test/org/apache/solr/common/util/ContentStreamTest.java (at line 48)
[ecj-lint] InputStream is = new SolrResourceLoader().openResource( "solrj/README" );
[ecj-lint] ^^^^^^^^^^^^^^^^^^^^^^^^
[ecj-lint] Resource leak: '<unassigned Closeable value>' is never closed
[ecj-lint] ----------
[ecj-lint] 42. WARNING in /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/solrj/src/test/org/apache/solr/common/util/ContentStreamTest.java (at line 77)
[ecj-lint] InputStream is = new SolrResourceLoader().openResource( "solrj/README" );
[ecj-lint] ^^^^^^^^^^^^^^^^^^^^^^^^
[ecj-lint] Resource leak: '<unassigned Closeable value>' is never closed
[ecj-lint] ----------
[ecj-lint] 42 problems (1 error, 41 warnings)
BUILD FAILED
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/build.xml:835: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/build.xml:101: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build.xml:684: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/common-build.xml:2111: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/common-build.xml:2144: Compile failed; see the compiler error output for details.
Total time: 80 minutes 51 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Lucene-Solr-Tests-7.x - Build # 291 - Still Failing
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-7.x/291/
3 tests failed.
FAILED: org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testEventFromRestoredState
Error Message:
Stack Trace:
java.lang.AssertionError
at __randomizedtesting.SeedInfo.seed([1F4E1C5F6521068F:1F78A89F4C2CA2E5]:0)
at org.junit.Assert.fail(Assert.java:92)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertTrue(Assert.java:54)
at org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testEventFromRestoredState(TestTriggerIntegration.java:674)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testTriggerThrottling
Error Message:
Two TriggerAction instances should have been created by now
Stack Trace:
java.lang.AssertionError: Two TriggerAction instances should have been created by now
at __randomizedtesting.SeedInfo.seed([1F4E1C5F6521068F:E46CB47AB78BE51D]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testTriggerThrottling(TestTriggerIntegration.java:205)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testSearchRate
Error Message:
The trigger did not fire at all
Stack Trace:
java.lang.AssertionError: The trigger did not fire at all
at __randomizedtesting.SeedInfo.seed([1F4E1C5F6521068F:420602D6AAE7A0C0]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testSearchRate(TestTriggerIntegration.java:1178)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
Build Log:
[...truncated 12182 lines...]
[junit4] Suite: org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration
[junit4] 2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.sim.TestTriggerIntegration_1F4E1C5F6521068F-001/init-core-data-001
[junit4] 2> 1237361 WARN (SUITE-TestTriggerIntegration-seed#[1F4E1C5F6521068F]-worker) [ ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=4 numCloses=4
[junit4] 2> 1237361 INFO (SUITE-TestTriggerIntegration-seed#[1F4E1C5F6521068F]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
[junit4] 2> 1237415 INFO (SUITE-TestTriggerIntegration-seed#[1F4E1C5F6521068F]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
[junit4] 2> 1237416 INFO (SUITE-TestTriggerIntegration-seed#[1F4E1C5F6521068F]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
[junit4] 2> 1237432 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
[junit4] 2> 1237432 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 0
[junit4] 2> 1237432 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10008_solr, 127.0.0.1:10007_solr]
[junit4] 2> 1237432 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 0, lastZnodeVersion -1
[junit4] 2> 1237432 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 0
[junit4] 2> 1237452 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 1237452 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 0, lastZnodeVersion 0
[junit4] 2> 1237452 DEBUG (ScheduledTrigger-1485-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1237562 DEBUG (ScheduledTrigger-1485-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1237565 INFO (TEST-TestTriggerIntegration.testListeners-seed#[1F4E1C5F6521068F]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testListeners
[junit4] 2> 1237566 DEBUG (TEST-TestTriggerIntegration.testListeners-seed#[1F4E1C5F6521068F]) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
[junit4] 2> 1237566 INFO (TEST-TestTriggerIntegration.testListeners-seed#[1F4E1C5F6521068F]) [ ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
[junit4] 2> 1237578 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
[junit4] 2> 1237598 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
[junit4] 2> 1237598 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 1237599 WARN (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Interrupted
[junit4] 2> java.lang.InterruptedException
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220)
[junit4] 2> at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335)
[junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:154)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 1237599 DEBUG (TEST-TestTriggerIntegration.testListeners-seed#[1F4E1C5F6521068F]) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
[junit4] 2> 1237612 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
[junit4] 2> 1237627 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 2
[junit4] 2> 1237627 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10008_solr, 127.0.0.1:10007_solr]
[junit4] 2> 1237627 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion -1
[junit4] 2> 1237627 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
[junit4] 2> 1237647 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 1237647 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2
[junit4] 2> 1237661 DEBUG (simCloudManagerPool-1484-thread-1) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 1237662 DEBUG (simCloudManagerPool-1484-thread-1) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 3
[junit4] 2> 1237691 DEBUG (ScheduledTrigger-1487-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1237711 DEBUG (ScheduledTrigger-1487-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1237712 DEBUG (simCloudManagerPool-1484-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10008_solr, 127.0.0.1:10007_solr]
[junit4] 2> 1237715 DEBUG (simCloudManagerPool-1484-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10008_solr, 127.0.0.1:10007_solr]
[junit4] 2> 1237715 DEBUG (simCloudManagerPool-1484-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated with properties: {event=nodeAdded, waitFor=2, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerAction}, {name=test1, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestDummyAction}], enabled=true}
[junit4] 2> 1237736 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
[junit4] 2> 1237737 INFO (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
[junit4] 2> 1237756 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
[junit4] 2> 1237769 DEBUG (ScheduledTrigger-1487-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1237806 DEBUG (ScheduledTrigger-1487-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
[junit4] 2> 1237794 DEBUG (simCloudManagerPool-1484-thread-2) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 1237810 DEBUG (simCloudManagerPool-1484-thread-2) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 4
[junit4] 2> 1237810 DEBUG (simCloudManagerPool-1484-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10008_solr, 127.0.0.1:10007_solr]
[junit4] 2> 1237810 DEBUG (simCloudManagerPool-1484-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10008_solr, 127.0.0.1:10007_solr]
[junit4] 2> 1237810 DEBUG (simCloudManagerPool-1484-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated with properties: {event=nodeAdded, waitFor=2, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerAction}, {name=test1, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestDummyAction}], enabled=true}
[junit4] 2> 1237810 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 4
[junit4] 2> 1237810 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion 4
[junit4] 2> 1237909 DEBUG (ScheduledTrigger-1487-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1237909 DEBUG (ScheduledTrigger-1487-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1237909 DEBUG (ScheduledTrigger-1487-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
[junit4] 2> 1237924 DEBUG (simCloudManagerPool-1484-thread-3) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 1237924 DEBUG (simCloudManagerPool-1484-thread-3) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 5
[junit4] 2> 1237924 DEBUG (simCloudManagerPool-1484-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10008_solr, 127.0.0.1:10007_solr]
[junit4] 2> 1237924 DEBUG (simCloudManagerPool-1484-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10008_solr, 127.0.0.1:10007_solr]
[junit4] 2> 1237924 DEBUG (simCloudManagerPool-1484-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated with properties: {event=nodeAdded, waitFor=2, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerAction}, {name=test1, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestDummyAction}], enabled=true}
[junit4] 2> 1237931 DEBUG (ScheduledTrigger-1487-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 1237932 DEBUG (ScheduledTrigger-1487-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1237932 DEBUG (ScheduledTrigger-1487-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1237932 DEBUG (ScheduledTrigger-1487-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10009_solr at time 14046340574959915
[junit4] 2> 1237938 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 5
[junit4] 2> 1237938 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 5, lastZnodeVersion 5
[junit4] 2> 1237952 DEBUG (ScheduledTrigger-1487-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 1237952 DEBUG (ScheduledTrigger-1487-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1237952 DEBUG (ScheduledTrigger-1487-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1237980 DEBUG (ScheduledTrigger-1487-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 1237980 DEBUG (ScheduledTrigger-1487-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1237980 DEBUG (ScheduledTrigger-1487-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1237981 DEBUG (ScheduledTrigger-1487-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger firing registered processor for nodes: [127.0.0.1:10009_solr] added at times [14046340574959915], now=14046343020226115
[junit4] 2> 1237981 DEBUG (ScheduledTrigger-1487-thread-4) [ ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
[junit4] 2> "id":"31e712845d592bTd2gueqt0mxaatvirnw4stveon",
[junit4] 2> "source":"node_added_trigger",
[junit4] 2> "eventTime":14046340574959915,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[14046340574959915],
[junit4] 2> "nodeNames":["127.0.0.1:10009_solr"]}}
[junit4] 2> 1237984 DEBUG (AutoscalingActionExecutor-1488-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
[junit4] 2> "id":"31e712845d592bTd2gueqt0mxaatvirnw4stveon",
[junit4] 2> "source":"node_added_trigger",
[junit4] 2> "eventTime":14046340574959915,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[14046340574959915],
[junit4] 2> "_enqueue_time_":14046343037851765,
[junit4] 2> "nodeNames":["127.0.0.1:10009_solr"]}}
[junit4] 2> 1237999 DEBUG (AutoscalingActionExecutor-1488-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing took 723 ms for event id=31e712845d592bTd2gueqt0mxaatvirnw4stveon
[junit4] 2> 1238001 DEBUG (ScheduledTrigger-1487-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 1238004 DEBUG (ScheduledTrigger-1487-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1238004 DEBUG (ScheduledTrigger-1487-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1238021 DEBUG (ScheduledTrigger-1487-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 1238024 DEBUG (ScheduledTrigger-1487-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1238024 DEBUG (ScheduledTrigger-1487-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1238041 DEBUG (ScheduledTrigger-1487-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 1238045 DEBUG (ScheduledTrigger-1487-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1238045 DEBUG (ScheduledTrigger-1487-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1238070 DEBUG (ScheduledTrigger-1487-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 4
[junit4] 2> 1238070 DEBUG (ScheduledTrigger-1487-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1238070 DEBUG (ScheduledTrigger-1487-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
[junit4] 2> 1238070 DEBUG (ScheduledTrigger-1487-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10010_solr at time 14046347489671215
[junit4] 2> 1238090 DEBUG (ScheduledTrigger-1487-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 4
[junit4] 2> 1238091 DEBUG (ScheduledTrigger-1487-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1238091 DEBUG (ScheduledTrigger-1487-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
[junit4] 2> 1238110 DEBUG (ScheduledTrigger-1487-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 4
[junit4] 2> 1238119 DEBUG (ScheduledTrigger-1487-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1238119 DEBUG (ScheduledTrigger-1487-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
[junit4] 2> 1238119 DEBUG (ScheduledTrigger-1487-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger firing registered processor for nodes: [127.0.0.1:10010_solr] added at times [14046347489671215], now=14046349925255115
[junit4] 2> 1238119 DEBUG (ScheduledTrigger-1487-thread-4) [ ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
[junit4] 2> "id":"31e7142083782fTd2gueqt0mxaatvirnw4stveoq",
[junit4] 2> "source":"node_added_trigger",
[junit4] 2> "eventTime":14046347489671215,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[14046347489671215],
[junit4] 2> "nodeNames":["127.0.0.1:10010_solr"]}}
[junit4] 2> 1238120 DEBUG (AutoscalingActionExecutor-1488-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
[junit4] 2> "id":"31e7142083782fTd2gueqt0mxaatvirnw4stveoq",
[junit4] 2> "source":"node_added_trigger",
[junit4] 2> "eventTime":14046347489671215,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[14046347489671215],
[junit4] 2> "_enqueue_time_":14046349942536665,
[junit4] 2> "nodeNames":["127.0.0.1:10010_solr"]}}
[junit4] 2> 1238120 INFO (AutoscalingActionExecutor-1488-thread-1) [ ] o.a.s.c.ActionThrottle The last action attempt started 6147ms ago.
[junit4] 2> 1238121 WARN (AutoscalingActionExecutor-1488-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Exception executing actions
[junit4] 2> java.lang.Exception: Error executing action: test1 for trigger event: {
[junit4] 2> "id":"31e7142083782fTd2gueqt0mxaatvirnw4stveoq",
[junit4] 2> "source":"node_added_trigger",
[junit4] 2> "eventTime":14046347489671215,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[14046347489671215],
[junit4] 2> "_enqueue_time_":14046349942536665,
[junit4] 2> "nodeNames":["127.0.0.1:10010_solr"]}}
[junit4] 2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:330)
[junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[junit4] 2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> Caused by: java.lang.RuntimeException: failure
[junit4] 2> at org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestDummyAction.process(TestTriggerIntegration.java:890)
[junit4] 2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:327)
[junit4] 2> ... 6 more
[junit4] 2> 1238121 DEBUG (AutoscalingActionExecutor-1488-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing took 95 ms for event id=31e7142083782fTd2gueqt0mxaatvirnw4stveoq
[junit4] 2> 1238135 DEBUG (ScheduledTrigger-1487-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 4
[junit4] 2> 1238140 DEBUG (ScheduledTrigger-1487-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1238140 DEBUG (ScheduledTrigger-1487-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
[junit4] 2> 1238155 DEBUG (ScheduledTrigger-1487-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 4
[junit4] 2> 1238163 DEBUG (ScheduledTrigger-1487-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1238163 DEBUG (ScheduledTrigger-1487-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
[junit4] 2> 1238164 INFO (TEST-TestTriggerIntegration.testListeners-seed#[1F4E1C5F6521068F]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testListeners
[junit4] 2> 1238167 INFO (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[1F4E1C5F6521068F]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testNodeMarkersRegistration
[junit4] 2> 1238167 DEBUG (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[1F4E1C5F6521068F]) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 6
[junit4] 2> 1238167 INFO (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[1F4E1C5F6521068F]) [ ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
[junit4] 2> 1238167 WARN (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
[junit4] 2> 1238167 DEBUG (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[1F4E1C5F6521068F]) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
[junit4] 2> 1238237 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
[junit4] 2> 1238237 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 7
[junit4] 2> 1238237 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10010_solr, 127.0.0.1:10009_solr, 127.0.0.1:10008_solr, 127.0.0.1:10007_solr, 127.0.0.1:10011_solr]
[junit4] 2> 1238237 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 7, lastZnodeVersion -1
[junit4] 2> 1238237 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 7
[junit4] 2> 1238200 INFO (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[1F4E1C5F6521068F]) [ ] o.a.s.c.a.s.TestTriggerIntegration ====== KILL OVERSEER 1
[junit4] 2> 1238237 INFO (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[1F4E1C5F6521068F]) [ ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
[junit4] 2> 1238257 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 1238257 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 7, lastZnodeVersion 7
[junit4] 2> 1238294 DEBUG (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[1F4E1C5F6521068F]) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
[junit4] 2> 1238302 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 7
[junit4] 2> 1238302 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10010_solr, 127.0.0.1:10008_solr, 127.0.0.1:10007_solr, 127.0.0.1:10011_solr]
[junit4] 2> 1238302 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 7, lastZnodeVersion -1
[junit4] 2> 1238302 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 7
[junit4] 2> 1238302 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger Adding lost node from marker path: 127.0.0.1:10009_solr
[junit4] 2> 1238302 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 1238302 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 7, lastZnodeVersion 7
[junit4] 2> 1238302 DEBUG (ScheduledTrigger-1491-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 4
[junit4] 2> 1238323 DEBUG (ScheduledTrigger-1491-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 4
[junit4] 2> 1238346 WARN (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
[junit4] 2> 1238379 DEBUG (ScheduledTrigger-1491-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 4
[junit4] 2> 1238408 DEBUG (ScheduledTrigger-1491-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 4
[junit4] 2> 1238411 INFO (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[1F4E1C5F6521068F]) [ ] o.a.s.c.a.s.TestTriggerIntegration ====== ADD TRIGGERS
[junit4] 2> 1238436 DEBUG (ScheduledTrigger-1491-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 4
[junit4] 2> 1238450 DEBUG (simCloudManagerPool-1484-thread-8) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 1238450 DEBUG (simCloudManagerPool-1484-thread-8) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 8
[junit4] 2> 1238450 DEBUG (simCloudManagerPool-1484-thread-8) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10010_solr, 127.0.0.1:10008_solr, 127.0.0.1:10007_solr, 127.0.0.1:10011_solr]
[junit4] 2> 1238450 DEBUG (simCloudManagerPool-1484-thread-8) [ ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10010_solr, 127.0.0.1:10008_solr, 127.0.0.1:10007_solr, 127.0.0.1:10011_solr]
[junit4] 2> 1238450 DEBUG (simCloudManagerPool-1484-thread-8) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated with properties: {event=nodeAdded, waitFor=1, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestEventMarkerAction}], enabled=true}
[junit4] 2> 1238451 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 8
[junit4] 2> 1238451 INFO (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.s.TestTriggerIntegration TestEventMarkerAction init
[junit4] 2> 1238456 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 8, lastZnodeVersion 8
[junit4] 2> 1238456 DEBUG (ScheduledTrigger-1491-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1238456 DEBUG (ScheduledTrigger-1491-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
[junit4] 2> 1238456 DEBUG (ScheduledTrigger-1491-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 4
[junit4] 2> 1238457 DEBUG (simCloudManagerPool-1484-thread-9) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 1238457 DEBUG (simCloudManagerPool-1484-thread-9) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 9
[junit4] 2> 1238457 DEBUG (simCloudManagerPool-1484-thread-9) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10010_solr, 127.0.0.1:10008_solr, 127.0.0.1:10007_solr, 127.0.0.1:10011_solr]
[junit4] 2> 1238457 DEBUG (simCloudManagerPool-1484-thread-9) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10010_solr, 127.0.0.1:10008_solr, 127.0.0.1:10007_solr, 127.0.0.1:10011_solr]
[junit4] 2> 1238457 DEBUG (simCloudManagerPool-1484-thread-9) [ ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10010_solr, 127.0.0.1:10008_solr, 127.0.0.1:10007_solr, 127.0.0.1:10011_solr]
[junit4] 2> 1238457 DEBUG (simCloudManagerPool-1484-thread-9) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated with properties: {event=nodeAdded, waitFor=1, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestEventMarkerAction}], enabled=true}
[junit4] 2> 1238457 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 9
[junit4] 2> 1238458 INFO (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.s.TestTriggerIntegration TestEventMarkerAction init
[junit4] 2> 1238458 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 9, lastZnodeVersion 9
[junit4] 2> 1238458 DEBUG (ScheduledTrigger-1491-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 4
[junit4] 2> 1238458 INFO (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[1F4E1C5F6521068F]) [ ] o.a.s.c.a.s.TestTriggerIntegration ====== ADD NODE 1
[junit4] 2> 1238477 DEBUG (ScheduledTrigger-1491-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
[junit4] 2> 1238497 DEBUG (ScheduledTrigger-1491-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
[junit4] 2> 1238497 DEBUG (ScheduledTrigger-1491-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1238497 DEBUG (ScheduledTrigger-1491-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 5
[junit4] 2> 1238497 DEBUG (ScheduledTrigger-1491-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10012_solr at time 14046368854241865
[junit4] 2> 1238497 DEBUG (ScheduledTrigger-1491-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 5
[junit4] 2> 1238521 DEBUG (ScheduledTrigger-1491-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
[junit4] 2> 1238521 DEBUG (ScheduledTrigger-1491-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1238521 DEBUG (ScheduledTrigger-1491-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 5
[junit4] 2> 1238521 DEBUG (ScheduledTrigger-1491-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger firing registered processor for nodes: [127.0.0.1:10012_solr] added at times [14046368854241865], now=14046370065150315
[junit4] 2> 1238522 DEBUG (ScheduledTrigger-1491-thread-2) [ ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
[junit4] 2> "id":"31e71919f0f249Td2gueqt0mxaatvirnw4stveot",
[junit4] 2> "source":"node_added_trigger",
[junit4] 2> "eventTime":14046368854241865,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[14046368854241865],
[junit4] 2> "nodeNames":["127.0.0.1:10012_solr"]}}
[junit4] 2> 1238550 DEBUG (AutoscalingActionExecutor-1492-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
[junit4] 2> "id":"31e71919f0f249Td2gueqt0mxaatvirnw4stveot",
[junit4] 2> "source":"node_added_trigger",
[junit4] 2> "eventTime":14046368854241865,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[14046368854241865],
[junit4] 2> "_enqueue_time_":14046370072147165,
[junit4] 2> "nodeNames":["127.0.0.1:10012_solr"]}}
[junit4] 2> 1238566 DEBUG (AutoscalingActionExecutor-1492-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing took 810 ms for event id=31e71919f0f249Td2gueqt0mxaatvirnw4stveot
[junit4] 2> 1238574 DEBUG (ScheduledTrigger-1491-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
[junit4] 2> 1238574 DEBUG (ScheduledTrigger-1491-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1238575 DEBUG (ScheduledTrigger-1491-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 5
[junit4] 2> 1238575 DEBUG (ScheduledTrigger-1491-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 5
[junit4] 2> 1238575 INFO (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[1F4E1C5F6521068F]) [ ] o.a.s.c.a.s.TestTriggerIntegration ====== KILL OVERSEER 2
[junit4] 2> 1238575 INFO (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[1F4E1C5F6521068F]) [ ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
[junit4] 2> 1238575 WARN (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
[junit4] 2> 1238575 DEBUG (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[1F4E1C5F6521068F]) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
[junit4] 2> 1238598 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 9
[junit4] 2> 1238598 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10010_solr, 127.0.0.1:10008_solr, 127.0.0.1:10012_solr, 127.0.0.1:10007_solr]
[junit4] 2> 1238598 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10010_solr, 127.0.0.1:10008_solr, 127.0.0.1:10012_solr, 127.0.0.1:10007_solr]
[junit4] 2> 1238611 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10010_solr, 127.0.0.1:10008_solr, 127.0.0.1:10012_solr, 127.0.0.1:10007_solr]
[junit4] 2> 1238611 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated with properties: {event=nodeAdded, waitFor=1, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestEventMarkerAction}], enabled=true}
[junit4] 2> 1238611 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 9, lastZnodeVersion -1
[junit4] 2> 1238611 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 9
[junit4] 2> 1238613 INFO (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.s.TestTriggerIntegration TestEventMarkerAction init
[junit4] 2> 1238613 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger Adding lost node from marker path: 127.0.0.1:10011_solr
[junit4] 2> 1238615 INFO (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.s.TestTriggerIntegration TestEventMarkerAction init
[junit4] 2> 1238623 DEBUG (ScheduledTrigger-1493-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 4
[junit4] 2> 1238623 DEBUG (ScheduledTrigger-1493-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10011_solr
[junit4] 2> 1238632 DEBUG (ScheduledTrigger-1493-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1238632 DEBUG (ScheduledTrigger-1493-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
[junit4] 2> 1238632 DEBUG (ScheduledTrigger-1493-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 4
[junit4] 2> 1238632 DEBUG (ScheduledTrigger-1493-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10011_solr
[junit4] 2> 1238632 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 9, lastZnodeVersion 9
[junit4] 2> 1238652 DEBUG (ScheduledTrigger-1493-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 4
[junit4] 2> 1238652 DEBUG (ScheduledTrigger-1493-thread-2) [ ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger firing registered processor for lost nodes: [127.0.0.1:10011_solr]
[junit4] 2> 1238652 DEBUG (ScheduledTrigger-1493-thread-2) [ ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
[junit4] 2> "id":"31e71a91ae99d9Td2gueqt0mxaatvirnw4stveow",
[junit4] 2> "source":"node_lost_trigger",
[junit4] 2> "eventTime":14046375158127065,
[junit4] 2> "eventType":"NODELOST",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[14046375158127065],
[junit4] 2> "nodeNames":["127.0.0.1:10011_solr"]}}
[junit4] 2> 1238653 DEBUG (ScheduledTrigger-1493-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1238653 DEBUG (ScheduledTrigger-1493-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
[junit4] 2> 1238689 DEBUG (ScheduledTrigger-1493-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 4
[junit4] 2> 1238698 DEBUG (AutoscalingActionExecutor-1494-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
[junit4] 2> "id":"31e71a91ae99d9Td2gueqt0mxaatvirnw4stveow",
[junit4] 2> "source":"node_lost_trigger",
[junit4] 2> "eventTime":14046375158127065,
[junit4] 2> "eventType":"NODELOST",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[14046375158127065],
[junit4] 2> "_enqueue_time_":14046376611006715,
[junit4] 2> "nodeNames":["127.0.0.1:10011_solr"]}}
[junit4] 2> 1238699 INFO (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[1F4E1C5F6521068F]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testNodeMarkersRegistration
[junit4] 2> 1238742 DEBUG (AutoscalingActionExecutor-1494-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing took 2185 ms for event id=31e71a91ae99d9Td2gueqt0mxaatvirnw4stveow
[junit4] 2> 1238744 INFO (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[1F4E1C5F6521068F]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testNodeAddedTrigger
[junit4] 2> 1238744 DEBUG (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[1F4E1C5F6521068F]) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 10
[junit4] 2> 1238744 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 10
[junit4] 2> 1238744 INFO (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[1F4E1C5F6521068F]) [ ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
[junit4] 2> 1238745 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
[junit4] 2> 1238745 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 1238745 DEBUG (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[1F4E1C5F6521068F]) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
[junit4] 2> 1238745 WARN (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Interrupted
[junit4] 2> java.lang.InterruptedException
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220)
[junit4] 2> at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335)
[junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:154)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 1238778 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
[junit4] 2> 1238778 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 11
[junit4] 2> 1238779 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10010_solr, 127.0.0.1:10008_solr, 127.0.0.1:10012_solr, 127.0.0.1:10007_solr]
[junit4] 2> 1238779 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 11, lastZnodeVersion -1
[junit4] 2> 1238779 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 11
[junit4] 2> 1238779 DEBUG (simCloudManagerPool-1484-thread-14) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 1238779 DEBUG (simCloudManagerPool-1484-thread-14) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 12
[junit4] 2> 1238784 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 1238811 DEBUG (simCloudManagerPool-1484-thread-14) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10010_solr, 127.0.0.1:10008_solr, 127.0.0.1:10012_solr, 127.0.0.1:10007_solr]
[junit4] 2> 1238811 DEBUG (simCloudManagerPool-1484-thread-14) [ ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10010_solr, 127.0.0.1:10008_solr, 127.0.0.1:10012_solr, 127.0.0.1:10007_solr]
[junit4] 2> 1238811 DEBUG (simCloudManagerPool-1484-thread-14) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated with properties: {event=nodeAdded, waitFor=3, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerAction}], enabled=true}
[junit4] 2> 1238811 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 12, lastZnodeVersion 11
[junit4] 2> 1238811 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 12
[junit4] 2> 1238811 INFO (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
[junit4] 2> 1238827 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 12, lastZnodeVersion 12
[junit4] 2> 1238827 DEBUG (ScheduledTrigger-1495-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
[junit4] 2> 1238827 DEBUG (ScheduledTrigger-1495-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1238827 DEBUG (ScheduledTrigger-1495-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 5
[junit4] 2> 1238827 DEBUG (ScheduledTrigger-1495-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10013_solr at time 14046385339987865
[junit4] 2> 1238852 DEBUG (ScheduledTrigger-1495-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
[junit4] 2> 1238852 DEBUG (ScheduledTrigger-1495-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1238852 DEBUG (ScheduledTrigger-1495-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 5
[junit4] 2> 1238884 DEBUG (ScheduledTrigger-1495-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
[junit4] 2> 1238884 DEBUG (ScheduledTrigger-1495-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1238884 DEBUG (ScheduledTrigger-1495-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 5
[junit4] 2> 1238904 DEBUG (ScheduledTrigger-1495-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
[junit4] 2> 1238904 DEBUG (ScheduledTrigger-1495-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1238904 DEBUG (ScheduledTrigger-1495-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 5
[junit4] 2> 1238904 DEBUG (ScheduledTrigger-1495-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger firing registered processor for nodes: [127.0.0.1:10013_solr] added at times [14046385339987865], now=14046389198907365
[junit4] 2> 1238904 DEBUG (ScheduledTrigger-1495-thread-3) [ ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
[junit4] 2> "id":"31e71cf0917799Td2gueqt0mxaatvirnw4stveoz",
[junit4] 2> "source":"node_added_trigger",
[junit4] 2> "eventTime":14046385339987865,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[14046385339987865],
[junit4] 2> "nodeNames":["127.0.0.1:10013_solr"]}}
[junit4] 2> 1238906 DEBUG (AutoscalingActionExecutor-1496-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
[junit4] 2> "id":"31e71cf0917799Td2gueqt0mxaatvirnw4stveoz",
[junit4] 2> "source":"node_added_trigger",
[junit4] 2> "eventTime":14046385339987865,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[14046385339987865],
[junit4] 2> "_enqueue_time_":14046389209028065,
[junit4] 2> "nodeNames":["127.0.0.1:10013_solr"]}}
[junit4] 2> 1238930 DEBUG (AutoscalingActionExecutor-1496-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing took 1212 ms for event id=31e71cf0917799Td2gueqt0mxaatvirnw4stveoz
[junit4] 2> 1238935 DEBUG (simCloudManagerPool-1484-thread-17) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 1238935 DEBUG (simCloudManagerPool-1484-thread-17) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 13
[junit4] 2> 1238935 DEBUG (simCloudManagerPool-1484-thread-17) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10010_solr, 127.0.0.1:10013_solr, 127.0.0.1:10008_solr, 127.0.0.1:10012_solr, 127.0.0.1:10007_solr]
[junit4] 2> 1238935 DEBUG (simCloudManagerPool-1484-thread-17) [ ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10010_solr, 127.0.0.1:10013_solr, 127.0.0.1:10008_solr, 127.0.0.1:10012_solr, 127.0.0.1:10007_solr]
[junit4] 2> 1238935 DEBUG (simCloudManagerPool-1484-thread-17) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated with properties: {event=nodeAdded, waitFor=3, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerAction}], enabled=true}
[junit4] 2> 1238935 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 13
[junit4] 2> 1238936 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 13, lastZnodeVersion 13
[junit4] 2> 1238949 DEBUG (ScheduledTrigger-1495-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
[junit4] 2> 1238949 DEBUG (ScheduledTrigger-1495-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1238949 DEBUG (ScheduledTrigger-1495-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 5
[junit4] 2> 1238969 DEBUG (ScheduledTrigger-1495-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
[junit4] 2> 1238981 INFO (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[1F4E1C5F6521068F]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testNodeAddedTrigger
[junit4] 2> 1238981 DEBUG (ScheduledTrigger-1495-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1238982 DEBUG (ScheduledTrigger-1495-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 5
[junit4] 2> 1239010 DEBUG (ScheduledTrigger-1495-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1239010 DEBUG (ScheduledTrigger-1495-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 5
[junit4] 2> 1239010 DEBUG (ScheduledTrigger-1495-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
[junit4] 2> 1239034 INFO (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[1F4E1C5F6521068F]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testNodeAddedTriggerRestoreState
[junit4] 2> 1239034 DEBUG (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[1F4E1C5F6521068F]) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 14
[junit4] 2> 1239034 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 14
[junit4] 2> 1239034 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
[junit4] 2> 1239034 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 1239034 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 14, lastZnodeVersion 14
[junit4] 2> 1239034 INFO (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[1F4E1C5F6521068F]) [ ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
[junit4] 2> 1239035 WARN (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Interrupted
[junit4] 2> java.lang.InterruptedException
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048)
[junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:166)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 1239035 DEBUG (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[1F4E1C5F6521068F]) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
[junit4] 2> 1239107 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
[junit4] 2> 1239108 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 15
[junit4] 2> 1239108 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10010_solr, 127.0.0.1:10013_solr, 127.0.0.1:10008_solr, 127.0.0.1:10012_solr, 127.0.0.1:10007_solr]
[junit4] 2> 1239108 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 15, lastZnodeVersion -1
[junit4] 2> 1239108 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 15
[junit4] 2> 1239128 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 1239128 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 15, lastZnodeVersion 15
[junit4] 2> 1239128 DEBUG (ScheduledTrigger-1497-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
[junit4] 2> 1239165 DEBUG (simCloudManagerPool-1484-thread-18) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 1239165 DEBUG (simCloudManagerPool-1484-thread-18) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 16
[junit4] 2> 1239180 DEBUG (ScheduledTrigger-1497-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 5
[junit4] 2> 1239201 DEBUG (simCloudManagerPool-1484-thread-18) [ ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10010_solr, 127.0.0.1:10013_solr, 127.0.0.1:10008_solr, 127.0.0.1:10012_solr, 127.0.0.1:10007_solr]
[junit4] 2> 1239201 DEBUG (simCloudManagerPool-1484-thread-18) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_restore_trigger instantiated with properties: {event=nodeAdded, waitFor=5, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerAction}], enabled=true}
[junit4] 2> 1239201 DEBUG (simCloudManagerPool-1484-thread-18) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10010_solr, 127.0.0.1:10013_solr, 127.0.0.1:10008_solr, 127.0.0.1:10012_solr, 127.0.0.1:10007_solr]
[junit4] 2> 1239201 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 16
[junit4] 2> 1239201 INFO (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
[junit4] 2> 1239202 DEBUG (ScheduledTrigger-1497-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 6
[junit4] 2> 1239202 DEBUG (ScheduledTrigger-1497-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_restore_trigger
[junit4] 2> 1239202 DEBUG (ScheduledTrigger-1497-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 6
[junit4] 2> 1239202 DEBUG (ScheduledTrigger-1497-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10014_solr at time 14046404093854365
[junit4] 2> 1239217 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 16, lastZnodeVersion 16
[junit4] 2> 1239225 DEBUG (ScheduledTrigger-1497-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 6
[junit4] 2> 1239225 DEBUG (ScheduledTrigger-1497-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_restore_trigger
[junit4] 2> 1239225 DEBUG (ScheduledTrigger-1497-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 6
[junit4] 2> 1239262 DEBUG (ScheduledTrigger-1497-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 6
[junit4] 2> 1239262 DEBUG (ScheduledTrigger-1497-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_restore_trigger
[junit4] 2> 1239262 DEBUG (ScheduledTrigger-1497-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 6
[junit4] 2> 1239263 DEBUG (simCloudManagerPool-1484-thread-19) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 1239263 DEBUG (simCloudManagerPool-1484-thread-19) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 17
[junit4] 2> 1239263 DEBUG (simCloudManagerPool-1484-thread-19) [ ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10010_solr, 127.0.0.1:10013_solr, 127.0.0.1:10008_solr, 127.0.0.1:10012_solr, 127.0.0.1:10014_solr, 127.0.0.1:10007_solr]
[junit4] 2> 1239263 DEBUG (simCloudManagerPool-1484-thread-19) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_restore_trigger instantiated with properties: {event=nodeAdded, waitFor=0, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerAction}], enabled=true}
[junit4] 2> 1239263 DEBUG (simCloudManagerPool-1484-thread-19) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10010_solr, 127.0.0.1:10013_solr, 127.0.0.1:10008_solr, 127.0.0.1:10012_solr, 127.0.0.1:10014_solr, 127.0.0.1:10007_solr]
[junit4] 2> 1239274 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 17
[junit4] 2> 1239274 INFO (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
[junit4] 2> 1239275 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 17, lastZnodeVersion 17
[junit4] 2> 1239275 DEBUG (ScheduledTrigger-1497-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_restore_trigger
[junit4] 2> 1239275 DEBUG (ScheduledTrigger-1497-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 6
[junit4] 2> 1239275 DEBUG (ScheduledTrigger-1497-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_restore_trigger firing registered processor for nodes: [127.0.0.1:10014_solr] added at times [14046404093854365], now=14046407729598315
[junit4] 2> 1239275 DEBUG (ScheduledTrigger-1497-thread-3) [ ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
[junit4] 2> "id":"31e7214e62c29dTd2gueqt0mxaatvirnw4stvep2",
[junit4] 2> "source":"node_added_restore_trigger",
[junit4] 2> "eventTime":14046404093854365,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[14046404093854365],
[junit4] 2> "nodeNames":["127.0.0.1:10014_solr"]}}
[junit4] 2> 1239315 DEBUG (AutoscalingActionExecutor-1498-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
[junit4] 2> "id":"31e7214e62c29dTd2gueqt0mxaatvirnw4stvep2",
[junit4] 2> "source":"node_added_restore_trigger",
[junit4] 2> "eventTime":14046404093854365,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[14046404093854365],
[junit4] 2> "_enqueue_time_":14046407736168815,
[junit4] 2> "nodeNames":["127.0.0.1:10014_solr"]}}
[junit4] 2> 1239315 INFO (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[1F4E1C5F6521068F]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testNodeAddedTriggerRestoreState
[junit4] 2> 1239400 DEBUG (AutoscalingActionExecutor-1498-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing took 4272 ms for event id=31e7214e62c29dTd2gueqt0mxaatvirnw4stvep2
[junit4] 2> 1239412 DEBUG (ScheduledTrigger-1497-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_restore_trigger
[junit4] 2> 1239412 DEBUG (ScheduledTrigger-1497-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 6
[junit4] 2> 1239412 DEBUG (ScheduledTrigger-1497-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 6
[junit4] 2> 1239437 DEBUG (ScheduledTrigger-1497-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 6
[junit4] 2> 1239457 DEBUG (ScheduledTrigger-1497-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 6
[junit4] 2> 1239461 DEBUG (ScheduledTrigger-1497-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_restore_trigger
[junit4] 2> 1239461 DEBUG (ScheduledTrigger-1497-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 6
[junit4] 2> 1239463 INFO (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[1F4E1C5F6521068F]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testTriggerThrottling
[junit4] 2> 1239463 DEBUG (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[1F4E1C5F6521068F]) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 18
[junit4] 2> 1239463 INFO (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[1F4E1C5F6521068F]) [ ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
[junit4] 2> 1239463 WARN (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
[junit4] 2> 1239463 DEBUG (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[1F4E1C5F6521068F]) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
[junit4] 2> 1239495 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
[junit4] 2> 1239495 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 19
[junit4] 2> 1239495 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10010_solr, 127.0.0.1:10013_solr, 127.0.0.1:10008_solr, 127.0.0.1:10012_solr, 127.0.0.1:10014_solr, 127.0.0.1:10007_solr]
[junit4] 2> 1239495 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 19, lastZnodeVersion -1
[junit4] 2> 1239495 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 19
[junit4] 2> 1239503 DEBUG (simCloudManagerPool-1484-thread-22) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 1239503 DEBUG (simCloudManagerPool-1484-thread-22) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 20
[junit4] 2> 1239503 DEBUG (simCloudManagerPool-1484-thread-22) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10010_solr, 127.0.0.1:10013_solr, 127.0.0.1:10008_solr, 127.0.0.1:10012_solr, 127.0.0.1:10014_solr, 127.0.0.1:10007_solr]
[junit4] 2> 1239503 DEBUG (simCloudManagerPool-1484-thread-22) [ ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10010_solr, 127.0.0.1:10013_solr, 127.0.0.1:10008_solr, 127.0.0.1:10012_solr, 127.0.0.1:10014_solr, 127.0.0.1:10007_solr]
[junit4] 2> 1239503 DEBUG (simCloudManagerPool-1484-thread-22) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 instantiated with properties: {event=nodeAdded, waitFor=0, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$ThrottlingTesterAction}], enabled=true}
[junit4] 2> 1239539 DEBUG (simCloudManagerPool-1484-thread-23) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 1239539 DEBUG (simCloudManagerPool-1484-thread-23) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 21
[junit4] 2> 1239539 DEBUG (simCloudManagerPool-1484-thread-23) [ ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10010_solr, 127.0.0.1:10013_solr, 127.0.0.1:10008_solr, 127.0.0.1:10012_solr, 127.0.0.1:10014_solr, 127.0.0.1:10007_solr]
[junit4] 2> 1239539 DEBUG (simCloudManagerPool-1484-thread-23) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger2 instantiated with properties: {event=nodeAdded, waitFor=0, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$ThrottlingTesterAction}], enabled=true}
[junit4] 2> 1239539 DEBUG (simCloudManagerPool-1484-thread-23) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10010_solr, 127.0.0.1:10013_solr, 127.0.0.1:10008_solr, 127.0.0.1:10012_solr, 127.0.0.1:10014_solr, 127.0.0.1:10007_solr]
[junit4] 2> 1239539 DEBUG (simCloudManagerPool-1484-thread-23) [ ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10010_solr, 127.0.0.1:10013_solr, 127.0.0.1:10008_solr, 127.0.0.1:10012_solr, 127.0.0.1:10014_solr, 127.0.0.1:10007_solr]
[junit4] 2> 1239539 DEBUG (simCloudManagerPool-1484-thread-23) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 instantiated with properties: {event=nodeAdded, waitFor=0, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$ThrottlingTesterAction}], enabled=true}
[junit4] 2> 1239540 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 1239540 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 21, lastZnodeVersion 19
[junit4] 2> 1239540 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 21
[junit4] 2> 1239540 INFO (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
[junit4] 2> 1239540 DEBUG (ScheduledTrigger-1499-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 6
[junit4] 2> 1239542 INFO (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
[junit4] 2> 1239543 DEBUG (ScheduledTrigger-1499-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
[junit4] 2> 1239543 DEBUG (ScheduledTrigger-1499-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 6
[junit4] 2> 1239543 DEBUG (ScheduledTrigger-1499-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
[junit4] 2> 1239543 DEBUG (ScheduledTrigger-1499-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 6
[junit4] 2> 1239546 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 21, lastZnodeVersion 21
[junit4] 2> 1239576 DEBUG (ScheduledTrigger-1499-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 7
[junit4] 2> 1239576 DEBUG (ScheduledTrigger-1499-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
[junit4] 2> 1239576 DEBUG (ScheduledTrigger-1499-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 7
[junit4] 2> 1239576 DEBUG (ScheduledTrigger-1499-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10015_solr at time 14046422780481915
[junit4] 2> 1239576 DEBUG (ScheduledTrigger-1499-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger2 firing registered processor for nodes: [127.0.0.1:10015_solr] added at times [14046422780481915], now=14046422782140465
[junit4] 2> 1239576 DEBUG (ScheduledTrigger-1499-thread-2) [ ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
[junit4] 2> "id":"31e725a832117bTd2gueqt0mxaatvirnw4stvep5",
[junit4] 2> "source":"node_added_trigger2",
[junit4] 2> "eventTime":14046422780481915,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[14046422780481915],
[junit4] 2> "nodeNames":["127.0.0.1:10015_solr"]}}
[junit4] 2> 1239616 DEBUG (AutoscalingActionExecutor-1500-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
[junit4] 2> "id":"31e725a832117bTd2gueqt0mxaatvirnw4stvep5",
[junit4] 2> "source":"node_added_trigger2",
[junit4] 2> "eventTime":14046422780481915,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[14046422780481915],
[junit4] 2> "_enqueue_time_":14046422789327765,
[junit4] 2> "nodeNames":["127.0.0.1:10015_solr"]}}
[junit4] 2> 1239616 INFO (AutoscalingActionExecutor-1500-thread-1) [ ] o.a.s.c.a.s.TestTriggerIntegration action executed from node_added_trigger2
[junit4] 2> 1239632 DEBUG (AutoscalingActionExecutor-1500-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing took 817 ms for event id=31e725a832117bTd2gueqt0mxaatvirnw4stvep5
[junit4] 2> 1239669 DEBUG (ScheduledTrigger-1499-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 7
[junit4] 2> 1239669 DEBUG (ScheduledTrigger-1499-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
[junit4] 2> 1239669 DEBUG (ScheduledTrigger-1499-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 7
[junit4] 2> 1239669 DEBUG (ScheduledTrigger-1499-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10015_solr at time 14046427447293415
[junit4] 2> 1239669 DEBUG (ScheduledTrigger-1499-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 firing registered processor for nodes: [127.0.0.1:10015_solr] added at times [14046427447293415], now=14046427449074715
[junit4] 2> 1239669 DEBUG (ScheduledTrigger-1499-thread-3) [ ] o.a.s.c.a.ScheduledTriggers -------- Cooldown period - rejecting event: {
[junit4] 2> "id":"31e726be5bf5e7Td2gueqt0mxaatvirnw4stvep8",
[junit4] 2> "source":"node_added_trigger1",
[junit4] 2> "eventTime":14046427447293415,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[14046427447293415],
[junit4] 2> "nodeNames":["127.0.0.1:10015_solr"]}}
[junit4] 2> 1239701 DEBUG (ScheduledTrigger-1499-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 7
[junit4] 2> 1239738 DEBUG (ScheduledTrigger-1499-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 7
[junit4] 2> 1239738 DEBUG (ScheduledTrigger-1499-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
[junit4] 2> 1239738 DEBUG (ScheduledTrigger-1499-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 7
[junit4] 2> 1239760 DEBUG (ScheduledTrigger-1499-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 7
[junit4] 2> 1239760 DEBUG (ScheduledTrigger-1499-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
[junit4] 2> 1239760 DEBUG (ScheduledTrigger-1499-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 7
[junit4] 2> 1239760 DEBUG (ScheduledTrigger-1499-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 firing registered processor for nodes: [127.0.0.1:10015_solr] added at times [14046427447293415], now=14046432002871215
[junit4] 2> 1239760 DEBUG (ScheduledTrigger-1499-thread-2) [ ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
[junit4] 2> "id":"31e726be5bf5e7Td2gueqt0mxaatvirnw4stvepa",
[junit4] 2> "source":"node_added_trigger1",
[junit4] 2> "eventTime":14046427447293415,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[14046427447293415],
[junit4] 2> "nodeNames":["127.0.0.1:10015_solr"]}}
[junit4] 2> 1239777 DEBUG (AutoscalingActionExecutor-1500-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
[junit4] 2> "id":"31e726be5bf5e7Td2gueqt0mxaatvirnw4stvepa",
[junit4] 2> "source":"node_added_trigger1",
[junit4] 2> "eventTime":14046427447293415,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[14046427447293415],
[junit4] 2> "_enqueue_time_":14046432010901865,
[junit4] 2> "nodeNames":["127.0.0.1:10015_solr"]}}
[junit4] 2> 1239777 INFO (AutoscalingActionExecutor-1500-thread-1) [ ] o.a.s.c.ActionThrottle The last action attempt started 8070ms ago.
[junit4] 2> 1239777 INFO (AutoscalingActionExecutor-1500-thread-1) [ ] o.a.s.c.a.s.TestTriggerIntegration last action at 14046424793308965 time = 14046432863020265
[junit4] 2> 1239777 INFO (AutoscalingActionExecutor-1500-thread-1) [ ] o.a.s.c.a.s.TestTriggerIntegration action executed from node_added_trigger1
[junit4] 2> 1239795 DEBUG (AutoscalingActionExecutor-1500-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing took 884 ms for event id=31e726be5bf5e7Td2gueqt0mxaatvirnw4stvepa
[junit4] 2> 1239815 DEBUG (ScheduledTrigger-1499-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 7
[junit4] 2> 1239815 DEBUG (ScheduledTrigger-1499-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
[junit4] 2> 1239815 DEBUG (ScheduledTrigger-1499-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 7
[junit4] 2> 1239815 DEBUG (ScheduledTrigger-1499-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
[junit4] 2> 1239835 DEBUG (ScheduledTrigger-1499-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
[junit4] 2> 1239852 DEBUG (ScheduledTrigger-1499-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 7
[junit4] 2> 1239835 DEBUG (ScheduledTrigger-1499-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 7
[junit4] 2> 1239852 DEBUG (ScheduledTrigger-1499-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 7
[junit4] 2> 1239824 DEBUG (simCloudManagerPool-1484-thread-29) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 1239860 DEBUG (simCloudManagerPool-1484-thread-29) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 22
[junit4] 2> 1239860 DEBUG (simCloudManagerPool-1484-thread-29) [ ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10010_solr, 127.0.0.1:10013_solr, 127.0.0.1:10008_solr, 127.0.0.1:10012_solr, 127.0.0.1:10014_solr, 127.0.0.1:10007_solr, 127.0.0.1:10015_solr]
[junit4] 2> 1239860 DEBUG (simCloudManagerPool-1484-thread-29) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger2 instantiated with properties: {event=nodeAdded, waitFor=0, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$ThrottlingTesterAction}], enabled=true}
[junit4] 2> 1239860 DEBUG (simCloudManagerPool-1484-thread-29) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10010_solr, 127.0.0.1:10013_solr, 127.0.0.1:10008_solr, 127.0.0.1:10012_solr, 127.0.0.1:10014_solr, 127.0.0.1:10007_solr, 127.0.0.1:10015_solr]
[junit4] 2> 1239860 DEBUG (simCloudManagerPool-1484-thread-29) [ ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10010_solr, 127.0.0.1:10013_solr, 127.0.0.1:10008_solr, 127.0.0.1:10012_solr, 127.0.0.1:10014_solr, 127.0.0.1:10007_solr, 127.0.0.1:10015_solr]
[junit4] 2> 1239860 DEBUG (simCloudManagerPool-1484-thread-29) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 instantiated with properties: {event=nodeAdded, waitFor=0, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$ThrottlingTesterAction}], enabled=true}
[junit4] 2> 1239860 DEBUG (simCloudManagerPool-1484-thread-29) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0
[...truncated too long message...]
ulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 44
[junit4] 2> 1244558 DEBUG (AutoscalingActionExecutor-1512-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
[junit4] 2> "id":"31e7574867cbb9Td2gueqt0mxaatvirnw4stveq7",
[junit4] 2> "source":"search_rate_trigger",
[junit4] 2> "eventTime":14046635921755065,
[junit4] 2> "eventType":"SEARCHRATE",
[junit4] 2> "properties":{
[junit4] 2> "node":{
[junit4] 2> "127.0.0.1:10018_solr":250.0,
[junit4] 2> "127.0.0.1:10014_solr":250.0},
[junit4] 2> "replica":[
[junit4] 2> "{\"core_node1\":{\n \"core\":\"collection1_shard1_replica_n1\",\n \"INDEX.sizeInBytes\":123450000,\n \"rate\":250.0,\n \"node_name\":\"127.0.0.1:10018_solr\",\n \"state\":\"active\",\n \"type\":\"NRT\",\n \"QUERY./select.requestTimes:1minRate\":250.0,\n \"shard\":\"shard1\",\n \"collection\":\"collection1\"}}",
[junit4] 2> "{\"core_node2\":{\n \"core\":\"collection1_shard1_replica_n2\",\n \"leader\":\"true\",\n \"INDEX.sizeInBytes\":123450000,\n \"rate\":250.0,\n \"node_name\":\"127.0.0.1:10014_solr\",\n \"state\":\"active\",\n \"type\":\"NRT\",\n \"QUERY./select.requestTimes:1minRate\":250.0,\n \"shard\":\"shard1\",\n \"collection\":\"collection1\"}}"],
[junit4] 2> "collection":{"collection1":500.0},
[junit4] 2> "shard":{"collection1":{"shard1":500.0}},
[junit4] 2> "_enqueue_time_":14046657699804065}}
[junit4] 2> 1244558 INFO (AutoscalingActionExecutor-1512-thread-1) [ ] o.a.s.c.ActionThrottle The last action attempt started 35077ms ago.
[junit4] 2> 1244558 WARN (AutoscalingActionExecutor-1512-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Exception executing actions
[junit4] 2> org.apache.lucene.store.AlreadyClosedException: ScheduledTrigger search_rate_trigger has been closed.
[junit4] 2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers$ScheduledTrigger.dequeue(ScheduledTriggers.java:525)
[junit4] 2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:337)
[junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[junit4] 2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 1244558 DEBUG (AutoscalingActionExecutor-1512-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing took 11991 ms for event id=31e7574867cbb9Td2gueqt0mxaatvirnw4stveq7
[junit4] 2> 1244566 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 1244567 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 44, lastZnodeVersion 44
[junit4] 2> 1244567 DEBUG (ScheduledTrigger-1513-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 8
[junit4] 2> 1244591 DEBUG (simCloudManagerPool-1484-thread-64) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 1244591 DEBUG (simCloudManagerPool-1484-thread-64) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 45
[junit4] 2> 1244608 DEBUG (ScheduledTrigger-1513-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 8
[junit4] 2> 1244608 DEBUG (simCloudManagerPool-1484-thread-64) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10018_solr, 127.0.0.1:10010_solr, 127.0.0.1:10016_solr, 127.0.0.1:10017_solr, 127.0.0.1:10008_solr, 127.0.0.1:10012_solr, 127.0.0.1:10014_solr, 127.0.0.1:10007_solr]
[junit4] 2> 1244609 DEBUG (simCloudManagerPool-1484-thread-64) [ ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10018_solr, 127.0.0.1:10010_solr, 127.0.0.1:10016_solr, 127.0.0.1:10017_solr, 127.0.0.1:10008_solr, 127.0.0.1:10012_solr, 127.0.0.1:10014_solr, 127.0.0.1:10007_solr]
[junit4] 2> 1244609 DEBUG (simCloudManagerPool-1484-thread-64) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated with properties: {event=nodeAdded, waitFor=10, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerAction}], enabled=true}
[junit4] 2> 1244644 DEBUG (ScheduledTrigger-1513-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 8
[junit4] 2> 1244644 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 45
[junit4] 2> 1244644 INFO (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
[junit4] 2> 1244644 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 45, lastZnodeVersion 45
[junit4] 2> 1244644 INFO (TEST-TestTriggerIntegration.testEventFromRestoredState-seed#[1F4E1C5F6521068F]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testEventFromRestoredState
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestTriggerIntegration -Dtests.method=testEventFromRestoredState -Dtests.seed=1F4E1C5F6521068F -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=es-PA -Dtests.timezone=Etc/GMT+1 -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
[junit4] FAILURE 1.06s J1 | TestTriggerIntegration.testEventFromRestoredState <<<
[junit4] > Throwable #1: java.lang.AssertionError
[junit4] > at __randomizedtesting.SeedInfo.seed([1F4E1C5F6521068F:1F78A89F4C2CA2E5]:0)
[junit4] > at org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testEventFromRestoredState(TestTriggerIntegration.java:674)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 1244645 DEBUG (ScheduledTrigger-1513-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1244645 DEBUG (ScheduledTrigger-1513-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 9
[junit4] 2> 1244645 DEBUG (ScheduledTrigger-1513-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10020_solr at time 14046676257320515
[junit4] 2> 1244681 DEBUG (SUITE-TestTriggerIntegration-seed#[1F4E1C5F6521068F]-worker) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
[junit4] 2> 1244717 WARN (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
[junit4] 2> NOTE: leaving temporary files on disk at: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.sim.TestTriggerIntegration_1F4E1C5F6521068F-001
[junit4] 2> NOTE: test params are: codec=FastCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST, chunkSize=8, maxDocsPerChunk=905, blockSize=75), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST, chunkSize=8, blockSize=75)), sim=RandomSimilarity(queryNorm=false): {}, locale=es-PA, timezone=Etc/GMT+1
[junit4] 2> NOTE: Linux 4.4.0-83-generic amd64/Oracle Corporation 1.8.0_152 (64-bit)/cpus=4,threads=1,free=288371624,total=527433728
[junit4] 2> NOTE: All tests run in this JVM: [MoveReplicaHDFSFailoverTest, TestAtomicUpdateErrorCases, TestLuceneMatchVersion, TestNestedDocsSort, TestClassicSimilarityFactory, SolrRequestParserTest, CdcrRequestHandlerTest, ShowFileRequestHandlerTest, DocValuesMultiTest, V2StandaloneTest, TestPolicyCloud, SolrGangliaReporterTest, DebugComponentTest, TestUninvertingReader, TestDeleteCollectionOnDownNodes, TestJavabinTupleStreamParser, SOLR749Test, RollingRestartTest, LargeFieldTest, TestDocumentBuilder, UUIDUpdateProcessorFallbackTest, SolrIndexMetricsTest, TestRemoteStreaming, DocumentAnalysisRequestHandlerTest, QueryEqualityTest, TestNodeLostTrigger, TestMinMaxOnMultiValuedField, TestFieldSortValues, SignatureUpdateProcessorFactoryTest, ReplaceNodeTest, CollectionsAPIDistributedZkTest, PreAnalyzedFieldTest, DeleteShardTest, ChaosMonkeySafeLeaderWithPullReplicasTest, TestSchemalessBufferedUpdates, AddReplicaTest, ZkFailoverTest, PeerSyncReplicationTest, TestRawTransformer, TestTriggerIntegration]
[junit4] Completed [134/763 (1!)] on J1 in 7.63s, 11 tests, 3 failures <<< FAILURES!
[...truncated 40458 lines...]
BUILD FAILED
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/build.xml:835: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/build.xml:101: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build.xml:142: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build.xml:499: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/common-build.xml:2630: Failed to load /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/dev-tools/doap/lucene.rdf
Total time: 276 minutes 50 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any