You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2018/02/23 06:12:29 UTC
[JENKINS] Lucene-Solr-NightlyTests-7.x - Build # 156 - Still
Unstable
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.x/156/
4 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.analytics.facet.QueryFacetTest
Error Message:
1 thread leaked from SUITE scope at org.apache.solr.analytics.facet.QueryFacetTest: 1) Thread[id=66, name=qtp777234554-66, state=TIMED_WAITING, group=TGRP-QueryFacetTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.analytics.facet.QueryFacetTest:
1) Thread[id=66, name=qtp777234554-66, state=TIMED_WAITING, group=TGRP-QueryFacetTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
at __randomizedtesting.SeedInfo.seed([C9F5ABC39B6061B]:0)
FAILED: junit.framework.TestSuite.org.apache.solr.analytics.facet.QueryFacetTest
Error Message:
There are still zombie threads that couldn't be terminated: 1) Thread[id=66, name=qtp777234554-66, state=TIMED_WAITING, group=TGRP-QueryFacetTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
1) Thread[id=66, name=qtp777234554-66, state=TIMED_WAITING, group=TGRP-QueryFacetTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
at __randomizedtesting.SeedInfo.seed([C9F5ABC39B6061B]:0)
FAILED: junit.framework.TestSuite.org.apache.solr.response.transform.TestSubQueryTransformerDistrib
Error Message:
1 thread leaked from SUITE scope at org.apache.solr.response.transform.TestSubQueryTransformerDistrib: 1) Thread[id=2605, name=qtp961437491-2605, state=TIMED_WAITING, group=TGRP-TestSubQueryTransformerDistrib] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.response.transform.TestSubQueryTransformerDistrib:
1) Thread[id=2605, name=qtp961437491-2605, state=TIMED_WAITING, group=TGRP-TestSubQueryTransformerDistrib]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
at __randomizedtesting.SeedInfo.seed([12FB931EC67001C6]:0)
FAILED: junit.framework.TestSuite.org.apache.solr.response.transform.TestSubQueryTransformerDistrib
Error Message:
There are still zombie threads that couldn't be terminated: 1) Thread[id=2605, name=qtp961437491-2605, state=TIMED_WAITING, group=TGRP-TestSubQueryTransformerDistrib] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
1) Thread[id=2605, name=qtp961437491-2605, state=TIMED_WAITING, group=TGRP-TestSubQueryTransformerDistrib]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
at __randomizedtesting.SeedInfo.seed([12FB931EC67001C6]:0)
Build Log:
[...truncated 13189 lines...]
[junit4] Suite: org.apache.solr.response.transform.TestSubQueryTransformerDistrib
[junit4] 2> 838582 INFO (SUITE-TestSubQueryTransformerDistrib-seed#[12FB931EC67001C6]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
[junit4] 2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.response.transform.TestSubQueryTransformerDistrib_12FB931EC67001C6-001/init-core-data-001
[junit4] 2> 838582 WARN (SUITE-TestSubQueryTransformerDistrib-seed#[12FB931EC67001C6]-worker) [ ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=2 numCloses=2
[junit4] 2> 838583 INFO (SUITE-TestSubQueryTransformerDistrib-seed#[12FB931EC67001C6]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
[junit4] 2> 838605 INFO (SUITE-TestSubQueryTransformerDistrib-seed#[12FB931EC67001C6]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
[junit4] 2> 838606 INFO (SUITE-TestSubQueryTransformerDistrib-seed#[12FB931EC67001C6]-worker) [ ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 5 servers in /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.response.transform.TestSubQueryTransformerDistrib_12FB931EC67001C6-001/tempDir-001
[junit4] 2> 838679 INFO (SUITE-TestSubQueryTransformerDistrib-seed#[12FB931EC67001C6]-worker) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 838711 INFO (Thread-816) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 838711 INFO (Thread-816) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 838728 ERROR (Thread-816) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 838799 INFO (SUITE-TestSubQueryTransformerDistrib-seed#[12FB931EC67001C6]-worker) [ ] o.a.s.c.ZkTestServer start zk server on port:46356
[junit4] 2> 838899 INFO (zkConnectionManagerCallback-507-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 839079 INFO (jetty-launcher-504-thread-2) [ ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T21:27:37Z, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
[junit4] 2> 839094 INFO (jetty-launcher-504-thread-1) [ ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T21:27:37Z, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
[junit4] 2> 839099 INFO (jetty-launcher-504-thread-4) [ ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T21:27:37Z, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
[junit4] 2> 839172 INFO (jetty-launcher-504-thread-3) [ ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T21:27:37Z, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
[junit4] 2> 839173 INFO (jetty-launcher-504-thread-5) [ ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T21:27:37Z, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
[junit4] 2> 839221 INFO (jetty-launcher-504-thread-4) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 839221 INFO (jetty-launcher-504-thread-4) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 839221 INFO (jetty-launcher-504-thread-4) [ ] o.e.j.s.session Scavenging every 600000ms
[junit4] 2> 839221 INFO (jetty-launcher-504-thread-4) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@290ac428{/solr,null,AVAILABLE}
[junit4] 2> 839241 INFO (jetty-launcher-504-thread-4) [ ] o.e.j.s.AbstractConnector Started ServerConnector@686bf0cc{SSL,[ssl, http/1.1]}{127.0.0.1:33378}
[junit4] 2> 839241 INFO (jetty-launcher-504-thread-4) [ ] o.e.j.s.Server Started @855602ms
[junit4] 2> 839241 INFO (jetty-launcher-504-thread-4) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=33378}
[junit4] 2> 839242 ERROR (jetty-launcher-504-thread-4) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 839242 INFO (jetty-launcher-504-thread-4) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 7.3.0
[junit4] 2> 839242 INFO (jetty-launcher-504-thread-4) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 839242 INFO (jetty-launcher-504-thread-4) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 839242 INFO (jetty-launcher-504-thread-4) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-02-23T04:43:06.520Z
[junit4] 2> 839279 INFO (jetty-launcher-504-thread-1) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 839279 INFO (jetty-launcher-504-thread-1) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 839279 INFO (jetty-launcher-504-thread-1) [ ] o.e.j.s.session Scavenging every 600000ms
[junit4] 2> 839312 INFO (jetty-launcher-504-thread-1) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6746a5a5{/solr,null,AVAILABLE}
[junit4] 2> 839345 INFO (jetty-launcher-504-thread-2) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 839345 INFO (jetty-launcher-504-thread-2) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 839345 INFO (jetty-launcher-504-thread-2) [ ] o.e.j.s.session Scavenging every 600000ms
[junit4] 2> 839361 INFO (jetty-launcher-504-thread-2) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@3a65814e{/solr,null,AVAILABLE}
[junit4] 2> 839362 INFO (jetty-launcher-504-thread-2) [ ] o.e.j.s.AbstractConnector Started ServerConnector@468f385c{SSL,[ssl, http/1.1]}{127.0.0.1:45632}
[junit4] 2> 839362 INFO (jetty-launcher-504-thread-2) [ ] o.e.j.s.Server Started @855722ms
[junit4] 2> 839362 INFO (jetty-launcher-504-thread-2) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=45632}
[junit4] 2> 839362 ERROR (jetty-launcher-504-thread-2) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 839362 INFO (jetty-launcher-504-thread-2) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 7.3.0
[junit4] 2> 839362 INFO (jetty-launcher-504-thread-2) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 839362 INFO (jetty-launcher-504-thread-2) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 839362 INFO (jetty-launcher-504-thread-2) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-02-23T04:43:06.640Z
[junit4] 2> 839429 INFO (jetty-launcher-504-thread-5) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 839429 INFO (jetty-launcher-504-thread-5) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 839429 INFO (jetty-launcher-504-thread-5) [ ] o.e.j.s.session Scavenging every 600000ms
[junit4] 2> 839442 INFO (jetty-launcher-504-thread-5) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@5c7799b6{/solr,null,AVAILABLE}
[junit4] 2> 839443 INFO (zkConnectionManagerCallback-511-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 839444 INFO (jetty-launcher-504-thread-3) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 839444 INFO (jetty-launcher-504-thread-3) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 839444 INFO (jetty-launcher-504-thread-3) [ ] o.e.j.s.session Scavenging every 660000ms
[junit4] 2> 839444 INFO (jetty-launcher-504-thread-3) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@5a54b6fb{/solr,null,AVAILABLE}
[junit4] 2> 839445 INFO (jetty-launcher-504-thread-3) [ ] o.e.j.s.AbstractConnector Started ServerConnector@26bfe776{SSL,[ssl, http/1.1]}{127.0.0.1:43638}
[junit4] 2> 839445 INFO (jetty-launcher-504-thread-3) [ ] o.e.j.s.Server Started @855805ms
[junit4] 2> 839445 INFO (jetty-launcher-504-thread-3) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=43638}
[junit4] 2> 839445 ERROR (jetty-launcher-504-thread-3) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 839445 INFO (jetty-launcher-504-thread-3) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 7.3.0
[junit4] 2> 839445 INFO (jetty-launcher-504-thread-3) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 839445 INFO (jetty-launcher-504-thread-3) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 839445 INFO (jetty-launcher-504-thread-3) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-02-23T04:43:06.723Z
[junit4] 2> 839462 INFO (jetty-launcher-504-thread-5) [ ] o.e.j.s.AbstractConnector Started ServerConnector@6fd1e736{SSL,[ssl, http/1.1]}{127.0.0.1:44338}
[junit4] 2> 839462 INFO (jetty-launcher-504-thread-5) [ ] o.e.j.s.Server Started @855823ms
[junit4] 2> 839463 INFO (jetty-launcher-504-thread-5) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=44338}
[junit4] 2> 839463 ERROR (jetty-launcher-504-thread-5) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 839463 INFO (jetty-launcher-504-thread-5) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 7.3.0
[junit4] 2> 839463 INFO (jetty-launcher-504-thread-5) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 839463 INFO (jetty-launcher-504-thread-5) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 839463 INFO (jetty-launcher-504-thread-5) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-02-23T04:43:06.741Z
[junit4] 2> 839499 INFO (zkConnectionManagerCallback-509-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 839520 INFO (jetty-launcher-504-thread-1) [ ] o.e.j.s.AbstractConnector Started ServerConnector@466fc8de{SSL,[ssl, http/1.1]}{127.0.0.1:43320}
[junit4] 2> 839520 INFO (jetty-launcher-504-thread-1) [ ] o.e.j.s.Server Started @855880ms
[junit4] 2> 839520 INFO (jetty-launcher-504-thread-1) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=43320}
[junit4] 2> 839520 ERROR (jetty-launcher-504-thread-1) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 839520 INFO (jetty-launcher-504-thread-1) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 7.3.0
[junit4] 2> 839520 INFO (jetty-launcher-504-thread-1) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 839520 INFO (jetty-launcher-504-thread-1) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 839520 INFO (jetty-launcher-504-thread-1) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-02-23T04:43:06.798Z
[junit4] 2> 839530 INFO (zkConnectionManagerCallback-515-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 839531 INFO (jetty-launcher-504-thread-2) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 839531 INFO (zkConnectionManagerCallback-517-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 839531 INFO (jetty-launcher-504-thread-4) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 839531 INFO (zkConnectionManagerCallback-513-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 839582 INFO (jetty-launcher-504-thread-1) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 839583 INFO (jetty-launcher-504-thread-5) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 839609 INFO (jetty-launcher-504-thread-3) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 839694 INFO (jetty-launcher-504-thread-2) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:46356/solr
[junit4] 2> 839755 INFO (jetty-launcher-504-thread-4) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:46356/solr
[junit4] 2> 839800 INFO (jetty-launcher-504-thread-5) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:46356/solr
[junit4] 2> 839826 INFO (jetty-launcher-504-thread-3) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:46356/solr
[junit4] 2> 839841 INFO (jetty-launcher-504-thread-1) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:46356/solr
[junit4] 2> 839857 INFO (zkConnectionManagerCallback-521-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 839893 INFO (zkConnectionManagerCallback-531-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 839934 INFO (zkConnectionManagerCallback-533-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 839934 INFO (zkConnectionManagerCallback-535-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 839987 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x100653b53ef0009, likely client has closed socket
[junit4] 2> 839987 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x100653b53ef0007, likely client has closed socket
[junit4] 2> 840077 INFO (zkConnectionManagerCallback-537-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 840149 INFO (zkConnectionManagerCallback-543-thread-1-processing-n:127.0.0.1:45632_solr) [n:127.0.0.1:45632_solr ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 840150 INFO (zkConnectionManagerCallback-539-thread-1-processing-n:127.0.0.1:33378_solr) [n:127.0.0.1:33378_solr ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 840173 INFO (zkConnectionManagerCallback-541-thread-1-processing-n:127.0.0.1:43638_solr) [n:127.0.0.1:43638_solr ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 840186 INFO (zkConnectionManagerCallback-545-thread-1-processing-n:127.0.0.1:43320_solr) [n:127.0.0.1:43320_solr ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 840189 INFO (zkConnectionManagerCallback-547-thread-1-processing-n:127.0.0.1:44338_solr) [n:127.0.0.1:44338_solr ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 840665 INFO (jetty-launcher-504-thread-2) [n:127.0.0.1:45632_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 840666 INFO (jetty-launcher-504-thread-4) [n:127.0.0.1:33378_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 840667 INFO (jetty-launcher-504-thread-2) [n:127.0.0.1:45632_solr ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:45632_solr
[junit4] 2> 840685 INFO (jetty-launcher-504-thread-4) [n:127.0.0.1:33378_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33378_solr
[junit4] 2> 840686 INFO (jetty-launcher-504-thread-2) [n:127.0.0.1:45632_solr ] o.a.s.c.Overseer Overseer (id=72168899523575820-127.0.0.1:45632_solr-n_0000000000) starting
[junit4] 2> 840725 INFO (zkCallback-542-thread-1-processing-n:127.0.0.1:45632_solr) [n:127.0.0.1:45632_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 840741 INFO (zkCallback-538-thread-1-processing-n:127.0.0.1:33378_solr) [n:127.0.0.1:33378_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 841046 INFO (jetty-launcher-504-thread-2) [n:127.0.0.1:45632_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45632_solr
[junit4] 2> 841266 INFO (zkCallback-538-thread-1-processing-n:127.0.0.1:33378_solr) [n:127.0.0.1:33378_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 841266 INFO (zkCallback-542-thread-1-processing-n:127.0.0.1:45632_solr) [n:127.0.0.1:45632_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 841525 INFO (jetty-launcher-504-thread-3) [n:127.0.0.1:43638_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 841667 INFO (jetty-launcher-504-thread-3) [n:127.0.0.1:43638_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 841793 INFO (jetty-launcher-504-thread-2) [n:127.0.0.1:45632_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_45632.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@9fbcf5e
[junit4] 2> 841795 INFO (jetty-launcher-504-thread-4) [n:127.0.0.1:33378_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33378.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@9fbcf5e
[junit4] 2> 841885 INFO (jetty-launcher-504-thread-2) [n:127.0.0.1:45632_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_45632.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@9fbcf5e
[junit4] 2> 841885 INFO (jetty-launcher-504-thread-2) [n:127.0.0.1:45632_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_45632.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@9fbcf5e
[junit4] 2> 841919 INFO (jetty-launcher-504-thread-2) [n:127.0.0.1:45632_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.response.transform.TestSubQueryTransformerDistrib_12FB931EC67001C6-001/tempDir-001/node2/.
[junit4] 2> 842029 INFO (jetty-launcher-504-thread-4) [n:127.0.0.1:33378_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33378.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@9fbcf5e
[junit4] 2> 842029 INFO (jetty-launcher-504-thread-4) [n:127.0.0.1:33378_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33378.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@9fbcf5e
[junit4] 2> 842030 INFO (jetty-launcher-504-thread-4) [n:127.0.0.1:33378_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.response.transform.TestSubQueryTransformerDistrib_12FB931EC67001C6-001/tempDir-001/node3/.
[junit4] 2> 842086 INFO (jetty-launcher-504-thread-3) [n:127.0.0.1:43638_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43638_solr
[junit4] 2> 842105 INFO (zkCallback-542-thread-1-processing-n:127.0.0.1:45632_solr) [n:127.0.0.1:45632_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 842138 INFO (zkCallback-538-thread-1-processing-n:127.0.0.1:33378_solr) [n:127.0.0.1:33378_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 842247 INFO (zkCallback-540-thread-1-processing-n:127.0.0.1:43638_solr) [n:127.0.0.1:43638_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 842443 INFO (jetty-launcher-504-thread-1) [n:127.0.0.1:43320_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
[junit4] 2> 842481 INFO (jetty-launcher-504-thread-1) [n:127.0.0.1:43320_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 842483 INFO (jetty-launcher-504-thread-1) [n:127.0.0.1:43320_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43320_solr
[junit4] 2> 842521 INFO (jetty-launcher-504-thread-3) [n:127.0.0.1:43638_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_43638.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@9fbcf5e
[junit4] 2> 842540 INFO (zkCallback-538-thread-1-processing-n:127.0.0.1:33378_solr) [n:127.0.0.1:33378_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 842540 INFO (zkCallback-542-thread-1-processing-n:127.0.0.1:45632_solr) [n:127.0.0.1:45632_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 842587 INFO (jetty-launcher-504-thread-3) [n:127.0.0.1:43638_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_43638.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@9fbcf5e
[junit4] 2> 842587 INFO (jetty-launcher-504-thread-3) [n:127.0.0.1:43638_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_43638.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@9fbcf5e
[junit4] 2> 842589 INFO (zkCallback-544-thread-1-processing-n:127.0.0.1:43320_solr) [n:127.0.0.1:43320_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 842601 INFO (zkCallback-540-thread-1-processing-n:127.0.0.1:43638_solr) [n:127.0.0.1:43638_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 842601 INFO (jetty-launcher-504-thread-3) [n:127.0.0.1:43638_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.response.transform.TestSubQueryTransformerDistrib_12FB931EC67001C6-001/tempDir-001/node5/.
[junit4] 2> 842625 INFO (jetty-launcher-504-thread-5) [n:127.0.0.1:44338_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
[junit4] 2> 842660 INFO (jetty-launcher-504-thread-5) [n:127.0.0.1:44338_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 842675 INFO (jetty-launcher-504-thread-5) [n:127.0.0.1:44338_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44338_solr
[junit4] 2> 842677 INFO (zkCallback-542-thread-1-processing-n:127.0.0.1:45632_solr) [n:127.0.0.1:45632_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
[junit4] 2> 842695 INFO (zkCallback-540-thread-1-processing-n:127.0.0.1:43638_solr) [n:127.0.0.1:43638_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
[junit4] 2> 842698 INFO (zkCallback-544-thread-1-processing-n:127.0.0.1:43320_solr) [n:127.0.0.1:43320_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
[junit4] 2> 842767 INFO (zkCallback-538-thread-1-processing-n:127.0.0.1:33378_solr) [n:127.0.0.1:33378_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
[junit4] 2> 842865 INFO (zkCallback-546-thread-1-processing-n:127.0.0.1:44338_solr) [n:127.0.0.1:44338_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
[junit4] 2> 843063 INFO (jetty-launcher-504-thread-1) [n:127.0.0.1:43320_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_43320.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@9fbcf5e
[junit4] 2> 843103 INFO (jetty-launcher-504-thread-1) [n:127.0.0.1:43320_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_43320.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@9fbcf5e
[junit4] 2> 843104 INFO (jetty-launcher-504-thread-1) [n:127.0.0.1:43320_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_43320.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@9fbcf5e
[junit4] 2> 843121 INFO (jetty-launcher-504-thread-1) [n:127.0.0.1:43320_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.response.transform.TestSubQueryTransformerDistrib_12FB931EC67001C6-001/tempDir-001/node1/.
[junit4] 2> 843132 INFO (jetty-launcher-504-thread-5) [n:127.0.0.1:44338_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_44338.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@9fbcf5e
[junit4] 2> 843291 INFO (jetty-launcher-504-thread-5) [n:127.0.0.1:44338_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_44338.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@9fbcf5e
[junit4] 2> 843309 INFO (jetty-launcher-504-thread-5) [n:127.0.0.1:44338_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_44338.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@9fbcf5e
[junit4] 2> 843310 INFO (jetty-launcher-504-thread-5) [n:127.0.0.1:44338_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.response.transform.TestSubQueryTransformerDistrib_12FB931EC67001C6-001/tempDir-001/node4/.
[junit4] 2> 843848 INFO (zkConnectionManagerCallback-559-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 843869 INFO (zkConnectionManagerCallback-563-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 843870 INFO (SUITE-TestSubQueryTransformerDistrib-seed#[12FB931EC67001C6]-worker) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (5)
[junit4] 2> 843871 INFO (SUITE-TestSubQueryTransformerDistrib-seed#[12FB931EC67001C6]-worker) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:46356/solr ready
[junit4] 2> 845510 INFO (qtp1118076649-2646) [n:127.0.0.1:43638_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=2&property.config=solrconfig-doctransformers.xml&property.schema=schema-docValuesJoin.xml&collection.configName=solrCloudCollectionConfig&name=people&nrtReplicas=2&action=CREATE&numShards=2&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 845549 INFO (OverseerThreadFactory-1083-thread-1-processing-n:127.0.0.1:45632_solr) [n:127.0.0.1:45632_solr ] o.a.s.c.a.c.CreateCollectionCmd Create collection people
[junit4] 2> 845729 INFO (OverseerStateUpdate-72168899523575820-127.0.0.1:45632_solr-n_0000000000) [n:127.0.0.1:45632_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"people",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"people_shard1_replica_n1",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:44338/solr",
[junit4] 2> "type":"NRT",
[junit4] 2> "waitForFinalState":"false"}
[junit4] 2> 845732 INFO (OverseerStateUpdate-72168899523575820-127.0.0.1:45632_solr-n_0000000000) [n:127.0.0.1:45632_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"people",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"people_shard1_replica_n2",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:33378/solr",
[junit4] 2> "type":"NRT",
[junit4] 2> "waitForFinalState":"false"}
[junit4] 2> 845773 INFO (OverseerStateUpdate-72168899523575820-127.0.0.1:45632_solr-n_0000000000) [n:127.0.0.1:45632_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"people",
[junit4] 2> "shard":"shard2",
[junit4] 2> "core":"people_shard2_replica_n4",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:43638/solr",
[junit4] 2> "type":"NRT",
[junit4] 2> "waitForFinalState":"false"}
[junit4] 2> 845774 INFO (OverseerStateUpdate-72168899523575820-127.0.0.1:45632_solr-n_0000000000) [n:127.0.0.1:45632_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"people",
[junit4] 2> "shard":"shard2",
[junit4] 2> "core":"people_shard2_replica_n6",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:43320/solr",
[junit4] 2> "type":"NRT",
[junit4] 2> "waitForFinalState":"false"}
[junit4] 2> 846202 INFO (qtp1118076649-2641) [n:127.0.0.1:43638_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=solrCloudCollectionConfig&newCollection=true&collection=people&version=2&replicaType=NRT&property.config=solrconfig-doctransformers.xml&property.schema=schema-docValuesJoin.xml&coreNodeName=core_node7&name=people_shard2_replica_n4&action=CREATE&numShards=2&shard=shard2&wt=javabin
[junit4] 2> 846203 INFO (qtp1118076649-2641) [n:127.0.0.1:43638_solr ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 846361 INFO (qtp194637842-2614) [n:127.0.0.1:43320_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=solrCloudCollectionConfig&newCollection=true&collection=people&version=2&replicaType=NRT&property.config=solrconfig-doctransformers.xml&property.schema=schema-docValuesJoin.xml&coreNodeName=core_node8&name=people_shard2_replica_n6&action=CREATE&numShards=2&shard=shard2&wt=javabin
[junit4] 2> 846362 INFO (qtp194637842-2614) [n:127.0.0.1:43320_solr ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 846375 INFO (qtp854481757-2636) [n:127.0.0.1:44338_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=solrCloudCollectionConfig&newCollection=true&collection=people&version=2&replicaType=NRT&property.config=solrconfig-doctransformers.xml&property.schema=schema-docValuesJoin.xml&coreNodeName=core_node3&name=people_shard1_replica_n1&action=CREATE&numShards=2&shard=shard1&wt=javabin
[junit4] 2> 846375 INFO (qtp854481757-2636) [n:127.0.0.1:44338_solr ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 846396 INFO (qtp961437491-2605) [n:127.0.0.1:33378_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=solrCloudCollectionConfig&newCollection=true&collection=people&version=2&replicaType=NRT&property.config=solrconfig-doctransformers.xml&property.schema=schema-docValuesJoin.xml&coreNodeName=core_node5&name=people_shard1_replica_n2&action=CREATE&numShards=2&shard=shard1&wt=javabin
[junit4] 2> 846397 INFO (qtp961437491-2605) [n:127.0.0.1:33378_solr ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 846660 INFO (zkCallback-544-thread-1-processing-n:127.0.0.1:43320_solr) [n:127.0.0.1:43320_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5])
[junit4] 2> 846660 INFO (zkCallback-540-thread-1-processing-n:127.0.0.1:43638_solr) [n:127.0.0.1:43638_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5])
[junit4] 2> 846663 INFO (zkCallback-540-thread-2-processing-n:127.0.0.1:43638_solr) [n:127.0.0.1:43638_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5])
[junit4] 2> 846663 INFO (zkCallback-544-thread-2-processing-n:127.0.0.1:43320_solr) [n:127.0.0.1:43320_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5])
[junit4] 2> 846672 INFO (zkCallback-538-thread-1-processing-n:127.0.0.1:33378_solr) [n:127.0.0.1:33378_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5])
[junit4] 2> 846672 INFO (zkCallback-538-thread-2-processing-n:127.0.0.1:33378_solr) [n:127.0.0.1:33378_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5])
[junit4] 2> 846676 INFO (zkCallback-546-thread-2-processing-n:127.0.0.1:44338_solr) [n:127.0.0.1:44338_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5])
[junit4] 2> 846679 INFO (zkCallback-546-thread-1-processing-n:127.0.0.1:44338_solr) [n:127.0.0.1:44338_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5])
[junit4] 2> 847551 INFO (qtp194637842-2614) [n:127.0.0.1:43320_solr c:people s:shard2 r:core_node8 x:people_shard2_replica_n6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
[junit4] 2> 847571 INFO (qtp961437491-2605) [n:127.0.0.1:33378_solr c:people s:shard1 r:core_node5 x:people_shard1_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
[junit4] 2> 847571 INFO (qtp1118076649-2641) [n:127.0.0.1:43638_solr c:people s:shard2 r:core_node7 x:people_shard2_replica_n4] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
[junit4] 2> 847706 INFO (qtp194637842-2614) [n:127.0.0.1:43320_solr c:people s:shard2 r:core_node8 x:people_shard2_replica_n6] o.a.s.s.IndexSchema [people_shard2_replica_n6] Schema name=doc-values-for-Join
[junit4] 2> 847706 INFO (qtp1118076649-2641) [n:127.0.0.1:43638_solr c:people s:shard2 r:core_node7 x:people_shard2_replica_n4] o.a.s.s.IndexSchema [people_shard2_replica_n4] Schema name=doc-values-for-Join
[junit4] 2> 847763 INFO (qtp961437491-2605) [n:127.0.0.1:33378_solr c:people s:shard1 r:core_node5 x:people_shard1_replica_n2] o.a.s.s.IndexSchema [people_shard1_replica_n2] Schema name=doc-values-for-Join
[junit4] 2> 847777 INFO (qtp1118076649-2641) [n:127.0.0.1:43638_solr c:people s:shard2 r:core_node7 x:people_shard2_replica_n4] o.a.s.s.IndexSchema Loaded schema doc-values-for-Join/1.6 with uniqueid field id
[junit4] 2> 847800 INFO (qtp961437491-2605) [n:127.0.0.1:33378_solr c:people s:shard1 r:core_node5 x:people_shard1_replica_n2] o.a.s.s.IndexSchema Loaded schema doc-values-for-Join/1.6 with uniqueid field id
[junit4] 2> 847815 INFO (qtp854481757-2636) [n:127.0.0.1:44338_solr c:people s:shard1 r:core_node3 x:people_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
[junit4] 2> 847817 INFO (qtp961437491-2605) [n:127.0.0.1:33378_solr c:people s:shard1 r:core_node5 x:people_shard1_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'people_shard1_replica_n2' using configuration from collection people, trusted=true
[junit4] 2> 847849 INFO (qtp961437491-2605) [n:127.0.0.1:33378_solr c:people s:shard1 r:core_node5 x:people_shard1_replica_n2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33378.solr.core.people.shard1.replica_n2' (registry 'solr.core.people.shard1.replica_n2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@9fbcf5e
[junit4] 2> 847849 INFO (qtp961437491-2605) [n:127.0.0.1:33378_solr c:people s:shard1 r:core_node5 x:people_shard1_replica_n2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 847849 INFO (qtp961437491-2605) [n:127.0.0.1:33378_solr c:people s:shard1 r:core_node5 x:people_shard1_replica_n2] o.a.s.c.SolrCore [[people_shard1_replica_n2] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.response.transform.TestSubQueryTransformerDistrib_12FB931EC67001C6-001/tempDir-001/node3/people_shard1_replica_n2], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.response.transform.TestSubQueryTransformerDistrib_12FB931EC67001C6-001/tempDir-001/node3/./people_shard1_replica_n2/data/]
[junit4] 2> 847850 INFO (qtp1118076649-2641) [n:127.0.0.1:43638_solr c:people s:shard2 r:core_node7 x:people_shard2_replica_n4] o.a.s.c.CoreContainer Creating SolrCore 'people_shard2_replica_n4' using configuration from collection people, trusted=true
[junit4] 2> 847851 INFO (qtp194637842-2614) [n:127.0.0.1:43320_solr c:people s:shard2 r:core_node8 x:people_shard2_replica_n6] o.a.s.s.IndexSchema Loaded schema doc-values-for-Join/1.6 with uniqueid field id
[junit4] 2> 847851 INFO (qtp1118076649-2641) [n:127.0.0.1:43638_solr c:people s:shard2 r:core_node7 x:people_shard2_replica_n4] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_43638.solr.core.people.shard2.replica_n4' (registry 'solr.core.people.shard2.replica_n4') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@9fbcf5e
[junit4] 2> 847851 INFO (qtp1118076649-2641) [n:127.0.0.1:43638_solr c:people s:shard2 r:core_node7 x:people_shard2_replica_n4] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 847851 INFO (qtp1118076649-2641) [n:127.0.0.1:43638_solr c:people s:shard2 r:core_node7 x:people_shard2_replica_n4] o.a.s.c.SolrCore [[people_shard2_replica_n4] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.response.transform.TestSubQueryTransformerDistrib_12FB931EC67001C6-001/tempDir-001/node5/people_shard2_replica_n4], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.response.transform.TestSubQueryTransformerDistrib_12FB931EC67001C6-001/tempDir-001/node5/./people_shard2_replica_n4/data/]
[junit4] 2> 847870 INFO (qtp854481757-2636) [n:127.0.0.1:44338_solr c:people s:shard1 r:core_node3 x:people_shard1_replica_n1] o.a.s.s.IndexSchema [people_shard1_replica_n1] Schema name=doc-values-for-Join
[junit4] 2> 847903 INFO (qtp854481757-2636) [n:127.0.0.1:44338_solr c:people s:shard1 r:core_node3 x:people_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema doc-values-for-Join/1.6 with uniqueid field id
[junit4] 2> 847993 INFO (qtp854481757-2636) [n:127.0.0.1:44338_solr c:people s:shard1 r:core_node3 x:people_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'people_shard1_replica_n1' using configuration from collection people, trusted=true
[junit4] 2> 847994 INFO (qtp854481757-2636) [n:127.0.0.1:44338_solr c:people s:shard1 r:core_node3 x:people_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_44338.solr.core.people.shard1.replica_n1' (registry 'solr.core.people.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@9fbcf5e
[junit4] 2> 847994 INFO (qtp854481757-2636) [n:127.0.0.1:44338_solr c:people s:shard1 r:core_node3 x:people_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 847994 INFO (qtp854481757-2636) [n:127.0.0.1:44338_solr c:people s:shard1 r:core_node3 x:people_shard1_replica_n1] o.a.s.c.SolrCore [[people_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.response.transform.TestSubQueryTransformerDistrib_12FB931EC67001C6-001/tempDir-001/node4/people_shard1_replica_n1], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.response.transform.TestSubQueryTransformerDistrib_12FB931EC67001C6-001/tempDir-001/node4/./people_shard1_replica_n1/data/]
[junit4] 2> 848228 INFO (qtp194637842-2614) [n:127.0.0.1:43320_solr c:people s:shard2 r:core_node8 x:people_shard2_replica_n6] o.a.s.c.CoreContainer Creating SolrCore 'people_shard2_replica_n6' using configuration from collection people, trusted=true
[junit4] 2> 848229 INFO (qtp194637842-2614) [n:127.0.0.1:43320_solr c:people s:shard2 r:core_node8 x:people_shard2_replica_n6] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_43320.solr.core.people.shard2.replica_n6' (registry 'solr.core.people.shard2.replica_n6') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@9fbcf5e
[junit4] 2> 848229 INFO (qtp194637842-2614) [n:127.0.0.1:43320_solr c:people s:shard2 r:core_node8 x:people_shard2_replica_n6] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 848229 INFO (qtp194637842-2614) [n:127.0.0.1:43320_solr c:people s:shard2 r:core_node8 x:people_shard2_replica_n6] o.a.s.c.SolrCore [[people_shard2_replica_n6] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.response.transform.TestSubQueryTransformerDistrib_12FB931EC67001C6-001/tempDir-001/node1/people_shard2_replica_n6], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.response.transform.TestSubQueryTransformerDistrib_12FB931EC67001C6-001/tempDir-001/node1/./people_shard2_replica_n6/data/]
[junit4] 2> 848484 INFO (qtp1118076649-2641) [n:127.0.0.1:43638_solr c:people s:shard2 r:core_node7 x:people_shard2_replica_n4] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 848484 INFO (qtp1118076649-2641) [n:127.0.0.1:43638_solr c:people s:shard2 r:core_node7 x:people_shard2_replica_n4] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 848485 INFO (qtp1118076649-2641) [n:127.0.0.1:43638_solr c:people s:shard2 r:core_node7 x:people_shard2_replica_n4] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 848485 INFO (qtp1118076649-2641) [n:127.0.0.1:43638_solr c:people s:shard2 r:core_node7 x:people_shard2_replica_n4] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 848487 INFO (qtp1118076649-2641) [n:127.0.0.1:43638_solr c:people s:shard2 r:core_node7 x:people_shard2_replica_n4] o.a.s.s.SolrIndexSearcher Opening [Searcher@1072851b[people_shard2_replica_n4] main]
[junit4] 2> 848501 INFO (qtp1118076649-2641) [n:127.0.0.1:43638_solr c:people s:shard2 r:core_node7 x:people_shard2_replica_n4] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/solrCloudCollectionConfig
[junit4] 2> 848501 INFO (qtp1118076649-2641) [n:127.0.0.1:43638_solr c:people s:shard2 r:core_node7 x:people_shard2_replica_n4] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/solrCloudCollectionConfig
[junit4] 2> 848502 INFO (qtp1118076649-2641) [n:127.0.0.1:43638_solr c:people s:shard2 r:core_node7 x:people_shard2_replica_n4] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 848502 INFO (qtp1118076649-2641) [n:127.0.0.1:43638_solr c:people s:shard2 r:core_node7 x:people_shard2_replica_n4] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1593165475511009280
[junit4] 2> 848537 INFO (searcherExecutor-1095-thread-1-processing-n:127.0.0.1:43638_solr x:people_shard2_replica_n4 s:shard2 c:people r:core_node7) [n:127.0.0.1:43638_solr c:people s:shard2 r:core_node7 x:people_shard2_replica_n4] o.a.s.c.SolrCore [people_shard2_replica_n4] Registered new searcher Searcher@1072851b[people_shard2_replica_n4] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 848575 INFO (qtp1118076649-2641) [n:127.0.0.1:43638_solr c:people s:shard2 r:core_node7 x:people_shard2_replica_n4] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard2: total=2 found=1 timeoutin=9999ms
[junit4] 2> 848618 INFO (qtp961437491-2605) [n:127.0.0.1:33378_solr c:people s:shard1 r:core_node5 x:people_shard1_replica_n2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 848618 INFO (qtp961437491-2605) [n:127.0.0.1:33378_solr c:people s:shard1 r:core_node5 x:people_shard1_replica_n2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 848620 INFO (qtp961437491-2605) [n:127.0.0.1:33378_solr c:people s:shard1 r:core_node5 x:people_shard1_replica_n2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 848620 INFO (qtp961437491-2605) [n:127.0.0.1:33378_solr c:people s:shard1 r:core_node5 x:people_shard1_replica_n2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 848621 INFO (qtp961437491-2605) [n:127.0.0.1:33378_solr c:people s:shard1 r:core_node5 x:people_shard1_replica_n2] o.a.s.s.SolrIndexSearcher Opening [Searcher@3820288e[people_shard1_replica_n2] main]
[junit4] 2> 848740 INFO (qtp961437491-2605) [n:127.0.0.1:33378_solr c:people s:shard1 r:core_node5 x:people_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/solrCloudCollectionConfig
[junit4] 2> 848760 INFO (qtp961437491-2605) [n:127.0.0.1:33378_solr c:people s:shard1 r:core_node5 x:people_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/solrCloudCollectionConfig
[junit4] 2> 848761 INFO (qtp961437491-2605) [n:127.0.0.1:33378_solr c:people s:shard1 r:core_node5 x:people_shard1_replica_n2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 848764 INFO (qtp854481757-2636) [n:127.0.0.1:44338_solr c:people s:shard1 r:core_node3 x:people_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 848764 INFO (qtp854481757-2636) [n:127.0.0.1:44338_solr c:people s:shard1 r:core_node3 x:people_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 848765 INFO (zkCallback-540-thread-2-processing-n:127.0.0.1:43638_solr) [n:127.0.0.1:43638_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5])
[junit4] 2> 848765 INFO (zkCallback-540-thread-1-processing-n:127.0.0.1:43638_solr) [n:127.0.0.1:43638_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5])
[junit4] 2> 848768 INFO (qtp194637842-2614) [n:127.0.0.1:43320_solr c:people s:shard2 r:core_node8 x:people_shard2_replica_n6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 848768 INFO (qtp194637842-2614) [n:127.0.0.1:43320_solr c:people s:shard2 r:core_node8 x:people_shard2_replica_n6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 848781 INFO (qtp961437491-2605) [n:127.0.0.1:33378_solr c:people s:shard1 r:core_node5 x:people_shard1_replica_n2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1593165475803561984
[junit4] 2> 848781 INFO (qtp194637842-2614) [n:127.0.0.1:43320_solr c:people s:shard2 r:core_node8 x:people_shard2_replica_n6] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 848781 INFO (qtp194637842-2614) [n:127.0.0.1:43320_solr c:people s:shard2 r:core_node8 x:people_shard2_replica_n6] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 848782 INFO (zkCallback-546-thread-2-processing-n:127.0.0.1:44338_solr) [n:127.0.0.1:44338_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5])
[junit4] 2> 848782 INFO (zkCallback-538-thread-2-processing-n:127.0.0.1:33378_solr) [n:127.0.0.1:33378_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5])
[junit4] 2> 848783 INFO (zkCallback-538-thread-1-processing-n:127.0.0.1:33378_solr) [n:127.0.0.1:33378_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5])
[junit4] 2> 848783 INFO (qtp194637842-2614) [n:127.0.0.1:43320_solr c:people s:shard2 r:core_node8 x:people_shard2_replica_n6] o.a.s.s.SolrIndexSearcher Opening [Searcher@4c2e793f[people_shard2_replica_n6] main]
[junit4] 2> 848784 INFO (qtp854481757-2636) [n:127.0.0.1:44338_solr c:people s:shard1 r:core_node3 x:people_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 848784 INFO (qtp854481757-2636) [n:127.0.0.1:44338_solr c:people s:shard1 r:core_node3 x:people_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 848787 INFO (zkCallback-546-thread-1-processing-n:127.0.0.1:44338_solr) [n:127.0.0.1:44338_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5])
[junit4] 2> 848788 INFO (zkCallback-544-thread-1-processing-n:127.0.0.1:43320_solr) [n:127.0.0.1:43320_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5])
[junit4] 2> 848793 INFO (zkCallback-544-thread-2-processing-n:127.0.0.1:43320_solr) [n:127.0.0.1:43320_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5])
[junit4] 2> 848819 INFO (qtp854481757-2636) [n:127.0.0.1:44338_solr c:people s:shard1 r:core_node3 x:people_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@1fbcca1a[people_shard1_replica_n1] main]
[junit4] 2> 848830 INFO (searcherExecutor-1094-thread-1-processing-n:127.0.0.1:33378_solr x:people_shard1_replica_n2 s:shard1 c:people r:core_node5) [n:127.0.0.1:33378_solr c:people s:shard1 r:core_node5 x:people_shard1_replica_n2] o.a.s.c.SolrCore [people_shard1_replica_n2] Registered new searcher Searcher@3820288e[people_shard1_replica_n2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 848833 INFO (qtp194637842-2614) [n:127.0.0.1:43320_solr c:people s:shard2 r:core_node8 x:people_shard2_replica_n6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/solrCloudCollectionConfig
[junit4] 2> 848834 INFO (qtp194637842-2614) [n:127.0.0.1:43320_solr c:people s:shard2 r:core_node8 x:people_shard2_replica_n6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/solrCloudCollectionConfig
[junit4] 2> 848834 INFO (qtp194637842-2614) [n:127.0.0.1:43320_solr c:people s:shard2 r:core_node8 x:people_shard2_replica_n6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 848835 INFO (qtp194637842-2614) [n:127.0.0.1:43320_solr c:people s:shard2 r:core_node8 x:people_shard2_replica_n6] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1593165475860185088
[junit4] 2> 848846 INFO (searcherExecutor-1098-thread-1-processing-n:127.0.0.1:43320_solr x:people_shard2_replica_n6 s:shard2 c:people r:core_node8) [n:127.0.0.1:43320_solr c:people s:shard2 r:core_node8 x:people_shard2_replica_n6] o.a.s.c.SolrCore [people_shard2_replica_n6] Registered new searcher Searcher@4c2e793f[people_shard2_replica_n6] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 848850 INFO (qtp961437491-2605) [n:127.0.0.1:33378_solr c:people s:shard1 r:core_node5 x:people_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard1: total=2 found=1 timeoutin=9999ms
[junit4] 2> 848851 INFO (qtp854481757-2636) [n:127.0.0.1:44338_solr c:people s:shard1 r:core_node3 x:people_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/solrCloudCollectionConfig
[junit4] 2> 848851 INFO (qtp854481757-2636) [n:127.0.0.1:44338_solr c:people s:shard1 r:core_node3 x:people_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/solrCloudCollectionConfig
[junit4] 2> 848852 INFO (qtp854481757-2636) [n:127.0.0.1:44338_solr c:people s:shard1 r:core_node3 x:people_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 848852 INFO (qtp854481757-2636) [n:127.0.0.1:44338_solr c:people s:shard1 r:core_node3 x:people_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1593165475878010880
[junit4] 2> 848883 INFO (searcherExecutor-1096-thread-1-processing-n:127.0.0.1:44338_solr x:people_shard1_replica_n1 s:shard1 c:people r:core_node3) [n:127.0.0.1:44338_solr c:people s:shard1 r:core_node3 x:people_shard1_replica_n1] o.a.s.c.SolrCore [people_shard1_replica_n1] Registered new searcher Searcher@1fbcca1a[people_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 849029 INFO (zkCallback-546-thread-2-processing-n:127.0.0.1:44338_solr) [n:127.0.0.1:44338_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5])
[junit4] 2> 849029 INFO (zkCallback-538-thread-1-processing-n:127.0.0.1:33378_solr) [n:127.0.0.1:33378_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5])
[junit4] 2> 849029 INFO (zkCallback-538-thread-2-processing-n:127.0.0.1:33378_solr) [n:127.0.0.1:33378_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5])
[junit4] 2> 849031 INFO (zkCallback-546-thread-1-processing-n:127.0.0.1:44338_solr) [n:127.0.0.1:44338_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5])
[junit4] 2> 849031 INFO (zkCallback-540-thread-2-processing-n:127.0.0.1:43638_solr) [n:127.0.0.1:43638_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5])
[junit4] 2> 849031 INFO (zkCallback-544-thread-1-processing-n:127.0.0.1:43320_solr) [n:127.0.0.1:43320_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5])
[junit4] 2> 849031 INFO (zkCallback-544-thread-2-processing-n:127.0.0.1:43320_solr) [n:127.0.0.1:43320_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5])
[junit4] 2> 849032 INFO (zkCallback-540-thread-1-processing-n:127.0.0.1:43638_solr) [n:127.0.0.1:43638_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5])
[junit4] 2> 849081 INFO (qtp1118076649-2641) [n:127.0.0.1:43638_solr c:people s:shard2 r:core_node7 x:people_shard2_replica_n4] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 849081 INFO (qtp1118076649-2641) [n:127.0.0.1:43638_solr c:people s:shard2 r:core_node7 x:people_shard2_replica_n4] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 849081 INFO (qtp1118076649-2641) [n:127.0.0.1:43638_solr c:people s:shard2 r:core_node7 x:people_shard2_replica_n4] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:43638/solr/people_shard2_replica_n4/
[junit4] 2> 849082 INFO (qtp1118076649-2641) [n:127.0.0.1:43638_solr c:people s:shard2 r:core_node7 x:people_shard2_replica_n4] o.a.s.u.PeerSync PeerSync: core=people_shard2_replica_n4 url=https://127.0.0.1:43638/solr START replicas=[https://127.0.0.1:43320/solr/people_shard2_replica_n6/] nUpdates=100
[junit4] 2> 849186 INFO (qtp194637842-2609) [n:127.0.0.1:43320_solr c:people s:shard2 r:core_node8 x:people_shard2_replica_n6] o.a.s.c.S.Request [people_shard2_replica_n6] webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=1
[junit4] 2> 849357 INFO (qtp961437491-2605) [n:127.0.0.1:33378_solr c:people s:shard1 r:core_node5 x:people_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 849358 INFO (qtp961437491-2605) [n:127.0.0.1:33378_solr c:people s:shard1 r:core_node5 x:people_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 849358 INFO (qtp961437491-2605) [n:127.0.0.1:33378_solr c:people s:shard1 r:core_node5 x:people_shard1_replica_n2] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:33378/solr/people_shard1_replica_n2/
[junit4] 2> 849358 INFO (qtp961437491-2605) [n:127.0.0.1:33378_solr c:people s:shard1 r:core_node5 x:people_shard1_replica_n2] o.a.s.u.PeerSync PeerSync: core=people_shard1_replica_n2 url=https://127.0.0.1:33378/solr START replicas=[https://127.0.0.1:44338/solr/people_shard1_replica_n1/] nUpdates=100
[junit4] 2> 849439 INFO (qtp1118076649-2641) [n:127.0.0.1:43638_solr c:people s:shard2 r:core_node7 x:people_shard2_replica_n4] o.a.s.u.PeerSync PeerSync: core=people_shard2_replica_n4 url=https://127.0.0.1:43638/solr DONE. We have no versions. sync failed.
[junit4] 2> 849439 INFO (qtp1118076649-2641) [n:127.0.0.1:43638_solr c:people s:shard2 r:core_node7 x:people_shard2_replica_n4] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate
[junit4] 2> 849439 INFO (qtp1118076649-2641) [n:127.0.0.1:43638_solr c:people s:shard2 r:core_node7 x:people_shard2_replica_n4] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
[junit4] 2> 849439 INFO (qtp1118076649-2641) [n:127.0.0.1:43638_solr c:people s:shard2 r:core_node7 x:people_shard2_replica_n4] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 849672 INFO (qtp1118076649-2641) [n:127.0.0.1:43638_solr c:people s:shard2 r:core_node7 x:people_shard2_replica_n4] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:43638/solr/people_shard2_replica_n4/ shard2
[junit4] 2> 849740 INFO (qtp961437491-2605) [n:127.0.0.1:33378_solr c:people s:shard1 r:core_node5 x:people_shard1_replica_n2] o.a.s.u.PeerSync PeerSync: core=people_shard1_replica_n2 url=https://127.0.0.1:33378/solr DONE. We have no versions. sync failed.
[junit4] 2> 849741 INFO (qtp854481757-2631) [n:127.0.0.1:44338_solr c:people s:shard1 r:core_node3 x:people_shard1_replica_n1] o.a.s.c.S.Request [people_shard1_replica_n1] webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=68
[junit4] 2> 849742 INFO (qtp961437491-2605) [n:127.0.0.1:33378_solr c:people s:shard1 r:core_node5 x:people_shard1_replica_n2] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate
[junit4] 2> 849742 INFO (qtp961437491-2605) [n:127.0.0.1:33378_solr c:people s:shard1 r:core_node5 x:people_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
[junit4] 2> 849742 INFO (qtp961437491-2605) [n:127.0.0.1:33378_solr c:people s:shard1 r:core_node5 x:people_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 849744 INFO (qtp961437491-2605) [n:127.0.0.1:33378_solr c:people s:shard1 r:core_node5 x:people_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:33378/solr/people_shard1_replica_n2/ shard1
[junit4] 2> 849866 INFO (zkCallback-544-thread-1-processing-n:127.0.0.1:43320_solr) [n:127.0.0.1:43320_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5])
[junit4] 2> 849866 INFO (zkCallback-544-thread-2-processing-n:127.0.0.1:43320_solr) [n:127.0.0.1:43320_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5])
[junit4] 2> 849866 INFO (zkCallback-538-thread-2-processing-n:127.0.0.1:33378_solr) [n:127.0.0.1:33378_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5])
[junit4] 2> 849866 INFO (zkCallback-546-thread-1-processing-n:127.0.0.1:44338_solr) [n:127.0.0.1:44338_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5])
[junit4] 2> 849866 INFO (zkCallback-546-thread-2-processing-n:127.0.0.1:44338_solr) [n:127.0.0.1:44338_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5])
[junit4] 2> 849869 INFO (zkCallback-538-thread-1-processing-n:127.0.0.1:33378_solr) [n:127.0.0.1:33378_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5])
[junit4] 2> 849869 INFO (zkCallback-540-thread-1-processing-n:127.0.0.1:43638_solr) [n:127.0.0.1:43638_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5])
[junit4] 2> 849869 INFO (zkCallback-540-thread-2-processing-n:127.0.0.1:43638_solr) [n:127.0.0.1:43638_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5])
[junit4] 2> 849906 INFO (qtp194637842-2614) [n:127.0.0.1:43320_solr c:people s:shard2 r:core_node8 x:people_shard2_replica_n6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=solrCloudCollectionConfig&newCollection=true&collection=people&version=2&replicaType=NRT&property.config=solrconfig-doctransformers.xml&property.schema=schema-docValuesJoin.xml&coreNodeName=core_node8&name=people_shard2_replica_n6&action=CREATE&numShards=2&shard=shard2&wt=javabin} status=0 QTime=3544
[junit4] 2> 849906 INFO (qtp961437491-2605) [n:127.0.0.1:33378_solr c:people s:shard1 r:core_node5 x:people_shard1_replica_n2] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 849922 INFO (qtp961437491-2605) [n:127.0.0.1:33378_solr c:people s:shard1 r:core_node5 x:people_shard1_replica_n2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=solrCloudCollectionConfig&newCollection=true&collection=people&version=2&replicaType=NRT&property.config=solrconfig-doctransformers.xml&property.schema=schema-docValuesJoin.xml&coreNodeName=core_node5&name=people_shard1_replica_n2&action=CREATE&numShards=2&shard=shard1&wt=javabin} status=0 QTime=3525
[junit4] 2> 849938 INFO (qtp1118076649-2641) [n:127.0.0.1:43638_solr c:people s:shard2 r:core_node7 x:people_shard2_replica_n4] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 849941 INFO (qtp1118076649-2641) [n:127.0.0.1:43638_solr c:people s:shard2 r:core_node7 x:people_shard2_replica_n4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=solrCloudCollectionConfig&newCollection=true&collection=people&version=2&replicaType=NRT&property.config=solrconfig-doctransformers.xml&property.schema=schema-docValuesJoin.xml&coreNodeName=core_node7&name=people_shard2_replica_n4&action=CREATE&numShards=2&shard=shard2&wt=javabin} status=0 QTime=3745
[junit4] 2> 849951 INFO (qtp854481757-2636) [n:127.0.0.1:44338_solr c:people s:shard1 r:core_node3 x:people_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=solrCloudCollectionConfig&newCollection=true&collection=people&version=2&replicaType=NRT&property.config=solrconfig-doctransformers.xml&property.schema=schema-docValuesJoin.xml&coreNodeName=core_node3&name=people_shard1_replica_n1&action=CREATE&numShards=2&shard=shard1&wt=javabin} status=0 QTime=3576
[junit4] 2> 850025 INFO (qtp1118076649-2646) [n:127.0.0.1:43638_solr ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
[junit4] 2> 850053 INFO (zkCallback-538-thread-2-processing-n:127.0.0.1:33378_solr) [n:127.0.0.1:33378_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5])
[junit4] 2> 850053 INFO (zkCallback-546-thread-1-processing-n:127.0.0.1:44338_solr) [n:127.0.0.1:44338_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5])
[junit4] 2> 850053 INFO (zkCallback-546-thread-2-processing-n:127.0.0.1:44338_solr) [n:127.0.0.1:44338_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5])
[junit4] 2> 850056 INFO (zkCallback-538-thread-1-processing-n:127.0.0.1:33378_solr) [n:127.0.0.1:33378_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5])
[junit4] 2> 850056 INFO (zkCallback-540-thread-2-processing-n:127.0.0.1:43638_solr) [n:127.0.0.1:43638_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5])
[junit4] 2> 850056 INFO (zkCallback-540-thread-1-processing-n:127.0.0.1:43638_solr) [n:127.0.0.1:43638_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5])
[junit4] 2> 850060 INFO (zkCallback-544-thread-2-processing-n:127.0.0.1:43320_solr) [n:127.0.0.1:43320_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5])
[junit4] 2> 850060 INFO (zkCallback-544-thread-1-processing-n:127.0.0.1:43320_solr) [n:127.0.0.1:43320_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/people/state.json] for collection [people] has occurred - updating... (live nodes size: [5])
[junit4] 2> 851032 INFO (qtp1118076649-2646) [n:127.0.0.1:43638_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=2&property.config=solrconfig-doctransformers.xml&property.schema=schema-docValuesJoin.xml&collection.configName=solrCloudCollectionConfig&name=people&nrtReplicas=2&action=CREATE&numShards=2&wt=javabin&version=2} status=0 QTime=5521
[junit4] 2> 851033 INFO (qtp1118076649-2644) [n:127.0.0.1:43638_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=2&property.config=solrconfig-doctransformers.xml&property.schema=schema-docValuesJoin.xml&collection.configName=solrCloudCollectionConfig&name=departments&nrtReplicas=2&action=CREATE&numShards=2&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 851048 INFO (OverseerCollectionConfigSetProcessor-72168899523575820-127.0.0.1:45632_solr-n_0000000000) [n:127.0.0.1:45632_solr ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 851049 INFO (OverseerThreadFactory-1083-thread-2-processing-n:127.0.0.1:45632_solr) [n:127.0.0.1:45632_solr ] o.a.s.c.a.c.CreateCollectionCmd Create collection departments
[junit4] 2> 851335 INFO (OverseerStateUpdate-72168899523575820-127.0.0.1:45632_solr-n_0000000000) [n:127.0.0.1:45632_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"departments",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"departments_shard1_replica_n1",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:45632/solr",
[junit4] 2> "type":"NRT",
[junit4] 2> "waitForFinalState":"false"}
[junit4] 2> 851338 INFO (OverseerStateUpdate-72168899523575820-127.0.0.1:45632_solr-n_0000000000) [n:127.0.0.1:45632_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"departments",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"departments_shard1_replica_n2",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:44338/solr",
[junit4] 2> "type":"NRT",
[junit4] 2> "waitForFinalState":"false"}
[junit4] 2> 851367 INFO (OverseerStateUpdate-72168899523575820-127.0.0.1:45632_solr-n_0000000000) [n:127.0.0.1:45632_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"departments",
[junit4] 2> "shard":"shard2",
[junit4] 2> "core":"departments_shard2_replica_n4",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:43320/solr",
[junit4] 2> "type":"NRT",
[junit4] 2> "waitForFinalState":"false"}
[junit4] 2> 851398 INFO (OverseerStateUpdate-72168899523575820-127.0.0.1:45632_solr-n_0000000000) [n:127.0.0.1:45632_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"departments",
[junit4] 2> "shard":"shard2",
[junit4] 2> "core":"departments_shard2_replica_n6",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:43638/solr",
[junit4] 2> "type":"NRT",
[junit4] 2> "waitForFinalState":"false"}
[junit4] 2> 851826 INFO (qtp194637842-2612) [n:127.0.0.1:43320_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=solrCloudCollectionConfig&newCollection=true&collection=departments&version=2&replicaType=NRT&property.config=solrconfig-doctransformers.xml&property.schema=schema-docValuesJoin.xml&coreNodeName=core_node7&name=departments_shard2_replica_n4&action=CREATE&numShards=2&shard=shard2&wt=javabin
[junit4] 2> 851826 INFO (qtp854481757-2634) [n:127.0.0.1:44338_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=solrCloudCollectionConfig&newCollection=true&collection=departments&version=2&replicaType=NRT&property.config=solrconfig-doctransformers.xml&property.schema=schema-docValuesJoin.xml&coreNodeName=core_node5&name=departments_shard1_replica_n2&action=CREATE&numShards=2&shard=shard1&wt=javabin
[junit4] 2> 851840 INFO (qtp1118076649-2647) [n:127.0.0.1:43638_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=solrCloudCollectionConfig&newCollection=true&collection=departments&version=2&replicaType=NRT&property.config=solrconfig-doctransformers.xml&property.schema=schema-docValuesJoin.xml&coreNodeName=core_node8&name=departments_shard2_replica_n6&action=CREATE&numShards=2&shard=shard2&wt=javabin
[junit4] 2> 851910 INFO (qtp1995350528-2623) [n:127.0.0.1:45632_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=solrCloudCollectionConfig&newCollection=true&collection=departments&version=2&replicaType=NRT&property.config=solrconfig-doctransformers.xml&property.schema=schema-docValuesJoin.xml&coreNodeName=core_node3&name=departments_shard1_replica_n1&action=CREATE&numShards=2&shard=shard1&wt=javabin
[junit4] 2> 851911 INFO (qtp1995350528-2623) [n:127.0.0.1:45632_solr ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 852057 INFO (zkCallback-540-thread-1-processing-n:127.0.0.1:43638_solr) [n:127.0.0.1:43638_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/departments/state.json] for collection [departments] has occurred - updating... (live nodes size: [5])
[junit4] 2> 852057 INFO (zkCallback-544-thread-1-processing-n:127.0.0.1:43320_solr) [n:127.0.0.1:43320_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/departments/state.json] for collection [departments] has occurred - updating... (live nodes size: [5])
[junit4] 2> 852057 INFO (zkCallback-540-thread-2-processing-n:127.0.0.1:43638_solr) [n:127.0.0.1:43638_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/departments/state.json] for collection [departments] has occurred - updating... (live nodes size: [5])
[junit4] 2> 852057 INFO (zkCallback-544-thread-2-processing-n:127.0.0.1:43320_solr) [n:127.0.0.1:43320_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/departments/state.json] for collection [departments] has occurred - updating... (live nodes size: [5])
[junit4] 2> 852058 INFO (zkCallback-542-thread-1-processing-n:127.0.0.1:45632_solr) [n:127.0.0.1:45632_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/departments/state.json] for collection [departments] has occurred - updating... (live nodes size: [5])
[junit4] 2> 852058 INFO (zkCallback-546-thread-2-processing-n:127.0.0.1:44338_solr) [n:127.0.0.1:44338_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/departments/state.json] for collection [departments] has occurred - updating... (live nodes size: [5])
[junit4] 2> 852058 INFO (zkCallback-546-thread-1-processing-n:127.0.0.1:44338_solr) [n:127.0.0.1:44338_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/departments/state.json] for collection [departments] has occurred - updating... (live nodes size: [5])
[junit4] 2> 852076 INFO (zkCallback-542-thread-2-processing-n:127.0.0.1:45632_solr) [n:127.0.0.1:45632_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/departments/state.json] for collection [departments] has occurred - updating... (live nodes size: [5])
[junit4] 2> 853041 INFO (qtp854481757-2634) [n:127.0.0.1:44338_solr c:departments s:shard1 r:core_node5 x:departments_shard1_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
[junit4] 2> 853056 INFO (qtp194637842-2612) [n:127.0.0.1:43320_solr c:departments s:shard2 r:core_node7 x:departments_shard2_replica_n4] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
[junit4] 2> 853094 INFO (qtp1118076649-2647) [n:127.0.0.1:43638_solr c:departments s:shard2 r:core_node8 x:departments_shard2_replica_n6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
[junit4] 2> 853110 INFO (qtp854481757-2634) [n:127.0.0.1:44338_solr c:departments s:shard1 r:core_node5 x:departments_shard1_replica_n2] o.a.s.s.IndexSchema [departments_shard1_replica_n2] Schema name=doc-values-for-Join
[junit4] 2> 853334 INFO (qtp194637842-2612) [n:127.0.0.1:43320_solr c:departments s:shard2 r:core_node7 x:departments_shard2_replica_n4] o.a.s.s.IndexSchema [departments_shard2_replica_n4] Schema name=doc-values-for-Join
[junit4] 2> 853350 INFO (qtp854481757-2634) [n:127.0.0.1:44338_solr c:departments s:shard1 r:core_node5 x:departments_shard1_replica_n2] o.a.s.s.IndexSchema Loaded schema doc-values-for-Join/1.6 with uniqueid field id
[junit4] 2> 853353 INFO (qtp1118076649-2647) [n:127.0.0.1:43638_solr c:departments s:shard2 r:core_node8 x
[...truncated too long message...]
TransformerDistrib (suite) <<<
[junit4] > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.response.transform.TestSubQueryTransformerDistrib:
[junit4] > 1) Thread[id=2605, name=qtp961437491-2605, state=TIMED_WAITING, group=TGRP-TestSubQueryTransformerDistrib]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
[junit4] > at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
[junit4] > at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] > at __randomizedtesting.SeedInfo.seed([12FB931EC67001C6]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
[junit4] > 1) Thread[id=2605, name=qtp961437491-2605, state=TIMED_WAITING, group=TGRP-TestSubQueryTransformerDistrib]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
[junit4] > at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
[junit4] > at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] > at __randomizedtesting.SeedInfo.seed([12FB931EC67001C6]:0)
[junit4] Completed [82/777 (1!)] on J1 in 85.38s, 1 test, 2 errors <<< FAILURES!
[...truncated 5324 lines...]
[junit4] Suite: org.apache.solr.analytics.facet.QueryFacetTest
[junit4] 2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/contrib/solr-analytics/test/J1/temp/solr.analytics.facet.QueryFacetTest_C9F5ABC39B6061B-001/init-core-data-001
[junit4] 2> Feb 23, 2018 5:45:39 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4] 2> WARNING: Will linger awaiting termination of 2 leaked thread(s).
[junit4] 2> Feb 23, 2018 5:45:59 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4] 2> SEVERE: 1 thread leaked from SUITE scope at org.apache.solr.analytics.facet.QueryFacetTest:
[junit4] 2> 1) Thread[id=66, name=qtp777234554-66, state=TIMED_WAITING, group=TGRP-QueryFacetTest]
[junit4] 2> at sun.misc.Unsafe.park(Native Method)
[junit4] 2> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
[junit4] 2> at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
[junit4] 2> at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> Feb 23, 2018 5:45:59 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
[junit4] 2> INFO: Starting to interrupt leaked threads:
[junit4] 2> 1) Thread[id=66, name=qtp777234554-66, state=TIMED_WAITING, group=TGRP-QueryFacetTest]
[junit4] 2> Feb 23, 2018 5:46:02 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
[junit4] 2> SEVERE: There are still zombie threads that couldn't be terminated:
[junit4] 2> 1) Thread[id=66, name=qtp777234554-66, state=TIMED_WAITING, group=TGRP-QueryFacetTest]
[junit4] 2> at sun.misc.Unsafe.park(Native Method)
[junit4] 2> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
[junit4] 2> at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
[junit4] 2> at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene70): {string_s=PostingsFormat(name=Asserting), id=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), string_sm=PostingsFormat(name=Direct)}, docValues:{int_i=DocValuesFormat(name=Lucene70), double_dm=DocValuesFormat(name=Lucene70), float_fm=DocValuesFormat(name=Lucene70), int_im=DocValuesFormat(name=Direct), date_dt=DocValuesFormat(name=Lucene70), date_dtm=DocValuesFormat(name=Direct), long_lm=DocValuesFormat(name=Direct), double_d=DocValuesFormat(name=Lucene70), float_f=DocValuesFormat(name=Lucene70), long_l=DocValuesFormat(name=Asserting), _version_=DocValuesFormat(name=Lucene70), string_s=DocValuesFormat(name=Lucene70), id=DocValuesFormat(name=Asserting), string_sm=DocValuesFormat(name=Direct)}, maxPointsInLeafNode=917, maxMBSortInHeap=7.553277962684154, sim=RandomSimilarity(queryNorm=false): {}, locale=zh-HK, timezone=America/Matamoros
[junit4] 2> NOTE: Linux 4.4.0-112-generic amd64/Oracle Corporation 1.8.0_152 (64-bit)/cpus=4,threads=2,free=253106816,total=303562752
[junit4] 2> NOTE: All tests run in this JVM: [FloorFunctionTest, LegacyExpressionTest, QueryFacetTest]
[junit4] 2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=QueryFacetTest -Dtests.seed=C9F5ABC39B6061B -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/test-data/enwiki.random.lines.txt -Dtests.locale=zh-HK -Dtests.timezone=America/Matamoros -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
[junit4] ERROR 0.00s J1 | QueryFacetTest (suite) <<<
[junit4] > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.analytics.facet.QueryFacetTest:
[junit4] > 1) Thread[id=66, name=qtp777234554-66, state=TIMED_WAITING, group=TGRP-QueryFacetTest]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
[junit4] > at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
[junit4] > at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] > at __randomizedtesting.SeedInfo.seed([C9F5ABC39B6061B]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
[junit4] > 1) Thread[id=66, name=qtp777234554-66, state=TIMED_WAITING, group=TGRP-QueryFacetTest]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
[junit4] > at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
[junit4] > at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] > at __randomizedtesting.SeedInfo.seed([C9F5ABC39B6061B]:0)
[junit4] Completed [63/78 (1!)] on J1 in 93.03s, 1 test, 2 errors <<< FAILURES!
[...truncated 46967 lines...]
[JENKINS] Lucene-Solr-NightlyTests-7.x - Build # 161 - Still
Unstable
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.x/161/
2 tests failed.
FAILED: org.apache.solr.cloud.api.collections.CollectionsAPIAsyncDistributedZkTest.testAsyncRequests
Error Message:
No live SolrServers available to handle this request:[http://127.0.0.1:33622/solr/myalias, http://127.0.0.1:33940/solr/myalias]
Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:33622/solr/myalias, http://127.0.0.1:33940/solr/myalias]
at __randomizedtesting.SeedInfo.seed([9CCB33331EDD4FB4:788F0F84B875016B]:0)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:462)
at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1104)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:942)
at org.apache.solr.cloud.api.collections.CollectionsAPIAsyncDistributedZkTest.testAsyncRequests(CollectionsAPIAsyncDistributedZkTest.java:166)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:33940/solr/myalias: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 404 Can not find: /solr/myalias/select</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing /solr/myalias/select. Reason:
<pre> Can not find: /solr/myalias/select</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.8.v20171121</a><hr/>
</body>
</html>
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:607)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
... 45 more
FAILED: org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload
Error Message:
expected:<[{indexVersion=1519793237444,generation=2,filelist=[_0.fdt, _0.fdx, _0.fnm, _0.nvd, _0.nvm, _0.si, _0_Direct_0.doc, _0_Direct_0.tim, _0_Direct_0.tip, _1.fdt, _1.fdx, _1.fnm, _1.nvd, _1.nvm, _1.si, _1_Direct_0.doc, _1_Direct_0.tim, _1_Direct_0.tip, _2.fdt, _2.fdx, _2.fnm, _2.nvd, _2.nvm, _2.si, _2_Direct_0.doc, _2_Direct_0.tim, _2_Direct_0.tip, _3.fdt, _3.fdx, _3.fnm, _3.nvd, _3.nvm, _3.si, _3_Direct_0.doc, _3_Direct_0.tim, _3_Direct_0.tip, _4.fdt, _4.fdx, _4.fnm, _4.nvd, _4.nvm, _4.si, _4_Direct_0.doc, _4_Direct_0.tim, _4_Direct_0.tip, _5.fdt, _5.fdx, _5.fnm, _5.nvd, _5.nvm, _5.si, _5_Direct_0.doc, _5_Direct_0.tim, _5_Direct_0.tip, segments_2]}]> but was:<[{indexVersion=1519793237444,generation=2,filelist=[_0.fdt, _0.fdx, _0.fnm, _0.nvd, _0.nvm, _0.si, _0_Direct_0.doc, _0_Direct_0.tim, _0_Direct_0.tip, _1.fdt, _1.fdx, _1.fnm, _1.nvd, _1.nvm, _1.si, _1_Direct_0.doc, _1_Direct_0.tim, _1_Direct_0.tip, _2.fdt, _2.fdx, _2.fnm, _2.nvd, _2.nvm, _2.si, _2_Direct_0.doc, _2_Direct_0.tim, _2_Direct_0.tip, _3.fdt, _3.fdx, _3.fnm, _3.nvd, _3.nvm, _3.si, _3_Direct_0.doc, _3_Direct_0.tim, _3_Direct_0.tip, _4.fdt, _4.fdx, _4.fnm, _4.nvd, _4.nvm, _4.si, _4_Direct_0.doc, _4_Direct_0.tim, _4_Direct_0.tip, _5.fdt, _5.fdx, _5.fnm, _5.nvd, _5.nvm, _5.si, _5_Direct_0.doc, _5_Direct_0.tim, _5_Direct_0.tip, segments_2]}, {indexVersion=1519793237444,generation=3,filelist=[_0.fdt, _0.fdx, _0.fnm, _0.nvd, _0.nvm, _0.si, _0_Direct_0.doc, _0_Direct_0.tim, _0_Direct_0.tip, _1.fdt, _1.fdx, _1.fnm, _1.nvd, _1.nvm, _1.si, _1_Direct_0.doc, _1_Direct_0.tim, _1_Direct_0.tip, _2.fdt, _2.fdx, _2.fnm, _2.nvd, _2.nvm, _2.si, _2_Direct_0.doc, _2_Direct_0.tim, _2_Direct_0.tip, _3.fdt, _3.fdx, _3.fnm, _3.nvd, _3.nvm, _3.si, _3_Direct_0.doc, _3_Direct_0.tim, _3_Direct_0.tip, _6.cfe, _6.cfs, _6.si, segments_3]}]>
Stack Trace:
java.lang.AssertionError: expected:<[{indexVersion=1519793237444,generation=2,filelist=[_0.fdt, _0.fdx, _0.fnm, _0.nvd, _0.nvm, _0.si, _0_Direct_0.doc, _0_Direct_0.tim, _0_Direct_0.tip, _1.fdt, _1.fdx, _1.fnm, _1.nvd, _1.nvm, _1.si, _1_Direct_0.doc, _1_Direct_0.tim, _1_Direct_0.tip, _2.fdt, _2.fdx, _2.fnm, _2.nvd, _2.nvm, _2.si, _2_Direct_0.doc, _2_Direct_0.tim, _2_Direct_0.tip, _3.fdt, _3.fdx, _3.fnm, _3.nvd, _3.nvm, _3.si, _3_Direct_0.doc, _3_Direct_0.tim, _3_Direct_0.tip, _4.fdt, _4.fdx, _4.fnm, _4.nvd, _4.nvm, _4.si, _4_Direct_0.doc, _4_Direct_0.tim, _4_Direct_0.tip, _5.fdt, _5.fdx, _5.fnm, _5.nvd, _5.nvm, _5.si, _5_Direct_0.doc, _5_Direct_0.tim, _5_Direct_0.tip, segments_2]}]> but was:<[{indexVersion=1519793237444,generation=2,filelist=[_0.fdt, _0.fdx, _0.fnm, _0.nvd, _0.nvm, _0.si, _0_Direct_0.doc, _0_Direct_0.tim, _0_Direct_0.tip, _1.fdt, _1.fdx, _1.fnm, _1.nvd, _1.nvm, _1.si, _1_Direct_0.doc, _1_Direct_0.tim, _1_Direct_0.tip, _2.fdt, _2.fdx, _2.fnm, _2.nvd, _2.nvm, _2.si, _2_Direct_0.doc, _2_Direct_0.tim, _2_Direct_0.tip, _3.fdt, _3.fdx, _3.fnm, _3.nvd, _3.nvm, _3.si, _3_Direct_0.doc, _3_Direct_0.tim, _3_Direct_0.tip, _4.fdt, _4.fdx, _4.fnm, _4.nvd, _4.nvm, _4.si, _4_Direct_0.doc, _4_Direct_0.tim, _4_Direct_0.tip, _5.fdt, _5.fdx, _5.fnm, _5.nvd, _5.nvm, _5.si, _5_Direct_0.doc, _5_Direct_0.tim, _5_Direct_0.tip, segments_2]}, {indexVersion=1519793237444,generation=3,filelist=[_0.fdt, _0.fdx, _0.fnm, _0.nvd, _0.nvm, _0.si, _0_Direct_0.doc, _0_Direct_0.tim, _0_Direct_0.tip, _1.fdt, _1.fdx, _1.fnm, _1.nvd, _1.nvm, _1.si, _1_Direct_0.doc, _1_Direct_0.tim, _1_Direct_0.tip, _2.fdt, _2.fdx, _2.fnm, _2.nvd, _2.nvm, _2.si, _2_Direct_0.doc, _2_Direct_0.tim, _2_Direct_0.tip, _3.fdt, _3.fdx, _3.fnm, _3.nvd, _3.nvm, _3.si, _3_Direct_0.doc, _3_Direct_0.tim, _3_Direct_0.tip, _6.cfe, _6.cfs, _6.si, segments_3]}]>
at __randomizedtesting.SeedInfo.seed([9CCB33331EDD4FB4:B91C28036E9541B7]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:147)
at org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload(TestReplicationHandler.java:1280)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
Build Log:
[...truncated 13523 lines...]
[junit4] Suite: org.apache.solr.handler.TestReplicationHandler
[junit4] 2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_9CCB33331EDD4FB4-001/init-core-data-001
[junit4] 2> 2365300 WARN (SUITE-TestReplicationHandler-seed#[9CCB33331EDD4FB4]-worker) [ ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=2 numCloses=2
[junit4] 2> 2365300 INFO (SUITE-TestReplicationHandler-seed#[9CCB33331EDD4FB4]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
[junit4] 2> 2365302 INFO (SUITE-TestReplicationHandler-seed#[9CCB33331EDD4FB4]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=None)
[junit4] 2> 2365302 INFO (SUITE-TestReplicationHandler-seed#[9CCB33331EDD4FB4]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
[junit4] 2> 2365304 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.a.s.SolrTestCaseJ4 ###Starting doTestReplicateAfterStartup
[junit4] 2> 2365304 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_9CCB33331EDD4FB4-001/solr-instance-001/collection1
[junit4] 2> 2365308 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T10:27:37-11:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
[junit4] 2> 2365308 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 2365308 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 2365308 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.e.j.s.session Scavenging every 660000ms
[junit4] 2> 2365309 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@2a0225ea{/solr,null,AVAILABLE}
[junit4] 2> 2365311 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@41adac07{HTTP/1.1,[http/1.1]}{127.0.0.1:43778}
[junit4] 2> 2365311 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.e.j.s.Server Started @2376908ms
[junit4] 2> 2365312 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_9CCB33331EDD4FB4-001/solr-instance-001/collection1/data, hostContext=/solr, hostPort=43778}
[junit4] 2> 2365312 ERROR (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 2365312 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 7.3.0
[junit4] 2> 2365312 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in standalone mode on port null
[junit4] 2> 2365312 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 2365312 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-02-28T04:42:11.112Z
[junit4] 2> 2365312 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_9CCB33331EDD4FB4-001/solr-instance-001/solr.xml
[junit4] 2> 2365316 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
[junit4] 2> 2365316 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
[junit4] 2> 2365317 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@4c47df2b, but no JMX reporters were configured - adding default JMX reporter.
[junit4] 2> 2365348 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4c47df2b
[junit4] 2> 2365357 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4c47df2b
[junit4] 2> 2365357 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4c47df2b
[junit4] 2> 2365359 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_9CCB33331EDD4FB4-001/solr-instance-001/.
[junit4] 2> 2365359 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
[junit4] 2> 2365359 INFO (coreLoadExecutor-5575-thread-1) [ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
[junit4] 2> 2365374 INFO (coreLoadExecutor-5575-thread-1) [ x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
[junit4] 2> 2365381 INFO (coreLoadExecutor-5575-thread-1) [ x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
[junit4] 2> 2365383 INFO (coreLoadExecutor-5575-thread-1) [ x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.2 with uniqueid field id
[junit4] 2> 2365383 INFO (coreLoadExecutor-5575-thread-1) [ x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_9CCB33331EDD4FB4-001/solr-instance-001/./collection1, trusted=true
[junit4] 2> 2365383 INFO (coreLoadExecutor-5575-thread-1) [ x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4c47df2b
[junit4] 2> 2365383 INFO (coreLoadExecutor-5575-thread-1) [ x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 2365383 INFO (coreLoadExecutor-5575-thread-1) [ x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_9CCB33331EDD4FB4-001/solr-instance-001/collection1], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_9CCB33331EDD4FB4-001/solr-instance-001/./collection1/data/]
[junit4] 2> 2365384 INFO (coreLoadExecutor-5575-thread-1) [ x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@7cd0574f
[junit4] 2> 2365430 INFO (coreLoadExecutor-5575-thread-1) [ x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 2365430 INFO (coreLoadExecutor-5575-thread-1) [ x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 2365431 INFO (coreLoadExecutor-5575-thread-1) [ x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=2, maxMergeAtOnceExplicit=6, maxMergedSegmentMB=32.0126953125, floorSegmentMB=0.9326171875, forceMergeDeletesPctAllowed=7.198921757157365, segmentsPerTier=15.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5456846020641917
[junit4] 2> 2365431 INFO (coreLoadExecutor-5575-thread-1) [ x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@6e37ea5f[collection1] main]
[junit4] 2> 2365432 INFO (coreLoadExecutor-5575-thread-1) [ x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_9CCB33331EDD4FB4-001/solr-instance-001/collection1/conf
[junit4] 2> 2365432 INFO (coreLoadExecutor-5575-thread-1) [ x:collection1] o.a.s.h.ReplicationHandler Replication enabled for following config files: schema.xml,xslt/dummy.xsl
[junit4] 2> 2365432 INFO (coreLoadExecutor-5575-thread-1) [ x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 2365433 INFO (searcherExecutor-5576-thread-1-processing-x:collection1) [ x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@6e37ea5f[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 2365434 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_9CCB33331EDD4FB4-001/solr-instance-002/collection1
[junit4] 2> 2365437 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T10:27:37-11:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
[junit4] 2> 2365437 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 2365437 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 2365437 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.e.j.s.session Scavenging every 600000ms
[junit4] 2> 2365438 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@2b30b5e{/solr,null,AVAILABLE}
[junit4] 2> 2365438 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@1eee28fa{HTTP/1.1,[http/1.1]}{127.0.0.1:39955}
[junit4] 2> 2365438 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.e.j.s.Server Started @2377034ms
[junit4] 2> 2365438 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_9CCB33331EDD4FB4-001/solr-instance-002/collection1/data, hostContext=/solr, hostPort=39955}
[junit4] 2> 2365438 ERROR (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 2365438 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 7.3.0
[junit4] 2> 2365438 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in standalone mode on port null
[junit4] 2> 2365438 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 2365439 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-02-28T04:42:11.239Z
[junit4] 2> 2365439 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_9CCB33331EDD4FB4-001/solr-instance-002/solr.xml
[junit4] 2> 2365442 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
[junit4] 2> 2365442 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
[junit4] 2> 2365443 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@4c47df2b, but no JMX reporters were configured - adding default JMX reporter.
[junit4] 2> 2365476 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4c47df2b
[junit4] 2> 2365486 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4c47df2b
[junit4] 2> 2365486 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4c47df2b
[junit4] 2> 2365488 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_9CCB33331EDD4FB4-001/solr-instance-002/.
[junit4] 2> 2365488 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
[junit4] 2> 2365488 INFO (coreLoadExecutor-5585-thread-1) [ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
[junit4] 2> 2365498 INFO (coreLoadExecutor-5585-thread-1) [ x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
[junit4] 2> 2365505 INFO (coreLoadExecutor-5585-thread-1) [ x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
[junit4] 2> 2365507 INFO (coreLoadExecutor-5585-thread-1) [ x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.2 with uniqueid field id
[junit4] 2> 2365507 INFO (coreLoadExecutor-5585-thread-1) [ x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_9CCB33331EDD4FB4-001/solr-instance-002/./collection1, trusted=true
[junit4] 2> 2365507 INFO (coreLoadExecutor-5585-thread-1) [ x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4c47df2b
[junit4] 2> 2365507 INFO (coreLoadExecutor-5585-thread-1) [ x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 2365507 INFO (coreLoadExecutor-5585-thread-1) [ x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_9CCB33331EDD4FB4-001/solr-instance-002/collection1], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_9CCB33331EDD4FB4-001/solr-instance-002/./collection1/data/]
[junit4] 2> 2365509 INFO (coreLoadExecutor-5585-thread-1) [ x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@39a02312
[junit4] 2> 2365557 INFO (coreLoadExecutor-5585-thread-1) [ x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 2365557 INFO (coreLoadExecutor-5585-thread-1) [ x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 2365558 INFO (coreLoadExecutor-5585-thread-1) [ x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=2, maxMergeAtOnceExplicit=6, maxMergedSegmentMB=32.0126953125, floorSegmentMB=0.9326171875, forceMergeDeletesPctAllowed=7.198921757157365, segmentsPerTier=15.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5456846020641917
[junit4] 2> 2365558 INFO (coreLoadExecutor-5585-thread-1) [ x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@57ecf0af[collection1] main]
[junit4] 2> 2365559 INFO (coreLoadExecutor-5585-thread-1) [ x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_9CCB33331EDD4FB4-001/solr-instance-002/collection1/conf
[junit4] 2> 2365560 INFO (coreLoadExecutor-5585-thread-1) [ x:collection1] o.a.s.h.ReplicationHandler Poll scheduled at an interval of 1000ms
[junit4] 2> 2365560 INFO (coreLoadExecutor-5585-thread-1) [ x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 2365561 INFO (searcherExecutor-5586-thread-1-processing-x:collection1) [ x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@57ecf0af[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 2365562 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.e.j.s.AbstractConnector Stopped ServerConnector@1eee28fa{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
[junit4] 2> 2365562 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=823271954
[junit4] 2> 2365562 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
[junit4] 2> 2365562 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@4767127f: rootName = null, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@500f6ba5
[junit4] 2> 2365567 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
[junit4] 2> 2365567 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@5a2fec91: rootName = null, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@32bbf439
[junit4] 2> 2365572 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
[junit4] 2> 2365572 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@7c66d833: rootName = null, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@352ae6c6
[junit4] 2> 2365572 INFO (coreCloseExecutor-5591-thread-1) [ x:collection1] o.a.s.c.SolrCore [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@1f3d5aa0
[junit4] 2> 2365573 INFO (coreCloseExecutor-5591-thread-1) [ x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=524114592
[junit4] 2> 2365573 INFO (coreCloseExecutor-5591-thread-1) [ x:collection1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@72b01598: rootName = null, domain = solr.core.collection1, service url = null, agent id = null] for registry solr.core.collection1 / com.codahale.metrics.MetricRegistry@57b26c9a
[junit4] 2> 2365580 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@2b30b5e{/solr,null,UNAVAILABLE}
[junit4] 2> 2365580 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.e.j.s.session Stopped scavenging
[junit4] 2> 2365582 INFO (qtp985153995-25602) [ x:collection1] o.a.s.u.DirectUpdateHandler2 [collection1] REMOVING ALL DOCUMENTS FROM INDEX
[junit4] 2> 2365582 INFO (qtp985153995-25602) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{deleteByQuery=*:*} 0 0
[junit4] 2> 2365582 INFO (qtp985153995-25604) [ x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 2365582 INFO (qtp985153995-25604) [ x:collection1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@7f65e061 commitCommandVersion:0
[junit4] 2> 2365583 INFO (qtp985153995-25604) [ x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@309f0bc6[collection1] main]
[junit4] 2> 2365583 INFO (qtp985153995-25604) [ x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 2365584 INFO (searcherExecutor-5576-thread-1-processing-x:collection1) [ x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@309f0bc6[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 2365584 INFO (qtp985153995-25604) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 1
[junit4] 2> 2365585 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.e.j.s.AbstractConnector Stopped ServerConnector@41adac07{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
[junit4] 2> 2365585 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=212732054
[junit4] 2> 2365586 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
[junit4] 2> 2365586 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@636a5731: rootName = null, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@1968eae5
[junit4] 2> 2365587 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
[junit4] 2> 2365587 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@16bda97e: rootName = null, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@32bbf439
[junit4] 2> 2365588 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
[junit4] 2> 2365588 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@7e1a1542: rootName = null, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@352ae6c6
[junit4] 2> 2365588 INFO (coreCloseExecutor-5593-thread-1) [ x:collection1] o.a.s.c.SolrCore [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@55209a5b
[junit4] 2> 2365589 INFO (coreCloseExecutor-5593-thread-1) [ x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=1428200027
[junit4] 2> 2365589 INFO (coreCloseExecutor-5593-thread-1) [ x:collection1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@1c598591: rootName = null, domain = solr.core.collection1, service url = null, agent id = null] for registry solr.core.collection1 / com.codahale.metrics.MetricRegistry@42f475b1
[junit4] 2> 2365591 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@2a0225ea{/solr,null,UNAVAILABLE}
[junit4] 2> 2365591 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.e.j.s.session Stopped scavenging
[junit4] 2> 2365592 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T10:27:37-11:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
[junit4] 2> 2365592 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 2365592 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 2365592 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.e.j.s.session Scavenging every 660000ms
[junit4] 2> 2365593 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@bbc64a1{/solr,null,AVAILABLE}
[junit4] 2> 2365593 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@7c5a5a75{HTTP/1.1,[http/1.1]}{127.0.0.1:35131}
[junit4] 2> 2365593 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.e.j.s.Server Started @2377189ms
[junit4] 2> 2365593 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_9CCB33331EDD4FB4-001/solr-instance-001/collection1/data, hostContext=/solr, hostPort=35131}
[junit4] 2> 2365593 ERROR (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 2365593 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 7.3.0
[junit4] 2> 2365594 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in standalone mode on port null
[junit4] 2> 2365594 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 2365594 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-02-28T04:42:11.394Z
[junit4] 2> 2365594 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_9CCB33331EDD4FB4-001/solr-instance-001/solr.xml
[junit4] 2> 2365597 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
[junit4] 2> 2365598 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
[junit4] 2> 2365599 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@4c47df2b, but no JMX reporters were configured - adding default JMX reporter.
[junit4] 2> 2365630 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4c47df2b
[junit4] 2> 2365638 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4c47df2b
[junit4] 2> 2365639 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4c47df2b
[junit4] 2> 2365640 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_9CCB33331EDD4FB4-001/solr-instance-001/.
[junit4] 2> 2365640 INFO (TEST-TestReplicationHandler.doTestReplicateAfterStartup-seed#[9CCB33331EDD4FB4]) [ ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
[junit4] 2> 2365641 INFO (coreLoadExecutor-5600-thread-1) [ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
[junit4] 2> 2365651 INFO (coreLoadExecutor-5600-thread-1) [ x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
[junit4] 2> 2365665 INFO (coreLoadExecutor-5600-thread-1) [ x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
[junit4] 2> 2365667 INFO (coreLoadExecutor-5600-thread-1) [ x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.2 with uniqueid field id
[junit4] 2> 2365667 INFO (coreLoadExecutor-5600-thread-1) [ x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_9CCB33331EDD4FB4-001/solr-instance-001/./collection1, trusted=true
[junit4] 2> 2365668 INFO (coreLoadExecutor-5600-thread-1) [ x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4c47df2b
[junit4] 2> 2365668 INFO (coreLoadExecutor-5600-thread-1) [ x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 2365668 INFO (coreLoadExecutor-5600-thread-1) [ x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_9CCB33331EDD4FB4-001/solr-instance-001/collection1], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_9CCB33331EDD4FB4-001/solr-instance-001/./collection1/data/]
[junit4] 2> 2365669 INFO (coreLoadExecutor-5600-thread-1) [ x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@2be8c5a0
[junit4] 2> 2365706 INFO (coreLoadExecutor-5600-thread-1) [ x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 2365706 INFO (coreLoadExecutor-5600-thread-1) [ x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 2365706 INFO (coreLoadExecutor-5600-thread-1) [ x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=2, maxMergeAtOnceExplicit=6, maxMergedSegmentMB=32.0126953125, floorSegmentMB=0.9326171875, forceMergeDeletesPctAllowed=7.198921757157365, segmentsPerTier=15.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5456846020641917
[junit4] 2> 2365707 INFO (coreLoadExecutor-5600-thread-1) [ x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@59a2bdd8[collection1] main]
[junit4] 2> 2365707 INFO (coreLoadExecutor-5600-thread-1) [ x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_9CCB33331EDD4FB4-001/solr-instance-001/collection1/conf
[junit4] 2> 2365708 INFO (coreLoadExecutor-5600-thread-1) [ x:collection1] o.a.s.h.ReplicationHandler Replication enabled for following config files: schema.xml
[junit4] 2> 2365708 INFO (coreLoadExecutor-5600-thread-1) [ x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 2365708 INFO (searcherExecutor-5601-thread-1-processing-x:collection1) [ x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@59a2bdd8[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 2365710 INFO (qtp1078453687-25643) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[0]} 0 0
[junit4] 2> 2365711 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[1]} 0 0
[junit4] 2> 2365711 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[2]} 0 0
[junit4] 2> 2365712 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[3]} 0 0
[junit4] 2> 2365712 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[4]} 0 0
[junit4] 2> 2365712 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[5]} 0 0
[junit4] 2> 2365712 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[6]} 0 0
[junit4] 2> 2365713 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[7]} 0 0
[junit4] 2> 2365713 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[8]} 0 0
[junit4] 2> 2365713 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[9]} 0 0
[junit4] 2> 2365713 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[10]} 0 0
[junit4] 2> 2365714 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[11]} 0 0
[junit4] 2> 2365714 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[12]} 0 0
[junit4] 2> 2365714 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[13]} 0 0
[junit4] 2> 2365714 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[14]} 0 0
[junit4] 2> 2365715 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[15]} 0 0
[junit4] 2> 2365715 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[16]} 0 0
[junit4] 2> 2365715 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[17]} 0 0
[junit4] 2> 2365715 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[18]} 0 0
[junit4] 2> 2365716 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[19]} 0 0
[junit4] 2> 2365716 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[20]} 0 0
[junit4] 2> 2365716 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[21]} 0 0
[junit4] 2> 2365716 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[22]} 0 0
[junit4] 2> 2365717 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[23]} 0 0
[junit4] 2> 2365717 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[24]} 0 0
[junit4] 2> 2365717 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[25]} 0 0
[junit4] 2> 2365717 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[26]} 0 0
[junit4] 2> 2365717 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[27]} 0 0
[junit4] 2> 2365718 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[28]} 0 0
[junit4] 2> 2365718 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[29]} 0 0
[junit4] 2> 2365718 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[30]} 0 0
[junit4] 2> 2365718 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[31]} 0 0
[junit4] 2> 2365719 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[32]} 0 0
[junit4] 2> 2365719 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[33]} 0 0
[junit4] 2> 2365719 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[34]} 0 0
[junit4] 2> 2365719 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[35]} 0 0
[junit4] 2> 2365720 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[36]} 0 0
[junit4] 2> 2365720 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[37]} 0 0
[junit4] 2> 2365720 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[38]} 0 0
[junit4] 2> 2365720 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[39]} 0 0
[junit4] 2> 2365720 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[40]} 0 0
[junit4] 2> 2365721 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[41]} 0 0
[junit4] 2> 2365721 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[42]} 0 0
[junit4] 2> 2365721 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[43]} 0 0
[junit4] 2> 2365721 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[44]} 0 0
[junit4] 2> 2365722 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[45]} 0 0
[junit4] 2> 2365722 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[46]} 0 0
[junit4] 2> 2365722 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[47]} 0 0
[junit4] 2> 2365722 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[48]} 0 0
[junit4] 2> 2365723 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[49]} 0 0
[junit4] 2> 2365723 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[50]} 0 0
[junit4] 2> 2365723 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[51]} 0 0
[junit4] 2> 2365723 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[52]} 0 0
[junit4] 2> 2365723 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[53]} 0 0
[junit4] 2> 2365724 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[54]} 0 0
[junit4] 2> 2365724 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[55]} 0 0
[junit4] 2> 2365724 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[56]} 0 0
[junit4] 2> 2365724 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[57]} 0 0
[junit4] 2> 2365725 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[58]} 0 0
[junit4] 2> 2365725 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[59]} 0 0
[junit4] 2> 2365725 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[60]} 0 0
[junit4] 2> 2365725 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[61]} 0 0
[junit4] 2> 2365726 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[62]} 0 0
[junit4] 2> 2365726 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[63]} 0 0
[junit4] 2> 2365726 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[64]} 0 0
[junit4] 2> 2365726 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[65]} 0 0
[junit4] 2> 2365726 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[66]} 0 0
[junit4] 2> 2365727 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[67]} 0 0
[junit4] 2> 2365727 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[68]} 0 0
[junit4] 2> 2365727 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[69]} 0 0
[junit4] 2> 2365727 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[70]} 0 0
[junit4] 2> 2365728 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[71]} 0 0
[junit4] 2> 2365728 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[72]} 0 0
[junit4] 2> 2365728 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[73]} 0 0
[junit4] 2> 2365728 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[74]} 0 0
[junit4] 2> 2365729 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[75]} 0 0
[junit4] 2> 2365729 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[76]} 0 0
[junit4] 2> 2365729 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[77]} 0 0
[junit4] 2> 2365729 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[78]} 0 0
[junit4] 2> 2365730 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[79]} 0 0
[junit4] 2> 2365730 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[80]} 0 0
[junit4] 2> 2365730 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[81]} 0 0
[junit4] 2> 2365730 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[82]} 0 0
[junit4] 2> 2365731 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[83]} 0 0
[junit4] 2> 2365731 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[84]} 0 0
[junit4] 2> 2365731 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[85]} 0 0
[junit4] 2> 2365732 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[86]} 0 0
[junit4] 2> 2365732 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[87]} 0 0
[junit4] 2> 2365732 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[88]} 0 0
[junit4] 2> 2365732 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[89]} 0 0
[junit4] 2> 2365732 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[90]} 0 0
[junit4] 2> 2365733 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[91]} 0 0
[junit4] 2> 2365733 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[92]} 0 0
[junit4] 2> 2365733 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[93]} 0 0
[junit4] 2> 2365733 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[94]} 0 0
[junit4] 2> 2365734 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[95]} 0 0
[junit4] 2> 2365734 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[96]} 0 0
[junit4] 2> 2365734 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[97]} 0 0
[junit4] 2> 2365734 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[98]} 0 0
[junit4] 2> 2365735 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[99]} 0 0
[junit4] 2> 2365735 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[100]} 0 0
[junit4] 2> 2365735 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[101]} 0 0
[junit4] 2> 2365735 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[102]} 0 0
[junit4] 2> 2365736 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[103]} 0 0
[junit4] 2> 2365736 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[104]} 0 0
[junit4] 2> 2365736 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[105]} 0 0
[junit4] 2> 2365736 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[106]} 0 0
[junit4] 2> 2365737 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[107]} 0 0
[junit4] 2> 2365737 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[108]} 0 0
[junit4] 2> 2365737 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[109]} 0 0
[junit4] 2> 2365737 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[110]} 0 0
[junit4] 2> 2365738 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[111]} 0 0
[junit4] 2> 2365738 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[112]} 0 0
[junit4] 2> 2365738 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[113]} 0 0
[junit4] 2> 2365738 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[114]} 0 0
[junit4] 2> 2365739 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[115]} 0 0
[junit4] 2> 2365739 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[116]} 0 0
[junit4] 2> 2365739 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[117]} 0 0
[junit4] 2> 2365739 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[118]} 0 0
[junit4] 2> 2365740 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[119]} 0 0
[junit4] 2> 2365740 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[120]} 0 0
[junit4] 2> 2365740 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[121]} 0 0
[junit4] 2> 2365740 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[122]} 0 0
[junit4] 2> 2365741 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[123]} 0 0
[junit4] 2> 2365741 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[124]} 0 0
[junit4] 2> 2365741 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[125]} 0 0
[junit4] 2> 2365741 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[126]} 0 0
[junit4] 2> 2365742 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[127]} 0 0
[junit4] 2> 2365742 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[128]} 0 0
[junit4] 2> 2365742 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[129]} 0 0
[junit4] 2> 2365742 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[130]} 0 0
[junit4] 2> 2365743 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[131]} 0 0
[junit4] 2> 2365743 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[132]} 0 0
[junit4] 2> 2365743 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[133]} 0 0
[junit4] 2> 2365743 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[134]} 0 0
[junit4] 2> 2365744 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[135]} 0 0
[junit4] 2> 2365744 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[136]} 0 0
[junit4] 2> 2365744 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[137]} 0 0
[junit4] 2> 2365744 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[138]} 0 0
[junit4] 2> 2365745 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[139]} 0 0
[junit4] 2> 2365745 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[140]} 0 0
[junit4] 2> 2365745 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[141]} 0 0
[junit4] 2> 2365745 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[142]} 0 0
[junit4] 2> 2365746 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[143]} 0 0
[junit4] 2> 2365746 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[144]} 0 0
[junit4] 2> 2365746 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[145]} 0 0
[junit4] 2> 2365746 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[146]} 0 0
[junit4] 2> 2365747 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[147]} 0 0
[junit4] 2> 2365747 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[148]} 0 0
[junit4] 2> 2365747 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[149]} 0 0
[junit4] 2> 2365747 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[150]} 0 0
[junit4] 2> 2365748 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[151]} 0 0
[junit4] 2> 2365748 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[152]} 0 0
[junit4] 2> 2365748 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[153]} 0 0
[junit4] 2> 2365748 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[154]} 0 0
[junit4] 2> 2365749 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[155]} 0 0
[junit4] 2> 2365749 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[156]} 0 0
[junit4] 2> 2365749 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[157]} 0 0
[junit4] 2> 2365749 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[158]} 0 0
[junit4] 2> 2365749 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[159]} 0 0
[junit4] 2> 2365750 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[160]} 0 0
[junit4] 2> 2365750 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[161]} 0 0
[junit4] 2> 2365750 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[162]} 0 0
[junit4] 2> 2365750 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[163]} 0 0
[junit4] 2> 2365751 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[164]} 0 0
[junit4] 2> 2365751 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[165]} 0 0
[junit4] 2> 2365751 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[166]} 0 0
[junit4] 2> 2365751 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[167]} 0 0
[junit4] 2> 2365752 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[168]} 0 0
[junit4] 2> 2365752 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[169]} 0 0
[junit4] 2> 2365752 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[170]} 0 0
[junit4] 2> 2365752 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[171]} 0 0
[junit4] 2> 2365753 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[172]} 0 0
[junit4] 2> 2365753 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[173]} 0 0
[junit4] 2> 2365753 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[174]} 0 0
[junit4] 2> 2365753 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[175]} 0 0
[junit4] 2> 2365754 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[176]} 0 0
[junit4] 2> 2365754 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[177]} 0 0
[junit4] 2> 2365754 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[178]} 0 0
[junit4] 2> 2365754 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[179]} 0 0
[junit4] 2> 2365755 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[180]} 0 0
[junit4] 2> 2365755 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[181]} 0 0
[junit4] 2> 2365755 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[182]} 0 0
[junit4] 2> 2365755 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[183]} 0 0
[junit4] 2> 2365755 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[184]} 0 0
[junit4] 2> 2365756 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[185]} 0 0
[junit4] 2> 2365756 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[186]} 0 0
[junit4] 2> 2365756 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[187]} 0 0
[junit4] 2> 2365756 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[188]} 0 0
[junit4] 2> 2365757 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[189]} 0 0
[junit4] 2> 2365757 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[190]} 0 0
[junit4] 2> 2365757 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[191]} 0 0
[junit4] 2> 2365757 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[192]} 0 0
[junit4] 2> 2365758 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[193]} 0 0
[junit4] 2> 2365758 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[194]} 0 0
[junit4] 2> 2365758 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[195]} 0 0
[junit4] 2> 2365758 INFO (qtp1078453687-25647) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[196]} 0 0
[junit4] 2> 2365759 INFO (qtp1078453687-25645) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={w
[...truncated too long message...]
porters.SolrJmxReporter@29aa3fa1: rootName = solr_33940, domain = solr.core.testasynccollectioncreation.shard1_1.replica0, service url = null, agent id = null] for registry solr.core.testasynccollectioncreation.shard1_1.replica0 / com.codahale.metrics.MetricRegistry@18b7a261
[junit4] 2> 4499146 INFO (coreCloseExecutor-1816-thread-3) [n:127.0.0.1:33622_solr c:testasynccollectioncreation s:shard1_1 r:core_node8 x:testasynccollectioncreation_shard1_1_replica_n6] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.testasynccollectioncreation.shard1_1.leader, tag=715425893
[junit4] 2> 4499146 INFO (coreCloseExecutor-1816-thread-4) [n:127.0.0.1:33622_solr c:testasynccollectioncreation s:shard1_0 r:core_node9 x:testasynccollectioncreation_shard1_0_replica0] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.testasynccollectioncreation.shard1_0.replica0, tag=699709934
[junit4] 2> 4499146 INFO (coreCloseExecutor-1816-thread-4) [n:127.0.0.1:33622_solr c:testasynccollectioncreation s:shard1_0 r:core_node9 x:testasynccollectioncreation_shard1_0_replica0] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@468b87cb: rootName = solr_33622, domain = solr.core.testasynccollectioncreation.shard1_0.replica0, service url = null, agent id = null] for registry solr.core.testasynccollectioncreation.shard1_0.replica0 / com.codahale.metrics.MetricRegistry@f14830
[junit4] 2> 4499147 INFO (coreCloseExecutor-1817-thread-2) [n:127.0.0.1:33940_solr c:testasynccollectioncreation s:shard1_1 r:core_node10 x:testasynccollectioncreation_shard1_1_replica0] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.testasynccollectioncreation.shard1_1.leader, tag=1565384122
[junit4] 2> 4499147 INFO (coreCloseExecutor-1817-thread-3) [n:127.0.0.1:33940_solr c:testAsyncIdRaceCondition s:shard1 r:core_node2 x:testAsyncIdRaceCondition_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.testAsyncIdRaceCondition.shard1.replica_n1, tag=108846950
[junit4] 2> 4499148 INFO (coreCloseExecutor-1817-thread-3) [n:127.0.0.1:33940_solr c:testAsyncIdRaceCondition s:shard1 r:core_node2 x:testAsyncIdRaceCondition_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@7e92136e: rootName = solr_33940, domain = solr.core.testAsyncIdRaceCondition.shard1.replica_n1, service url = null, agent id = null] for registry solr.core.testAsyncIdRaceCondition.shard1.replica_n1 / com.codahale.metrics.MetricRegistry@217aa006
[junit4] 2> 4499492 INFO (coreCloseExecutor-1816-thread-4) [n:127.0.0.1:33622_solr c:testasynccollectioncreation s:shard1_0 r:core_node9 x:testasynccollectioncreation_shard1_0_replica0] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.testasynccollectioncreation.shard1_0.leader, tag=699709934
[junit4] 2> 4499492 INFO (coreCloseExecutor-1817-thread-3) [n:127.0.0.1:33940_solr c:testAsyncIdRaceCondition s:shard1 r:core_node2 x:testAsyncIdRaceCondition_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.testAsyncIdRaceCondition.shard1.leader, tag=108846950
[junit4] 2> 4499493 INFO (coreCloseExecutor-1817-thread-4) [n:127.0.0.1:33940_solr c:testAsyncOperations s:shard1 r:core_node6 x:testAsyncOperations_shard1_replica_n5] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.testAsyncOperations.shard1.replica_n5, tag=1241111240
[junit4] 2> 4499493 INFO (coreCloseExecutor-1817-thread-4) [n:127.0.0.1:33940_solr c:testAsyncOperations s:shard1 r:core_node6 x:testAsyncOperations_shard1_replica_n5] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@1cfdfcc3: rootName = solr_33940, domain = solr.core.testAsyncOperations.shard1.replica_n5, service url = null, agent id = null] for registry solr.core.testAsyncOperations.shard1.replica_n5 / com.codahale.metrics.MetricRegistry@5f8b4405
[junit4] 2> 4499494 INFO (coreCloseExecutor-1816-thread-2) [n:127.0.0.1:33622_solr c:testasynccollectioncreation s:shard1_0 r:core_node7 x:testasynccollectioncreation_shard1_0_replica_n5] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.testasynccollectioncreation.shard1_0.leader, tag=1669410430
[junit4] 2> 4499663 INFO (coreCloseExecutor-1817-thread-4) [n:127.0.0.1:33940_solr c:testAsyncOperations s:shard1 r:core_node6 x:testAsyncOperations_shard1_replica_n5] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.testAsyncOperations.shard1.leader, tag=1241111240
[junit4] 2> 4499711 INFO (jetty-closer-1699-thread-2) [ ] o.a.s.c.Overseer Overseer (id=72197344648626181-127.0.0.1:33622_solr-n_0000000000) closing
[junit4] 2> 4499711 INFO (OverseerStateUpdate-72197344648626181-127.0.0.1:33622_solr-n_0000000000) [n:127.0.0.1:33622_solr ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:33622_solr
[junit4] 2> 4499712 WARN (OverseerAutoScalingTriggerThread-72197344648626181-127.0.0.1:33622_solr-n_0000000000) [n:127.0.0.1:33622_solr ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
[junit4] 2> 4499732 INFO (jetty-closer-1699-thread-1) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@3e273181{/solr,null,UNAVAILABLE}
[junit4] 2> 4499745 INFO (jetty-closer-1699-thread-1) [ ] o.e.j.s.session Stopped scavenging
[junit4] 2> 4501247 INFO (jetty-closer-1699-thread-2) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@35119b40{/solr,null,UNAVAILABLE}
[junit4] 2> 4501247 INFO (jetty-closer-1699-thread-2) [ ] o.e.j.s.session Stopped scavenging
[junit4] 2> 4501259 ERROR (SUITE-CollectionsAPIAsyncDistributedZkTest-seed#[9CCB33331EDD4FB4]-worker) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 4501271 INFO (SUITE-CollectionsAPIAsyncDistributedZkTest-seed#[9CCB33331EDD4FB4]-worker) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:33770 33770
[junit4] 2> 4501393 INFO (Thread-3054) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:33770 33770
[junit4] 2> 4501409 WARN (Thread-3054) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 5 /solr/collections/testAsyncOperations/terms/shard1
[junit4] 2> 5 /solr/collections/testasynccollectioncreation/terms/shard1_1
[junit4] 2> 5 /solr/collections/testasynccollectioncreation/terms/shard1
[junit4] 2> 3 /solr/aliases.json
[junit4] 2> 3 /solr/collections/testasynccollectioncreation/terms/shard1_0
[junit4] 2> 3 /solr/clusterprops.json
[junit4] 2> 2 /solr/collections/testAsyncOperations/terms/shard2
[junit4] 2> 2 /solr/security.json
[junit4] 2> 2 /solr/configs/conf1
[junit4] 2> 2 /solr/collections/testAsyncIdRaceCondition/terms/shard1
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 54 /solr/collections/testasynccollectioncreation/state.json
[junit4] 2> 29 /solr/collections/testAsyncOperations/state.json
[junit4] 2> 6 /solr/collections/testAsyncIdRaceCondition/state.json
[junit4] 2> 3 /solr/clusterstate.json
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 3 /solr/live_nodes
[junit4] 2> 3 /solr/collections
[junit4] 2>
[junit4] 2> NOTE: leaving temporary files on disk at: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.api.collections.CollectionsAPIAsyncDistributedZkTest_9CCB33331EDD4FB4-001
[junit4] 2> Feb 28, 2018 5:17:47 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4] 2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene70): {id=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), _route_=PostingsFormat(name=Direct)}, docValues:{_version_=DocValuesFormat(name=Asserting)}, maxPointsInLeafNode=1851, maxMBSortInHeap=5.0667666858963765, sim=RandomSimilarity(queryNorm=false): {}, locale=sq, timezone=Asia/Thimbu
[junit4] 2> NOTE: Linux 4.4.0-112-generic amd64/Oracle Corporation 1.8.0_152 (64-bit)/cpus=4,threads=1,free=222601688,total=524288000
[junit4] 2> NOTE: All tests run in this JVM: [MissingSegmentRecoveryTest, HdfsAutoAddReplicasIntegrationTest, TestSubQueryTransformerCrossCore, TestHttpShardHandlerFactory, TestStressLucene, BinaryUpdateRequestHandlerTest, TestRandomCollapseQParserPlugin, DefaultValueUpdateProcessorTest, NumberUtilsTest, TestCustomStream, StatsComponentTest, TestLeaderElectionWithEmptyReplica, SpatialRPTFieldTypeTest, TestCSVLoader, TestWriterPerf, BasicAuthStandaloneTest, TestStreamBody, TestPointFields, TestDelegationWithHadoopAuth, SolrCLIZkUtilsTest, TestFieldCacheWithThreads, TestStressCloudBlindAtomicUpdates, SolrPluginUtilsTest, TestClusterStateMutator, TestDefaultStatsCache, SolrRequestParserTest, OutputWriterTest, TestSolrCoreParser, OutOfBoxZkACLAndCredentialsProvidersTest, TestLegacyNumericUtils, TestSolrDeletionPolicy1, DocumentAnalysisRequestHandlerTest, TestSolrConfigHandlerConcurrent, TestConfigSetsAPI, SimpleFacetsTest, TestClusterStateProvider, TestCSVResponseWriter, TestSolrCloudSnapshots, DeleteLastCustomShardedReplicaTest, TestPrepRecovery, TestComponentsName, CheckHdfsIndexTest, CollectionsAPIAsyncDistributedZkTest]
[junit4] Completed [293/777 (2!)] on J0 in 44.20s, 4 tests, 1 error <<< FAILURES!
[...truncated 49545 lines...]
[JENKINS] Lucene-Solr-NightlyTests-7.x - Build # 160 - Still
unstable
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.x/160/
6 tests failed.
FAILED: org.apache.solr.cloud.TestAuthenticationFramework.testBasics
Error Message:
Error from server at https://127.0.0.1:33519/solr/testcollection_shard1_replica_n3: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/> <title>Error 404 Can not find: /solr/testcollection_shard1_replica_n3/update</title> </head> <body><h2>HTTP ERROR 404</h2> <p>Problem accessing /solr/testcollection_shard1_replica_n3/update. Reason: <pre> Can not find: /solr/testcollection_shard1_replica_n3/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.8.v20171121</a><hr/> </body> </html>
Stack Trace:
org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error from server at https://127.0.0.1:33519/solr/testcollection_shard1_replica_n3: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 404 Can not find: /solr/testcollection_shard1_replica_n3/update</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing /solr/testcollection_shard1_replica_n3/update. Reason:
<pre> Can not find: /solr/testcollection_shard1_replica_n3/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.8.v20171121</a><hr/>
</body>
</html>
at __randomizedtesting.SeedInfo.seed([8250FC5E8F2831B0:BF885272B7C66FC0]:0)
at org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:550)
at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1013)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:946)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:946)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:946)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:946)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:946)
at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
at org.apache.solr.client.solrj.request.UpdateRequest.commit(UpdateRequest.java:233)
at org.apache.solr.cloud.TestAuthenticationFramework.collectionCreateSearchDeleteTwice(TestAuthenticationFramework.java:126)
at org.apache.solr.cloud.TestAuthenticationFramework.testBasics(TestAuthenticationFramework.java:74)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:33519/solr/testcollection_shard1_replica_n3: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 404 Can not find: /solr/testcollection_shard1_replica_n3/update</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing /solr/testcollection_shard1_replica_n3/update. Reason:
<pre> Can not find: /solr/testcollection_shard1_replica_n3/update</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.8.v20171121</a><hr/>
</body>
</html>
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:607)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
at org.apache.solr.client.solrj.impl.CloudSolrClient.lambda$directUpdate$0(CloudSolrClient.java:527)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more
FAILED: org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.test_stored_idx
Error Message:
expected:<930> but was:<745>
Stack Trace:
java.lang.AssertionError: expected:<930> but was:<745>
at __randomizedtesting.SeedInfo.seed([8250FC5E8F2831B0:923151A79A8A749E]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.checkField(TestStressCloudBlindAtomicUpdates.java:267)
at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.test_stored_idx(TestStressCloudBlindAtomicUpdates.java:236)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testEventQueue
Error Message:
action wasn't interrupted
Stack Trace:
java.lang.AssertionError: action wasn't interrupted
at __randomizedtesting.SeedInfo.seed([8250FC5E8F2831B0:4BE5BEF0864FF745]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testEventQueue(TriggerIntegrationTest.java:726)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload
Error Message:
expected:<[{indexVersion=1519692498541,generation=2,filelist=[_88.fdt, _88.fdx, _88.fnm, _88.nvd, _88.nvm, _88.si, _88_TestBloomFilteredLucenePostings_0.blm, _88_TestBloomFilteredLucenePostings_0.doc, _88_TestBloomFilteredLucenePostings_0.tim, _88_TestBloomFilteredLucenePostings_0.tip, _89.fdt, _89.fdx, _89.fnm, _89.nvd, _89.nvm, _89.si, _89_TestBloomFilteredLucenePostings_0.blm, _89_TestBloomFilteredLucenePostings_0.doc, _89_TestBloomFilteredLucenePostings_0.tim, _89_TestBloomFilteredLucenePostings_0.tip, _8a.fdt, _8a.fdx, _8a.fnm, _8a.nvd, _8a.nvm, _8a.si, _8a_TestBloomFilteredLucenePostings_0.blm, _8a_TestBloomFilteredLucenePostings_0.doc, _8a_TestBloomFilteredLucenePostings_0.tim, _8a_TestBloomFilteredLucenePostings_0.tip, _8b.fdt, _8b.fdx, _8b.fnm, _8b.nvd, _8b.nvm, _8b.si, _8b_TestBloomFilteredLucenePostings_0.blm, _8b_TestBloomFilteredLucenePostings_0.doc, _8b_TestBloomFilteredLucenePostings_0.tim, _8b_TestBloomFilteredLucenePostings_0.tip, _8c.fdt, _8c.fdx, _8c.fnm, _8c.nvd, _8c.nvm, _8c.si, _8c_TestBloomFilteredLucenePostings_0.blm, _8c_TestBloomFilteredLucenePostings_0.doc, _8c_TestBloomFilteredLucenePostings_0.tim, _8c_TestBloomFilteredLucenePostings_0.tip, _8d.fdt, _8d.fdx, _8d.fnm, _8d.nvd, _8d.nvm, _8d.si, _8d_TestBloomFilteredLucenePostings_0.blm, _8d_TestBloomFilteredLucenePostings_0.doc, _8d_TestBloomFilteredLucenePostings_0.tim, _8d_TestBloomFilteredLucenePostings_0.tip, _8e.fdt, _8e.fdx, _8e.fnm, _8e.nvd, _8e.nvm, _8e.si, _8e_TestBloomFilteredLucenePostings_0.blm, _8e_TestBloomFilteredLucenePostings_0.doc, _8e_TestBloomFilteredLucenePostings_0.tim, _8e_TestBloomFilteredLucenePostings_0.tip, _8f.fdt, _8f.fdx, _8f.fnm, _8f.nvd, _8f.nvm, _8f.si, _8f_TestBloomFilteredLucenePostings_0.blm, _8f_TestBloomFilteredLucenePostings_0.doc, _8f_TestBloomFilteredLucenePostings_0.tim, _8f_TestBloomFilteredLucenePostings_0.tip, _8h.fdt, _8h.fdx, _8h.fnm, _8h.nvd, _8h.nvm, _8h.si, _8h_TestBloomFilteredLucenePostings_0.blm, _8h_TestBloomFilteredLucenePostings_0.doc, _8h_TestBloomFilteredLucenePostings_0.tim, _8h_TestBloomFilteredLucenePostings_0.tip, _8i.fdt, _8i.fdx, _8i.fnm, _8i.nvd, _8i.nvm, _8i.si, _8i_TestBloomFilteredLucenePostings_0.blm, _8i_TestBloomFilteredLucenePostings_0.doc, _8i_TestBloomFilteredLucenePostings_0.tim, _8i_TestBloomFilteredLucenePostings_0.tip, segments_2]}]> but was:<[{indexVersion=1519692498541,generation=2,filelist=[_88.fdt, _88.fdx, _88.fnm, _88.nvd, _88.nvm, _88.si, _88_TestBloomFilteredLucenePostings_0.blm, _88_TestBloomFilteredLucenePostings_0.doc, _88_TestBloomFilteredLucenePostings_0.tim, _88_TestBloomFilteredLucenePostings_0.tip, _89.fdt, _89.fdx, _89.fnm, _89.nvd, _89.nvm, _89.si, _89_TestBloomFilteredLucenePostings_0.blm, _89_TestBloomFilteredLucenePostings_0.doc, _89_TestBloomFilteredLucenePostings_0.tim, _89_TestBloomFilteredLucenePostings_0.tip, _8a.fdt, _8a.fdx, _8a.fnm, _8a.nvd, _8a.nvm, _8a.si, _8a_TestBloomFilteredLucenePostings_0.blm, _8a_TestBloomFilteredLucenePostings_0.doc, _8a_TestBloomFilteredLucenePostings_0.tim, _8a_TestBloomFilteredLucenePostings_0.tip, _8b.fdt, _8b.fdx, _8b.fnm, _8b.nvd, _8b.nvm, _8b.si, _8b_TestBloomFilteredLucenePostings_0.blm, _8b_TestBloomFilteredLucenePostings_0.doc, _8b_TestBloomFilteredLucenePostings_0.tim, _8b_TestBloomFilteredLucenePostings_0.tip, _8c.fdt, _8c.fdx, _8c.fnm, _8c.nvd, _8c.nvm, _8c.si, _8c_TestBloomFilteredLucenePostings_0.blm, _8c_TestBloomFilteredLucenePostings_0.doc, _8c_TestBloomFilteredLucenePostings_0.tim, _8c_TestBloomFilteredLucenePostings_0.tip, _8d.fdt, _8d.fdx, _8d.fnm, _8d.nvd, _8d.nvm, _8d.si, _8d_TestBloomFilteredLucenePostings_0.blm, _8d_TestBloomFilteredLucenePostings_0.doc, _8d_TestBloomFilteredLucenePostings_0.tim, _8d_TestBloomFilteredLucenePostings_0.tip, _8e.fdt, _8e.fdx, _8e.fnm, _8e.nvd, _8e.nvm, _8e.si, _8e_TestBloomFilteredLucenePostings_0.blm, _8e_TestBloomFilteredLucenePostings_0.doc, _8e_TestBloomFilteredLucenePostings_0.tim, _8e_TestBloomFilteredLucenePostings_0.tip, _8f.fdt, _8f.fdx, _8f.fnm, _8f.nvd, _8f.nvm, _8f.si, _8f_TestBloomFilteredLucenePostings_0.blm, _8f_TestBloomFilteredLucenePostings_0.doc, _8f_TestBloomFilteredLucenePostings_0.tim, _8f_TestBloomFilteredLucenePostings_0.tip, _8h.fdt, _8h.fdx, _8h.fnm, _8h.nvd, _8h.nvm, _8h.si, _8h_TestBloomFilteredLucenePostings_0.blm, _8h_TestBloomFilteredLucenePostings_0.doc, _8h_TestBloomFilteredLucenePostings_0.tim, _8h_TestBloomFilteredLucenePostings_0.tip, _8i.fdt, _8i.fdx, _8i.fnm, _8i.nvd, _8i.nvm, _8i.si, _8i_TestBloomFilteredLucenePostings_0.blm, _8i_TestBloomFilteredLucenePostings_0.doc, _8i_TestBloomFilteredLucenePostings_0.tim, _8i_TestBloomFilteredLucenePostings_0.tip, segments_2]}, {indexVersion=1519692498541,generation=3,filelist=[_8g.fdt, _8g.fdx, _8g.fnm, _8g.nvd, _8g.nvm, _8g.si, _8g_TestBloomFilteredLucenePostings_0.blm, _8g_TestBloomFilteredLucenePostings_0.doc, _8g_TestBloomFilteredLucenePostings_0.tim, _8g_TestBloomFilteredLucenePostings_0.tip, _8h.fdt, _8h.fdx, _8h.fnm, _8h.nvd, _8h.nvm, _8h.si, _8h_TestBloomFilteredLucenePostings_0.blm, _8h_TestBloomFilteredLucenePostings_0.doc, _8h_TestBloomFilteredLucenePostings_0.tim, _8h_TestBloomFilteredLucenePostings_0.tip, _8i.fdt, _8i.fdx, _8i.fnm, _8i.nvd, _8i.nvm, _8i.si, _8i_TestBloomFilteredLucenePostings_0.blm, _8i_TestBloomFilteredLucenePostings_0.doc, _8i_TestBloomFilteredLucenePostings_0.tim, _8i_TestBloomFilteredLucenePostings_0.tip, segments_3]}]>
Stack Trace:
java.lang.AssertionError: expected:<[{indexVersion=1519692498541,generation=2,filelist=[_88.fdt, _88.fdx, _88.fnm, _88.nvd, _88.nvm, _88.si, _88_TestBloomFilteredLucenePostings_0.blm, _88_TestBloomFilteredLucenePostings_0.doc, _88_TestBloomFilteredLucenePostings_0.tim, _88_TestBloomFilteredLucenePostings_0.tip, _89.fdt, _89.fdx, _89.fnm, _89.nvd, _89.nvm, _89.si, _89_TestBloomFilteredLucenePostings_0.blm, _89_TestBloomFilteredLucenePostings_0.doc, _89_TestBloomFilteredLucenePostings_0.tim, _89_TestBloomFilteredLucenePostings_0.tip, _8a.fdt, _8a.fdx, _8a.fnm, _8a.nvd, _8a.nvm, _8a.si, _8a_TestBloomFilteredLucenePostings_0.blm, _8a_TestBloomFilteredLucenePostings_0.doc, _8a_TestBloomFilteredLucenePostings_0.tim, _8a_TestBloomFilteredLucenePostings_0.tip, _8b.fdt, _8b.fdx, _8b.fnm, _8b.nvd, _8b.nvm, _8b.si, _8b_TestBloomFilteredLucenePostings_0.blm, _8b_TestBloomFilteredLucenePostings_0.doc, _8b_TestBloomFilteredLucenePostings_0.tim, _8b_TestBloomFilteredLucenePostings_0.tip, _8c.fdt, _8c.fdx, _8c.fnm, _8c.nvd, _8c.nvm, _8c.si, _8c_TestBloomFilteredLucenePostings_0.blm, _8c_TestBloomFilteredLucenePostings_0.doc, _8c_TestBloomFilteredLucenePostings_0.tim, _8c_TestBloomFilteredLucenePostings_0.tip, _8d.fdt, _8d.fdx, _8d.fnm, _8d.nvd, _8d.nvm, _8d.si, _8d_TestBloomFilteredLucenePostings_0.blm, _8d_TestBloomFilteredLucenePostings_0.doc, _8d_TestBloomFilteredLucenePostings_0.tim, _8d_TestBloomFilteredLucenePostings_0.tip, _8e.fdt, _8e.fdx, _8e.fnm, _8e.nvd, _8e.nvm, _8e.si, _8e_TestBloomFilteredLucenePostings_0.blm, _8e_TestBloomFilteredLucenePostings_0.doc, _8e_TestBloomFilteredLucenePostings_0.tim, _8e_TestBloomFilteredLucenePostings_0.tip, _8f.fdt, _8f.fdx, _8f.fnm, _8f.nvd, _8f.nvm, _8f.si, _8f_TestBloomFilteredLucenePostings_0.blm, _8f_TestBloomFilteredLucenePostings_0.doc, _8f_TestBloomFilteredLucenePostings_0.tim, _8f_TestBloomFilteredLucenePostings_0.tip, _8h.fdt, _8h.fdx, _8h.fnm, _8h.nvd, _8h.nvm, _8h.si, _8h_TestBloomFilteredLucenePostings_0.blm, _8h_TestBloomFilteredLucenePostings_0.doc, _8h_TestBloomFilteredLucenePostings_0.tim, _8h_TestBloomFilteredLucenePostings_0.tip, _8i.fdt, _8i.fdx, _8i.fnm, _8i.nvd, _8i.nvm, _8i.si, _8i_TestBloomFilteredLucenePostings_0.blm, _8i_TestBloomFilteredLucenePostings_0.doc, _8i_TestBloomFilteredLucenePostings_0.tim, _8i_TestBloomFilteredLucenePostings_0.tip, segments_2]}]> but was:<[{indexVersion=1519692498541,generation=2,filelist=[_88.fdt, _88.fdx, _88.fnm, _88.nvd, _88.nvm, _88.si, _88_TestBloomFilteredLucenePostings_0.blm, _88_TestBloomFilteredLucenePostings_0.doc, _88_TestBloomFilteredLucenePostings_0.tim, _88_TestBloomFilteredLucenePostings_0.tip, _89.fdt, _89.fdx, _89.fnm, _89.nvd, _89.nvm, _89.si, _89_TestBloomFilteredLucenePostings_0.blm, _89_TestBloomFilteredLucenePostings_0.doc, _89_TestBloomFilteredLucenePostings_0.tim, _89_TestBloomFilteredLucenePostings_0.tip, _8a.fdt, _8a.fdx, _8a.fnm, _8a.nvd, _8a.nvm, _8a.si, _8a_TestBloomFilteredLucenePostings_0.blm, _8a_TestBloomFilteredLucenePostings_0.doc, _8a_TestBloomFilteredLucenePostings_0.tim, _8a_TestBloomFilteredLucenePostings_0.tip, _8b.fdt, _8b.fdx, _8b.fnm, _8b.nvd, _8b.nvm, _8b.si, _8b_TestBloomFilteredLucenePostings_0.blm, _8b_TestBloomFilteredLucenePostings_0.doc, _8b_TestBloomFilteredLucenePostings_0.tim, _8b_TestBloomFilteredLucenePostings_0.tip, _8c.fdt, _8c.fdx, _8c.fnm, _8c.nvd, _8c.nvm, _8c.si, _8c_TestBloomFilteredLucenePostings_0.blm, _8c_TestBloomFilteredLucenePostings_0.doc, _8c_TestBloomFilteredLucenePostings_0.tim, _8c_TestBloomFilteredLucenePostings_0.tip, _8d.fdt, _8d.fdx, _8d.fnm, _8d.nvd, _8d.nvm, _8d.si, _8d_TestBloomFilteredLucenePostings_0.blm, _8d_TestBloomFilteredLucenePostings_0.doc, _8d_TestBloomFilteredLucenePostings_0.tim, _8d_TestBloomFilteredLucenePostings_0.tip, _8e.fdt, _8e.fdx, _8e.fnm, _8e.nvd, _8e.nvm, _8e.si, _8e_TestBloomFilteredLucenePostings_0.blm, _8e_TestBloomFilteredLucenePostings_0.doc, _8e_TestBloomFilteredLucenePostings_0.tim, _8e_TestBloomFilteredLucenePostings_0.tip, _8f.fdt, _8f.fdx, _8f.fnm, _8f.nvd, _8f.nvm, _8f.si, _8f_TestBloomFilteredLucenePostings_0.blm, _8f_TestBloomFilteredLucenePostings_0.doc, _8f_TestBloomFilteredLucenePostings_0.tim, _8f_TestBloomFilteredLucenePostings_0.tip, _8h.fdt, _8h.fdx, _8h.fnm, _8h.nvd, _8h.nvm, _8h.si, _8h_TestBloomFilteredLucenePostings_0.blm, _8h_TestBloomFilteredLucenePostings_0.doc, _8h_TestBloomFilteredLucenePostings_0.tim, _8h_TestBloomFilteredLucenePostings_0.tip, _8i.fdt, _8i.fdx, _8i.fnm, _8i.nvd, _8i.nvm, _8i.si, _8i_TestBloomFilteredLucenePostings_0.blm, _8i_TestBloomFilteredLucenePostings_0.doc, _8i_TestBloomFilteredLucenePostings_0.tim, _8i_TestBloomFilteredLucenePostings_0.tip, segments_2]}, {indexVersion=1519692498541,generation=3,filelist=[_8g.fdt, _8g.fdx, _8g.fnm, _8g.nvd, _8g.nvm, _8g.si, _8g_TestBloomFilteredLucenePostings_0.blm, _8g_TestBloomFilteredLucenePostings_0.doc, _8g_TestBloomFilteredLucenePostings_0.tim, _8g_TestBloomFilteredLucenePostings_0.tip, _8h.fdt, _8h.fdx, _8h.fnm, _8h.nvd, _8h.nvm, _8h.si, _8h_TestBloomFilteredLucenePostings_0.blm, _8h_TestBloomFilteredLucenePostings_0.doc, _8h_TestBloomFilteredLucenePostings_0.tim, _8h_TestBloomFilteredLucenePostings_0.tip, _8i.fdt, _8i.fdx, _8i.fnm, _8i.nvd, _8i.nvm, _8i.si, _8i_TestBloomFilteredLucenePostings_0.blm, _8i_TestBloomFilteredLucenePostings_0.doc, _8i_TestBloomFilteredLucenePostings_0.tim, _8i_TestBloomFilteredLucenePostings_0.tip, segments_3]}]>
at __randomizedtesting.SeedInfo.seed([8250FC5E8F2831B0:A787E76EFF603FB3]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:147)
at org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload(TestReplicationHandler.java:1277)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testSearchRate
Error Message:
The trigger did not fire at all
Stack Trace:
java.lang.AssertionError: The trigger did not fire at all
at __randomizedtesting.SeedInfo.seed([8250FC5E8F2831B0:DF18E2D740EE97FF]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testSearchRate(TestTriggerIntegration.java:1179)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: org.apache.solr.cloud.ReplaceNodeNoTargetTest.test
Error Message:
Stack Trace:
java.lang.AssertionError
at __randomizedtesting.SeedInfo.seed([8250FC5E8F2831B0:A04C38421D45C48]:0)
at org.junit.Assert.fail(Assert.java:92)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertTrue(Assert.java:54)
at org.apache.solr.cloud.ReplaceNodeNoTargetTest.test(ReplaceNodeNoTargetTest.java:92)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
Build Log:
[...truncated 12571 lines...]
[junit4] Suite: org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration
[junit4] 2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.sim.TestTriggerIntegration_8250FC5E8F2831B0-001/init-core-data-001
[junit4] 2> 27430 WARN (SUITE-TestTriggerIntegration-seed#[8250FC5E8F2831B0]-worker) [ ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=2 numCloses=2
[junit4] 2> 27433 INFO (SUITE-TestTriggerIntegration-seed#[8250FC5E8F2831B0]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
[junit4] 2> 27435 INFO (SUITE-TestTriggerIntegration-seed#[8250FC5E8F2831B0]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
[junit4] 2> 27435 INFO (SUITE-TestTriggerIntegration-seed#[8250FC5E8F2831B0]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
[junit4] 2> 27448 INFO (SUITE-TestTriggerIntegration-seed#[8250FC5E8F2831B0]-worker) [ ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
[junit4] 2> 27639 INFO (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[8250FC5E8F2831B0]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testTriggerThrottling
[junit4] 2> 27641 INFO (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[8250FC5E8F2831B0]) [ ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
[junit4] 2> 27641 DEBUG (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[8250FC5E8F2831B0]) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
[junit4] 2> 27671 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
[junit4] 2> 27675 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
[junit4] 2> 27677 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
[junit4] 2> 27732 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr]
[junit4] 2> 27733 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 2
[junit4] 2> 27736 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr]
[junit4] 2> 27737 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion -1
[junit4] 2> 27737 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
[junit4] 2> 27757 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 27758 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2
[junit4] 2> 27759 DEBUG (ScheduledTrigger-360-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 27795 DEBUG (ScheduledTrigger-360-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 27803 DEBUG (simCloudManagerPool-359-thread-1) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 27803 DEBUG (simCloudManagerPool-359-thread-1) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 3
[junit4] 2> 27803 DEBUG (simCloudManagerPool-359-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr]
[junit4] 2> 27807 DEBUG (simCloudManagerPool-359-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr]
[junit4] 2> 27807 DEBUG (simCloudManagerPool-359-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 instantiated with properties: {event=nodeAdded, waitFor=0, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$ThrottlingTesterAction}], enabled=true}
[junit4] 2> 27807 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
[junit4] 2> 27810 INFO (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
[junit4] 2> 27817 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
[junit4] 2> 27821 DEBUG (ScheduledTrigger-360-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 27826 DEBUG (ScheduledTrigger-360-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
[junit4] 2> 27830 DEBUG (ScheduledTrigger-360-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
[junit4] 2> 27828 DEBUG (simCloudManagerPool-359-thread-2) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 27831 DEBUG (simCloudManagerPool-359-thread-2) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 4
[junit4] 2> 27831 DEBUG (simCloudManagerPool-359-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr]
[junit4] 2> 27831 DEBUG (simCloudManagerPool-359-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger2 instantiated with properties: {event=nodeAdded, waitFor=0, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$ThrottlingTesterAction}], enabled=true}
[junit4] 2> 27831 DEBUG (simCloudManagerPool-359-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr]
[junit4] 2> 27831 DEBUG (simCloudManagerPool-359-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr]
[junit4] 2> 27831 DEBUG (simCloudManagerPool-359-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 instantiated with properties: {event=nodeAdded, waitFor=0, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$ThrottlingTesterAction}], enabled=true}
[junit4] 2> 27831 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 4
[junit4] 2> 27831 INFO (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
[junit4] 2> 27831 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion 4
[junit4] 2> 27831 DEBUG (ScheduledTrigger-360-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
[junit4] 2> 27831 DEBUG (ScheduledTrigger-360-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
[junit4] 2> 27842 DEBUG (ScheduledTrigger-360-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 27850 DEBUG (ScheduledTrigger-360-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
[junit4] 2> 27850 DEBUG (ScheduledTrigger-360-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 27851 DEBUG (ScheduledTrigger-360-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10002_solr at time 19944131856138116
[junit4] 2> 27851 DEBUG (ScheduledTrigger-360-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 firing registered processor for nodes: [127.0.0.1:10002_solr] added at times [19944131856138116], now=19944131860953966
[junit4] 2> 27851 DEBUG (ScheduledTrigger-360-thread-3) [ ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
[junit4] 2> "id":"46db150ea37f84T5w8cubiljb630n2wps32kql2",
[junit4] 2> "source":"node_added_trigger1",
[junit4] 2> "eventTime":19944131856138116,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[19944131856138116],
[junit4] 2> "nodeNames":["127.0.0.1:10002_solr"]}}
[junit4] 2> 27851 DEBUG (ScheduledTrigger-360-thread-3) [ ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas, node_added_trigger2, node_added_trigger1]
[junit4] 2> 27857 DEBUG (AutoscalingActionExecutor-361-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
[junit4] 2> "id":"46db150ea37f84T5w8cubiljb630n2wps32kql2",
[junit4] 2> "source":"node_added_trigger1",
[junit4] 2> "eventTime":19944131856138116,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[19944131856138116],
[junit4] 2> "_enqueue_time_":19944131920420266,
[junit4] 2> "nodeNames":["127.0.0.1:10002_solr"]}}
[junit4] 2> 27859 INFO (AutoscalingActionExecutor-361-thread-1) [ ] o.a.s.c.a.s.TestTriggerIntegration action executed from node_added_trigger1
[junit4] 2> 27863 DEBUG (AutoscalingActionExecutor-361-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Resuming trigger: .auto_add_replicas after 100ms
[junit4] 2> 27863 DEBUG (AutoscalingActionExecutor-361-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_trigger2 after 100ms
[junit4] 2> 27863 DEBUG (AutoscalingActionExecutor-361-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_trigger1 after 100ms
[junit4] 2> 27863 DEBUG (AutoscalingActionExecutor-361-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing took 292 ms for event id=46db150ea37f84T5w8cubiljb630n2wps32kql2
[junit4] 2> 28003 DEBUG (ScheduledTrigger-360-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 28003 DEBUG (ScheduledTrigger-360-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
[junit4] 2> 28003 DEBUG (ScheduledTrigger-360-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 28003 DEBUG (ScheduledTrigger-360-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
[junit4] 2> 28003 DEBUG (ScheduledTrigger-360-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 28004 DEBUG (ScheduledTrigger-360-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10002_solr at time 19944139508344366
[junit4] 2> 28004 DEBUG (ScheduledTrigger-360-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger2 firing registered processor for nodes: [127.0.0.1:10002_solr] added at times [19944139508344366], now=19944139512069216
[junit4] 2> 28004 DEBUG (ScheduledTrigger-360-thread-2) [ ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
[junit4] 2> "id":"46db16d6bee62eT5w8cubiljb630n2wps32kql5",
[junit4] 2> "source":"node_added_trigger2",
[junit4] 2> "eventTime":19944139508344366,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[19944139508344366],
[junit4] 2> "nodeNames":["127.0.0.1:10002_solr"]}}
[junit4] 2> 28004 DEBUG (ScheduledTrigger-360-thread-2) [ ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas, node_added_trigger2, node_added_trigger1]
[junit4] 2> 28017 DEBUG (AutoscalingActionExecutor-361-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
[junit4] 2> "id":"46db16d6bee62eT5w8cubiljb630n2wps32kql5",
[junit4] 2> "source":"node_added_trigger2",
[junit4] 2> "eventTime":19944139508344366,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[19944139508344366],
[junit4] 2> "_enqueue_time_":19944139534917916,
[junit4] 2> "nodeNames":["127.0.0.1:10002_solr"]}}
[junit4] 2> 28017 INFO (AutoscalingActionExecutor-361-thread-1) [ ] o.a.s.c.a.s.TestTriggerIntegration last action at 19944132270146866 time = 19944140165999616
[junit4] 2> 28017 INFO (AutoscalingActionExecutor-361-thread-1) [ ] o.a.s.c.a.s.TestTriggerIntegration action executed from node_added_trigger2
[junit4] 2> 28024 DEBUG (simCloudManagerPool-359-thread-6) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 28029 DEBUG (AutoscalingActionExecutor-361-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Resuming trigger: .auto_add_replicas after 100ms
[junit4] 2> 28029 DEBUG (AutoscalingActionExecutor-361-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_trigger2 after 100ms
[junit4] 2> 28029 DEBUG (AutoscalingActionExecutor-361-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_trigger1 after 100ms
[junit4] 2> 28029 DEBUG (AutoscalingActionExecutor-361-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing took 626 ms for event id=46db16d6bee62eT5w8cubiljb630n2wps32kql5
[junit4] 2> 28030 DEBUG (simCloudManagerPool-359-thread-6) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 5
[junit4] 2> 28031 DEBUG (simCloudManagerPool-359-thread-6) [ ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr, 127.0.0.1:10002_solr]
[junit4] 2> 28031 DEBUG (simCloudManagerPool-359-thread-6) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger2 instantiated with properties: {event=nodeAdded, waitFor=0, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$ThrottlingTesterAction}], enabled=true}
[junit4] 2> 28031 DEBUG (simCloudManagerPool-359-thread-6) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr, 127.0.0.1:10002_solr]
[junit4] 2> 28031 DEBUG (simCloudManagerPool-359-thread-6) [ ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr, 127.0.0.1:10002_solr]
[junit4] 2> 28031 DEBUG (simCloudManagerPool-359-thread-6) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 instantiated with properties: {event=nodeAdded, waitFor=0, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$ThrottlingTesterAction}], enabled=true}
[junit4] 2> 28031 DEBUG (simCloudManagerPool-359-thread-6) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr, 127.0.0.1:10002_solr]
[junit4] 2> 28031 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 5
[junit4] 2> 28031 INFO (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
[junit4] 2> 28031 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 5, lastZnodeVersion 5
[junit4] 2> 28031 DEBUG (ScheduledTrigger-360-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 3
[junit4] 2> 28035 DEBUG (simCloudManagerPool-359-thread-8) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 28036 DEBUG (simCloudManagerPool-359-thread-8) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 6
[junit4] 2> 28036 DEBUG (simCloudManagerPool-359-thread-8) [ ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr, 127.0.0.1:10002_solr]
[junit4] 2> 28036 DEBUG (simCloudManagerPool-359-thread-8) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger2 instantiated with properties: {event=nodeAdded, waitFor=0, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$ThrottlingTesterAction}], enabled=true}
[junit4] 2> 28036 DEBUG (simCloudManagerPool-359-thread-8) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr, 127.0.0.1:10002_solr]
[junit4] 2> 28036 DEBUG (simCloudManagerPool-359-thread-8) [ ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr, 127.0.0.1:10002_solr]
[junit4] 2> 28036 DEBUG (simCloudManagerPool-359-thread-8) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 instantiated with properties: {event=nodeAdded, waitFor=0, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$ThrottlingTesterAction}], enabled=true}
[junit4] 2> 28036 DEBUG (simCloudManagerPool-359-thread-8) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr, 127.0.0.1:10002_solr]
[junit4] 2> 28036 DEBUG (simCloudManagerPool-359-thread-8) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr, 127.0.0.1:10002_solr]
[junit4] 2> 28036 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 6
[junit4] 2> 28037 INFO (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
[junit4] 2> 28037 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 6, lastZnodeVersion 6
[junit4] 2> 28037 DEBUG (ScheduledTrigger-360-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger2 with currently live nodes: 3
[junit4] 2> 28052 DEBUG (ScheduledTrigger-360-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 2
[junit4] 2> 28052 DEBUG (ScheduledTrigger-360-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10002_solr
[junit4] 2> 28052 DEBUG (ScheduledTrigger-360-thread-1) [ ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger firing registered processor for lost nodes: [127.0.0.1:10002_solr]
[junit4] 2> 28052 DEBUG (ScheduledTrigger-360-thread-1) [ ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
[junit4] 2> "id":"46db17672637c6T5w8cubiljb630n2wps32kql8",
[junit4] 2> "source":"node_lost_trigger1",
[junit4] 2> "eventTime":19944141931034566,
[junit4] 2> "eventType":"NODELOST",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[19944141931034566],
[junit4] 2> "nodeNames":["127.0.0.1:10002_solr"]}}
[junit4] 2> 28052 DEBUG (ScheduledTrigger-360-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [node_lost_trigger2, .auto_add_replicas, node_added_trigger2, node_lost_trigger1, node_added_trigger1]
[junit4] 2> 28057 DEBUG (AutoscalingActionExecutor-361-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
[junit4] 2> "id":"46db17672637c6T5w8cubiljb630n2wps32kql8",
[junit4] 2> "source":"node_lost_trigger1",
[junit4] 2> "eventTime":19944141931034566,
[junit4] 2> "eventType":"NODELOST",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[19944141931034566],
[junit4] 2> "_enqueue_time_":19944141949211766,
[junit4] 2> "nodeNames":["127.0.0.1:10002_solr"]}}
[junit4] 2> 28057 INFO (AutoscalingActionExecutor-361-thread-1) [ ] o.a.s.c.a.s.TestTriggerIntegration action executed from node_lost_trigger1
[junit4] 2> 28060 DEBUG (AutoscalingActionExecutor-361-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_lost_trigger2 after 100ms
[junit4] 2> 28060 DEBUG (AutoscalingActionExecutor-361-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Resuming trigger: .auto_add_replicas after 100ms
[junit4] 2> 28060 DEBUG (AutoscalingActionExecutor-361-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_trigger2 after 100ms
[junit4] 2> 28060 DEBUG (AutoscalingActionExecutor-361-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_lost_trigger1 after 100ms
[junit4] 2> 28060 DEBUG (AutoscalingActionExecutor-361-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_trigger1 after 100ms
[junit4] 2> 28060 DEBUG (AutoscalingActionExecutor-361-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing took 155 ms for event id=46db17672637c6T5w8cubiljb630n2wps32kql8
[junit4] 2> 28161 DEBUG (ScheduledTrigger-360-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
[junit4] 2> 28161 DEBUG (ScheduledTrigger-360-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
[junit4] 2> 28161 DEBUG (ScheduledTrigger-360-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 2
[junit4] 2> 28161 DEBUG (ScheduledTrigger-360-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
[junit4] 2> 28162 DEBUG (ScheduledTrigger-360-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
[junit4] 2> 28161 DEBUG (ScheduledTrigger-360-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger2 with currently live nodes: 2
[junit4] 2> 28162 DEBUG (ScheduledTrigger-360-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10002_solr
[junit4] 2> 28162 DEBUG (ScheduledTrigger-360-thread-4) [ ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger firing registered processor for lost nodes: [127.0.0.1:10002_solr]
[junit4] 2> 28162 DEBUG (ScheduledTrigger-360-thread-4) [ ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
[junit4] 2> "id":"46db18adf06298T5w8cubiljb630n2wps32kqlb",
[junit4] 2> "source":"node_lost_trigger2",
[junit4] 2> "eventTime":19944147413656216,
[junit4] 2> "eventType":"NODELOST",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[19944147413656216],
[junit4] 2> "nodeNames":["127.0.0.1:10002_solr"]}}
[junit4] 2> 28162 DEBUG (ScheduledTrigger-360-thread-4) [ ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [node_lost_trigger2, .auto_add_replicas, node_added_trigger2, node_lost_trigger1, node_added_trigger1]
[junit4] 2> 28161 DEBUG (ScheduledTrigger-360-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 28162 DEBUG (ScheduledTrigger-360-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10002_solr
[junit4] 2> 28164 DEBUG (AutoscalingActionExecutor-361-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
[junit4] 2> "id":"46db18adf06298T5w8cubiljb630n2wps32kqlb",
[junit4] 2> "source":"node_lost_trigger2",
[junit4] 2> "eventTime":19944147413656216,
[junit4] 2> "eventType":"NODELOST",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[19944147413656216],
[junit4] 2> "_enqueue_time_":19944147428017966,
[junit4] 2> "nodeNames":["127.0.0.1:10002_solr"]}}
[junit4] 2> 28164 INFO (AutoscalingActionExecutor-361-thread-1) [ ] o.a.s.c.a.s.TestTriggerIntegration last action at 19944142192616966 time = 19944147513323866
[junit4] 2> 28164 INFO (AutoscalingActionExecutor-361-thread-1) [ ] o.a.s.c.a.s.TestTriggerIntegration action executed from node_lost_trigger2
[junit4] 2> 28164 INFO (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[8250FC5E8F2831B0]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testTriggerThrottling
[junit4] 2> 28164 INFO (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[8250FC5E8F2831B0]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase
[junit4] 2>
[junit4] 2> 28164 INFO (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[8250FC5E8F2831B0]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase #############################################
[junit4] 2> 28164 INFO (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[8250FC5E8F2831B0]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ############ FINAL CLUSTER STATS ############
[junit4] 2> 28164 INFO (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[8250FC5E8F2831B0]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase #############################################
[junit4] 2>
[junit4] 2> 28164 INFO (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[8250FC5E8F2831B0]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Live nodes: 2
[junit4] 2> 28164 INFO (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[8250FC5E8F2831B0]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Empty nodes: 2
[junit4] 2> 28164 INFO (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[8250FC5E8F2831B0]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Dead nodes: 1
[junit4] 2> 28165 INFO (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[8250FC5E8F2831B0]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## 127.0.0.1:10002_solr
[junit4] 2> 28166 INFO (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[8250FC5E8F2831B0]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Collections: []
[junit4] 2> 28166 INFO (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[8250FC5E8F2831B0]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Max replicas per node: 0
[junit4] 2> 28166 INFO (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[8250FC5E8F2831B0]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Min replicas per node: 0
[junit4] 2> 28166 INFO (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[8250FC5E8F2831B0]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Total replicas: 0
[junit4] 2> 28166 INFO (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[8250FC5E8F2831B0]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ######### Final Solr op counts ##########
[junit4] 2> 28166 INFO (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[8250FC5E8F2831B0]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - autoscaling 4
[junit4] 2> 28166 INFO (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[8250FC5E8F2831B0]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - update 7
[junit4] 2> 28166 INFO (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[8250FC5E8F2831B0]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ######### Autoscaling event counts ###########
[junit4] 2> 28167 INFO (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[8250FC5E8F2831B0]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## * Trigger: node_added_trigger1
[junit4] 2> 28168 INFO (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[8250FC5E8F2831B0]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - STARTED 1
[junit4] 2> 28168 INFO (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[8250FC5E8F2831B0]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - SUCCEEDED 1
[junit4] 2> 28168 INFO (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[8250FC5E8F2831B0]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## * Trigger: node_added_trigger2
[junit4] 2> 28168 INFO (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[8250FC5E8F2831B0]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - STARTED 1
[junit4] 2> 28168 INFO (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[8250FC5E8F2831B0]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - SUCCEEDED 1
[junit4] 2> 28168 INFO (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[8250FC5E8F2831B0]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## * Trigger: node_lost_trigger1
[junit4] 2> 28168 INFO (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[8250FC5E8F2831B0]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - STARTED 1
[junit4] 2> 28168 INFO (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[8250FC5E8F2831B0]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - SUCCEEDED 1
[junit4] 2> 28168 INFO (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[8250FC5E8F2831B0]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## * Trigger: node_lost_trigger2
[junit4] 2> 28168 INFO (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[8250FC5E8F2831B0]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - STARTED 1
[junit4] 2> 28190 DEBUG (AutoscalingActionExecutor-361-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_lost_trigger2 after 100ms
[junit4] 2> 28191 DEBUG (AutoscalingActionExecutor-361-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Resuming trigger: .auto_add_replicas after 100ms
[junit4] 2> 28191 DEBUG (AutoscalingActionExecutor-361-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_trigger2 after 100ms
[junit4] 2> 28191 DEBUG (AutoscalingActionExecutor-361-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_lost_trigger1 after 100ms
[junit4] 2> 28191 DEBUG (AutoscalingActionExecutor-361-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_trigger1 after 100ms
[junit4] 2> 28191 DEBUG (AutoscalingActionExecutor-361-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing took 1354 ms for event id=46db18adf06298T5w8cubiljb630n2wps32kqlb
[junit4] 2> 28193 INFO (TEST-TestTriggerIntegration.testEventFromRestoredState-seed#[8250FC5E8F2831B0]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testEventFromRestoredState
[junit4] 2> 28193 DEBUG (TEST-TestTriggerIntegration.testEventFromRestoredState-seed#[8250FC5E8F2831B0]) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 7
[junit4] 2> 28194 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 7
[junit4] 2> 28194 DEBUG (TEST-TestTriggerIntegration.testEventFromRestoredState-seed#[8250FC5E8F2831B0]) [ ] o.a.s.c.a.s.SimNodeStateProvider - removing dead node values: 127.0.0.1:10002_solr
[junit4] 2> 28194 INFO (TEST-TestTriggerIntegration.testEventFromRestoredState-seed#[8250FC5E8F2831B0]) [ ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
[junit4] 2> 28194 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
[junit4] 2> 28195 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 28195 WARN (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Interrupted
[junit4] 2> java.lang.InterruptedException
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220)
[junit4] 2> at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335)
[junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:163)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 28197 DEBUG (TEST-TestTriggerIntegration.testEventFromRestoredState-seed#[8250FC5E8F2831B0]) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
[junit4] 2> 28206 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
[junit4] 2> 28206 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 8
[junit4] 2> 28206 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr]
[junit4] 2> 28206 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 8, lastZnodeVersion -1
[junit4] 2> 28206 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 8
[junit4] 2> 28213 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 28213 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 8, lastZnodeVersion 8
[junit4] 2> 28213 DEBUG (ScheduledTrigger-363-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 28234 DEBUG (ScheduledTrigger-363-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 28254 DEBUG (ScheduledTrigger-363-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 28274 DEBUG (ScheduledTrigger-363-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 28295 DEBUG (ScheduledTrigger-363-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 28310 DEBUG (simCloudManagerPool-362-thread-1) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 28310 DEBUG (simCloudManagerPool-362-thread-1) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 9
[junit4] 2> 28312 DEBUG (simCloudManagerPool-362-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr]
[junit4] 2> 28313 DEBUG (simCloudManagerPool-362-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr]
[junit4] 2> 28313 DEBUG (simCloudManagerPool-362-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated with properties: {event=nodeAdded, waitFor=10, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerAction}], enabled=true}
[junit4] 2> 28313 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 9
[junit4] 2> 28314 INFO (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
[junit4] 2> 28314 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 9, lastZnodeVersion 9
[junit4] 2> 28314 DEBUG (ScheduledTrigger-363-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 28314 DEBUG (ScheduledTrigger-363-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
[junit4] 2> 28315 DEBUG (ScheduledTrigger-363-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 28334 DEBUG (ScheduledTrigger-363-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 28334 DEBUG (ScheduledTrigger-363-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 28334 DEBUG (ScheduledTrigger-363-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10003_solr at time 19944156036417466
[junit4] 2> 28335 DEBUG (ScheduledTrigger-363-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 28354 DEBUG (ScheduledTrigger-363-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 28354 DEBUG (ScheduledTrigger-363-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 28355 DEBUG (ScheduledTrigger-363-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 28375 DEBUG (ScheduledTrigger-363-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 28376 DEBUG (ScheduledTrigger-363-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 28376 DEBUG (ScheduledTrigger-363-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 28398 DEBUG (ScheduledTrigger-363-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 28399 DEBUG (ScheduledTrigger-363-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 28399 DEBUG (ScheduledTrigger-363-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 28419 DEBUG (ScheduledTrigger-363-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 28419 DEBUG (ScheduledTrigger-363-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 28419 DEBUG (ScheduledTrigger-363-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 28439 DEBUG (ScheduledTrigger-363-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 28439 DEBUG (ScheduledTrigger-363-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 28439 DEBUG (ScheduledTrigger-363-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 28459 DEBUG (ScheduledTrigger-363-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 28459 DEBUG (ScheduledTrigger-363-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 28459 DEBUG (ScheduledTrigger-363-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 28479 DEBUG (ScheduledTrigger-363-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 28479 DEBUG (ScheduledTrigger-363-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 28480 DEBUG (ScheduledTrigger-363-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 28500 DEBUG (ScheduledTrigger-363-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 28500 DEBUG (ScheduledTrigger-363-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 28500 DEBUG (ScheduledTrigger-363-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 28521 DEBUG (ScheduledTrigger-363-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 28522 DEBUG (ScheduledTrigger-363-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 28522 DEBUG (ScheduledTrigger-363-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 28542 DEBUG (ScheduledTrigger-363-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 28542 DEBUG (ScheduledTrigger-363-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 28542 DEBUG (ScheduledTrigger-363-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 28542 DEBUG (ScheduledTrigger-363-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger firing registered processor for nodes: [127.0.0.1:10003_solr] added at times [19944156036417466], now=19944166421504766
[junit4] 2> 28542 DEBUG (ScheduledTrigger-363-thread-3) [ ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
[junit4] 2> "id":"46db1aafe547baT5w8cubiljb630n2wps32kqle",
[junit4] 2> "source":"node_added_trigger",
[junit4] 2> "eventTime":19944156036417466,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[19944156036417466],
[junit4] 2> "nodeNames":["127.0.0.1:10003_solr"]}}
[junit4] 2> 28542 DEBUG (ScheduledTrigger-363-thread-3) [ ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas, node_added_trigger]
[junit4] 2> 28560 DEBUG (AutoscalingActionExecutor-364-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
[junit4] 2> "id":"46db1aafe547baT5w8cubiljb630n2wps32kqle",
[junit4] 2> "source":"node_added_trigger",
[junit4] 2> "eventTime":19944156036417466,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[19944156036417466],
[junit4] 2> "_enqueue_time_":19944166441280416,
[junit4] 2> "nodeNames":["127.0.0.1:10003_solr"]}}
[junit4] 2> 28562 DEBUG (AutoscalingActionExecutor-364-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Resuming trigger: .auto_add_replicas after 100ms
[junit4] 2> 28562 DEBUG (AutoscalingActionExecutor-364-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_trigger after 100ms
[junit4] 2> 28562 DEBUG (AutoscalingActionExecutor-364-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing took 83 ms for event id=46db1aafe547baT5w8cubiljb630n2wps32kqle
[junit4] 2> 28662 DEBUG (ScheduledTrigger-363-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 4
[junit4] 2> 28662 DEBUG (ScheduledTrigger-363-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 28662 DEBUG (ScheduledTrigger-363-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
[junit4] 2> 28662 DEBUG (ScheduledTrigger-363-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10004_solr at time 19944172449156116
[junit4] 2> 28682 DEBUG (ScheduledTrigger-363-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 4
[junit4] 2> 28683 DEBUG (ScheduledTrigger-363-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 28683 DEBUG (ScheduledTrigger-363-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
[junit4] 2> 28703 DEBUG (ScheduledTrigger-363-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 4
[junit4] 2> 28703 DEBUG (ScheduledTrigger-363-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 28703 DEBUG (ScheduledTrigger-363-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
[junit4] 2> 28723 DEBUG (ScheduledTrigger-363-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 4
[junit4] 2> 28723 DEBUG (ScheduledTrigger-363-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 28723 DEBUG (ScheduledTrigger-363-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
[junit4] 2> 28743 DEBUG (ScheduledTrigger-363-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 4
[junit4] 2> 28743 DEBUG (ScheduledTrigger-363-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 28743 DEBUG (ScheduledTrigger-363-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
[junit4] 2> 28761 INFO (TEST-TestTriggerIntegration.testEventFromRestoredState-seed#[8250FC5E8F2831B0]) [ ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
[junit4] 2> 28762 WARN (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
[junit4] 2> 28762 DEBUG (TEST-TestTriggerIntegration.testEventFromRestoredState-seed#[8250FC5E8F2831B0]) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
[junit4] 2> 28763 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 9
[junit4] 2> 28764 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10004_solr, 127.0.0.1:10000_solr, 127.0.0.1:10003_solr]
[junit4] 2> 28764 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10004_solr, 127.0.0.1:10000_solr, 127.0.0.1:10003_solr]
[junit4] 2> 28764 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated with properties: {event=nodeAdded, waitFor=10, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerAction}], enabled=true}
[junit4] 2> 28764 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 9, lastZnodeVersion -1
[junit4] 2> 28764 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 9
[junit4] 2> 28765 INFO (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
[junit4] 2> 28765 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeAddedTrigger Adding node from marker path: 127.0.0.1:10004_solr
[junit4] 2> 28765 DEBUG (ScheduledTrigger-366-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 4
[junit4] 2> 28766 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 9, lastZnodeVersion 9
[junit4] 2> 28766 DEBUG (ScheduledTrigger-366-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 28766 DEBUG (ScheduledTrigger-366-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
[junit4] 2> 28786 DEBUG (ScheduledTrigger-366-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 4
[junit4] 2> 28787 DEBUG (ScheduledTrigger-366-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 28787 DEBUG (ScheduledTrigger-366-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
[junit4] 2> 28806 DEBUG (ScheduledTrigger-366-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 4
[junit4] 2> 28807 DEBUG (ScheduledTrigger-366-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 28807 DEBUG (ScheduledTrigger-366-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
[junit4] 2> 28826 DEBUG (ScheduledTrigger-366-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 4
[junit4] 2> 28829 DEBUG (ScheduledTrigger-366-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 28829 DEBUG (ScheduledTrigger-366-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
[junit4] 2> 28851 DEBUG (ScheduledTrigger-366-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 28851 DEBUG (ScheduledTrigger-366-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
[junit4] 2> 28851 DEBUG (ScheduledTrigger-366-thread-4) [ ] o.a.s.c.
[...truncated too long message...]
> 4591606 INFO (jetty-closer-6814-thread-3) [ ] o.e.j.s.session Stopped scavenging
[junit4] 2> 4593098 WARN (zkCallback-7102-thread-2-processing-n:127.0.0.1:48192_solr) [n:127.0.0.1:48192_solr ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
[junit4] 2> 4593099 INFO (jetty-closer-6814-thread-4) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@32b9c764{/solr,null,UNAVAILABLE}
[junit4] 2> 4593099 INFO (jetty-closer-6814-thread-4) [ ] o.e.j.s.session Stopped scavenging
[junit4] 2> 4593099 ERROR (SUITE-TriggerIntegrationTest-seed#[8250FC5E8F2831B0]-worker) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 4593100 INFO (SUITE-TriggerIntegrationTest-seed#[8250FC5E8F2831B0]-worker) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:53192 53192
[junit4] 2> 4593266 INFO (Thread-20917) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:53192 53192
[junit4] 2> 4593268 WARN (Thread-20917) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 42 /solr/aliases.json
[junit4] 2> 42 /solr/clusterprops.json
[junit4] 2> 39 /solr/collections/collection1/terms/shard1
[junit4] 2> 27 /solr/security.json
[junit4] 2> 21 /solr/collections/testMetricTrigger/terms/shard1
[junit4] 2> 11 /solr/configs/conf
[junit4] 2> 7 /solr/collections/testMetricTrigger/terms/shard2
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 67 /solr/collections/testMetricTrigger/state.json
[junit4] 2> 53 /solr/collections/collection1/state.json
[junit4] 2> 42 /solr/clusterstate.json
[junit4] 2> 20 /solr/autoscaling.json
[junit4] 2> 2 /solr/overseer_elect/election/73364932001136687-127.0.0.1:47375_solr-n_0000000014
[junit4] 2> 2 /solr/collections/testMetricTrigger/leader_elect/shard1/election/73364932001136711-core_node5-n_0000000000
[junit4] 2> 2 /solr/collections/collection1/leader_elect/shard1/election/73364932001136663-core_node3-n_0000000000
[junit4] 2> 2 /solr/overseer_elect/election/73364932001136672-127.0.0.1:49836_solr-n_0000000009
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 352 /solr/live_nodes
[junit4] 2> 72 /solr/collections
[junit4] 2> 20 /solr/overseer/queue-work
[junit4] 2> 20 /solr/autoscaling/events/.auto_add_replicas
[junit4] 2> 20 /solr/overseer/queue
[junit4] 2> 20 /solr/overseer/collection-queue-work
[junit4] 2> 2 /solr/autoscaling/events/node_lost_triggerMR
[junit4] 2> 2 /solr/autoscaling/events/node_added_triggerCTOOR
[junit4] 2>
[junit4] 2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.TriggerIntegrationTest_8250FC5E8F2831B0-001
[junit4] 2> Feb 27, 2018 1:59:30 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4] 2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
[junit4] 2> NOTE: test params are: codec=CheapBastard, sim=RandomSimilarity(queryNorm=false): {}, locale=zh-HK, timezone=America/Creston
[junit4] 2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=4,threads=1,free=96407936,total=509083648
[junit4] 2> NOTE: All tests run in this JVM: [TestRecovery, TestFoldingMultitermQuery, TestTriggerIntegration, UpdateRequestProcessorFactoryTest, TestSimpleTrackingShardHandler, TestFieldCache, TestTolerantUpdateProcessorCloud, SolrGangliaReporterTest, TestNumericRangeQuery64, FileUtilsTest, TestGenericDistributedQueue, RequiredFieldsTest, DistributedMLTComponentTest, DistributedFacetSimpleRefinementLongTailTest, NodeLostTriggerTest, LoggingHandlerTest, SolrInfoBeanTest, TestLegacyField, BlockJoinFacetSimpleTest, ZkStateReaderTest, TestSweetSpotSimilarityFactory, JsonLoaderTest, LegacyCloudClusterPropTest, BlockCacheTest, SimplePostToolTest, HdfsThreadLeakTest, MoveReplicaHDFSFailoverTest, HttpTriggerListenerTest, TestFileDictionaryLookup, TestReqParamsAPI, PrimUtilsTest, CdcrUpdateLogTest, UninvertDocValuesMergePolicyTest, TestSmileRequest, CurrencyFieldTypeTest, TestPayloadScoreQParserPlugin, TestReplicationHandler, HttpSolrCallGetCoreTest, TestRTGBase, QueryEqualityTest, TestSolrCoreSnapshots, DeleteNodeTest, BasicFunctionalityTest, OverseerRolesTest, ZkControllerTest, ReplaceNodeNoTargetTest, TestCloudDeleteByQuery, HLLSerializationTest, PeerSyncTest, AsyncCallRequestStatusResponseTest, TestNamedUpdateProcessors, TestPostingsSolrHighlighter, TestJettySolrRunner, SystemInfoHandlerTest, AddReplicaTest, TestRandomRequestDistribution, TestMacroExpander, BitVectorTest, PreAnalyzedUpdateProcessorTest, TestXIncludeConfig, TestComplexPhraseLeadingWildcard, SuggesterTest, AnalyticsQueryTest, ClassificationUpdateProcessorIntegrationTest, TestSizeLimitedDistributedMap, DistributedFacetPivotLargeTest, AlternateDirectoryTest, BasicZkTest, UpdateLogTest, HdfsCollectionsAPIDistributedZkTest, TestConfigSets, TestHashQParserPlugin, DistribJoinFromCollectionTest, TestBulkSchemaConcurrent, PolyFieldTest, FastVectorHighlighterTest, SolrCoreTest, TestSQLHandlerNonCloud, TestDeleteCollectionOnDownNodes, TestReloadDeadlock, ResourceLoaderTest, DistributedDebugComponentTest, SolrShardReporterTest, TestSolrCLIRunExample, HighlighterTest, HdfsWriteToMultipleCollectionsTest, TestCloudPivotFacet, TestMiniSolrCloudClusterSSL, TestDistributedStatsComponentCardinality, TestFunctionQuery, PKIAuthenticationIntegrationTest, TestSolrConfigHandler, CoreSorterTest, TestLRUStatsCache, SolrMetricManagerTest, TestLocalFSCloudBackupRestore, SchemaApiFailureTest, TestRuleBasedAuthorizationPlugin, TestInfoStreamLogging, ShardRoutingCustomTest, TestReload, TestSolr4Spatial, TestPullReplica, SampleTest, TestStressInPlaceUpdates, TestElisionMultitermQuery, DirectUpdateHandlerOptimizeTest, TestSystemIdResolver, AutoscalingHistoryHandlerTest, ZkFailoverTest, TestSearchPerf, DOMUtilTest, TestComputePlanAction, SpatialHeatmapFacetsTest, SortSpecParsingTest, DocValuesNotIndexedTest, TestCloudSearcherWarming, AtomicUpdateProcessorFactoryTest, TestSolrCloudWithKerberosAlt, TestManagedSchemaAPI, QueryParsingTest, TestReloadAndDeleteDocs, TestConfigSetsAPIExclusivity, TlogReplayBufferedWhileIndexingTest, SuggesterFSTTest, DocumentBuilderTest, ChaosMonkeySafeLeaderTest, BasicDistributedZk2Test, UnloadDistributedZkTest, SyncSliceTest, OverseerTest, TestRandomFaceting, TestZkChroot, TestRandomDVFaceting, TestDistributedGrouping, TestHashPartitioner, DistributedSpellCheckComponentTest, TestStressVersions, TestCoreContainer, TestGroupingSearch, QueryElevationComponentTest, BadIndexSchemaTest, ConvertedLegacyTest, DirectUpdateHandlerTest, SoftAutoCommitTest, TestCoreDiscovery, SignatureUpdateProcessorFactoryTest, DocValuesTest, TestTrie, SuggesterWFSTTest, SchemaVersionSpecificBehaviorTest, TestUpdate, FieldMutatingUpdateProcessorTest, TestAtomicUpdateErrorCases, TestWordDelimiterFilterFactory, SortByFunctionTest, TestRemoteStreaming, DocValuesMultiTest, XsltUpdateRequestHandlerTest, DebugComponentTest, CacheHeaderTest, TestSurroundQueryParser, LukeRequestHandlerTest, TestQueryUtils, TestReversedWildcardFilterFactory, PrimitiveFieldTypeTest, TermsComponentTest, RequestHandlersTest, TestValueSourceCache, PathHierarchyTokenizerFactoryTest, TestIndexingPerformance, FieldAnalysisRequestHandlerTest, RegexBoostProcessorTest, IndexSchemaTest, TestJmxIntegration, ReturnFieldsTest, MBeansHandlerTest, UniqFieldsUpdateProcessorFactoryTest, BinaryUpdateRequestHandlerTest, TestPartialUpdateDeduplication, CSVRequestHandlerTest, SearchHandlerTest, TestLFUCache, SOLR749Test, UpdateParamsTest, TestSolrIndexConfig, CopyFieldTest, ResponseLogComponentTest, SolrIndexConfigTest, BadComponentTest, TestStressRecovery, TestMergePolicyConfig, TestSolrDeletionPolicy2, TestDocSet, TestBinaryField, MinimalSchemaTest, TestFuzzyAnalyzedSuggestions, TestSolrCoreProperties, DirectSolrConnectionTest, TestPhraseSuggestions, SpellPossibilityIteratorTest, TestCharFilters, TestCodecSupport, SynonymTokenizerTest, EchoParamsTest, TestDFRSimilarityFactory, TestPerFieldSimilarity, TestLMJelinekMercerSimilarityFactory, TestIBSimilarityFactory, TestBM25SimilarityFactory, TimeZoneUtilsTest, ScriptEngineTest, URLClassifyProcessorTest, TestFastLRUCache, ChaosMonkeyNothingIsSafeTest, DateFieldTest, SpellingQueryConverterTest, RAMDirectoryFactoryTest, ClusterStateTest, TestSolrJ, TestLRUCache, ZkNodePropsTest, SliceStateTest, UUIDFieldTest, DistributedIntervalFacetingTest, TestCrossCoreJoin, TestCursorMarkWithoutUniqueKey, TestDistributedMissingSort, TestTolerantSearch, TestLuceneIndexBackCompat, TestEmbeddedSolrServerAdminHandler, TestEmbeddedSolrServerConstructors, ConnectionReuseTest, ActionThrottleTest, AssignBackwardCompatibilityTest, ChaosMonkeyNothingIsSafeWithPullReplicasTest, ConfigSetsAPITest, CreateCollectionCleanupTest, DeleteInactiveReplicaTest, DeleteReplicaTest, DistribCursorPagingTest, DistributedQueueTest, HttpPartitionTest, LeaderFailureAfterFreshStartTest, MultiThreadedOCPTest, RemoteQueryErrorTest, ReplicationFactorTest, RestartWhileUpdatingTest, TestStressCloudBlindAtomicUpdates, TestTolerantUpdateProcessorRandomCloud, VMParamsZkACLAndCredentialsProvidersTest, CollectionReloadTest, CollectionsAPIDistributedZkTest, TestCollectionAPI, TestReplicaProperties, AutoAddReplicasPlanActionTest, ComputePlanActionTest, TestPolicyCloud, TriggerIntegrationTest]
[junit4] Completed [580/777 (6!)] on J1 in 283.06s, 14 tests, 1 failure <<< FAILURES!
[...truncated 48492 lines...]
[JENKINS] Lucene-Solr-NightlyTests-7.x - Build # 159 - Failure
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.x/159/
8 tests failed.
FAILED: org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.test
Error Message:
Test abandoned because suite timeout was reached.
Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
at __randomizedtesting.SeedInfo.seed([6E8117FDF9225FF5]:0)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
Error Message:
Suite timeout exceeded (>= 7200000 msec).
Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
at __randomizedtesting.SeedInfo.seed([6E8117FDF9225FF5]:0)
FAILED: org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testSearchRate
Error Message:
The trigger did not fire at all
Stack Trace:
java.lang.AssertionError: The trigger did not fire at all
at __randomizedtesting.SeedInfo.seed([6E8117FDF9225FF5:33C9097436E4F9BA]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testSearchRate(TestTriggerIntegration.java:1179)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: org.apache.solr.cloud.hdfs.HdfsBasicDistributedZk2Test.test
Error Message:
Doc Counts do not add up expected:<67> but was:<36>
Stack Trace:
java.lang.AssertionError: Doc Counts do not add up expected:<67> but was:<36>
at __randomizedtesting.SeedInfo.seed([6E8117FDF9225FF5:E6D5282757DE320D]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.assertDocCounts(AbstractFullDistribZkTestBase.java:1453)
at org.apache.solr.cloud.BasicDistributedZk2Test.test(BasicDistributedZk2Test.java:101)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: junit.framework.TestSuite.org.apache.solr.metrics.reporters.SolrJmxReporterCloudTest
Error Message:
Clean up static fields (in @AfterClass?), your test seems to hang on to approximately 26,773,928 bytes (threshold is 10,485,760). Field reference sizes (counted individually): - 26,773,312 bytes, private static javax.management.MBeanServer org.apache.solr.metrics.reporters.SolrJmxReporterCloudTest.mBeanServer - 192 bytes, public static org.junit.rules.TestRule org.apache.solr.SolrTestCaseJ4.solrClassRules - 128 bytes, private static java.lang.String org.apache.solr.SolrTestCaseJ4.factoryProp - 112 bytes, private static java.lang.String org.apache.solr.metrics.reporters.SolrJmxReporterCloudTest.COLLECTION - 72 bytes, private static java.util.Map org.apache.solr.SolrTestCaseJ4.savedClassLogLevels - 64 bytes, private static java.lang.String org.apache.solr.SolrTestCaseJ4.coreName - 48 bytes, private static java.lang.String org.apache.solr.SolrTestCaseJ4.initialRootLogLevel
Stack Trace:
junit.framework.AssertionFailedError: Clean up static fields (in @AfterClass?), your test seems to hang on to approximately 26,773,928 bytes (threshold is 10,485,760). Field reference sizes (counted individually):
- 26,773,312 bytes, private static javax.management.MBeanServer org.apache.solr.metrics.reporters.SolrJmxReporterCloudTest.mBeanServer
- 192 bytes, public static org.junit.rules.TestRule org.apache.solr.SolrTestCaseJ4.solrClassRules
- 128 bytes, private static java.lang.String org.apache.solr.SolrTestCaseJ4.factoryProp
- 112 bytes, private static java.lang.String org.apache.solr.metrics.reporters.SolrJmxReporterCloudTest.COLLECTION
- 72 bytes, private static java.util.Map org.apache.solr.SolrTestCaseJ4.savedClassLogLevels
- 64 bytes, private static java.lang.String org.apache.solr.SolrTestCaseJ4.coreName
- 48 bytes, private static java.lang.String org.apache.solr.SolrTestCaseJ4.initialRootLogLevel
at __randomizedtesting.SeedInfo.seed([6E8117FDF9225FF5]:0)
at com.carrotsearch.randomizedtesting.rules.StaticFieldsInvariantRule$1.afterAlways(StaticFieldsInvariantRule.java:170)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: junit.framework.TestSuite.org.apache.solr.client.solrj.request.TestV2Request
Error Message:
1 thread leaked from SUITE scope at org.apache.solr.client.solrj.request.TestV2Request: 1) Thread[id=56, name=qtp297481271-56, state=TIMED_WAITING, group=TGRP-TestV2Request] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.client.solrj.request.TestV2Request:
1) Thread[id=56, name=qtp297481271-56, state=TIMED_WAITING, group=TGRP-TestV2Request]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
at __randomizedtesting.SeedInfo.seed([9384056312F5B55]:0)
FAILED: junit.framework.TestSuite.org.apache.solr.client.solrj.request.TestV2Request
Error Message:
There are still zombie threads that couldn't be terminated: 1) Thread[id=56, name=qtp297481271-56, state=TIMED_WAITING, group=TGRP-TestV2Request] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
1) Thread[id=56, name=qtp297481271-56, state=TIMED_WAITING, group=TGRP-TestV2Request]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
at __randomizedtesting.SeedInfo.seed([9384056312F5B55]:0)
FAILED: org.apache.solr.cloud.ReplaceNodeNoTargetTest.test
Error Message:
Stack Trace:
java.lang.AssertionError
at __randomizedtesting.SeedInfo.seed([6E8117FDF9225FF5:E6D5282757DE320D]:0)
at org.junit.Assert.fail(Assert.java:92)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertTrue(Assert.java:54)
at org.apache.solr.cloud.ReplaceNodeNoTargetTest.test(ReplaceNodeNoTargetTest.java:92)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
Build Log:
[...truncated 13169 lines...]
[junit4] Suite: org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration
[junit4] 2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.sim.TestTriggerIntegration_6E8117FDF9225FF5-001/init-core-data-001
[junit4] 2> 1787746 WARN (SUITE-TestTriggerIntegration-seed#[6E8117FDF9225FF5]-worker) [ ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=1 numCloses=1
[junit4] 2> 1787747 INFO (SUITE-TestTriggerIntegration-seed#[6E8117FDF9225FF5]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
[junit4] 2> 1787748 INFO (SUITE-TestTriggerIntegration-seed#[6E8117FDF9225FF5]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
[junit4] 2> 1787748 INFO (SUITE-TestTriggerIntegration-seed#[6E8117FDF9225FF5]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
[junit4] 2> 1787934 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
[junit4] 2> 1787935 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 0
[junit4] 2> 1787935 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr]
[junit4] 2> 1787935 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 0, lastZnodeVersion -1
[junit4] 2> 1787935 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 0
[junit4] 2> 1787965 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 1787965 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 0, lastZnodeVersion 0
[junit4] 2> 1787966 DEBUG (ScheduledTrigger-2336-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1787989 DEBUG (ScheduledTrigger-2336-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1788009 INFO (TEST-TestTriggerIntegration.testListeners-seed#[6E8117FDF9225FF5]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testListeners
[junit4] 2> 1788009 DEBUG (TEST-TestTriggerIntegration.testListeners-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
[junit4] 2> 1788022 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
[junit4] 2> 1788043 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
[junit4] 2> 1788043 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 1788043 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 1
[junit4] 2> 1788044 INFO (TEST-TestTriggerIntegration.testListeners-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
[junit4] 2> 1788045 WARN (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Interrupted
[junit4] 2> java.lang.InterruptedException
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048)
[junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:175)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 1788063 DEBUG (TEST-TestTriggerIntegration.testListeners-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
[junit4] 2> 1788079 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
[junit4] 2> 1788080 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 2
[junit4] 2> 1788080 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr]
[junit4] 2> 1788080 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion -1
[junit4] 2> 1788080 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
[junit4] 2> 1788081 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 1788081 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2
[junit4] 2> 1788081 DEBUG (ScheduledTrigger-2339-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1788101 DEBUG (ScheduledTrigger-2339-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1788138 DEBUG (ScheduledTrigger-2339-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1788172 DEBUG (ScheduledTrigger-2339-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1788209 DEBUG (ScheduledTrigger-2339-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1788262 DEBUG (ScheduledTrigger-2339-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1788283 DEBUG (ScheduledTrigger-2339-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1788318 DEBUG (ScheduledTrigger-2339-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1788355 DEBUG (ScheduledTrigger-2339-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1788400 DEBUG (ScheduledTrigger-2339-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1788417 DEBUG (simCloudManagerPool-2338-thread-1) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 1788417 DEBUG (simCloudManagerPool-2338-thread-1) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 3
[junit4] 2> 1788421 DEBUG (ScheduledTrigger-2339-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1788447 DEBUG (simCloudManagerPool-2338-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr]
[junit4] 2> 1788456 DEBUG (ScheduledTrigger-2339-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1788477 DEBUG (ScheduledTrigger-2339-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1788530 DEBUG (ScheduledTrigger-2339-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1788540 DEBUG (simCloudManagerPool-2338-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr]
[junit4] 2> 1788540 DEBUG (simCloudManagerPool-2338-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated with properties: {event=nodeAdded, waitFor=2, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerAction}, {name=test1, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestDummyAction}], enabled=true}
[junit4] 2> 1788541 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
[junit4] 2> 1788541 INFO (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
[junit4] 2> 1788542 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
[junit4] 2> 1788546 DEBUG (ScheduledTrigger-2339-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1788602 DEBUG (ScheduledTrigger-2339-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
[junit4] 2> 1788550 DEBUG (ScheduledTrigger-2339-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1788631 DEBUG (ScheduledTrigger-2339-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1788631 DEBUG (ScheduledTrigger-2339-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
[junit4] 2> 1788631 DEBUG (ScheduledTrigger-2339-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1788633 DEBUG (simCloudManagerPool-2338-thread-2) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 1788635 DEBUG (simCloudManagerPool-2338-thread-2) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 4
[junit4] 2> 1788635 DEBUG (simCloudManagerPool-2338-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr]
[junit4] 2> 1788635 DEBUG (simCloudManagerPool-2338-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr]
[junit4] 2> 1788635 DEBUG (simCloudManagerPool-2338-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated with properties: {event=nodeAdded, waitFor=2, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerAction}, {name=test1, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestDummyAction}], enabled=true}
[junit4] 2> 1788635 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 4
[junit4] 2> 1788652 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion 4
[junit4] 2> 1788668 DEBUG (simCloudManagerPool-2338-thread-3) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 1788669 DEBUG (ScheduledTrigger-2339-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1788669 DEBUG (simCloudManagerPool-2338-thread-3) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 5
[junit4] 2> 1788669 DEBUG (simCloudManagerPool-2338-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr]
[junit4] 2> 1788669 DEBUG (simCloudManagerPool-2338-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr]
[junit4] 2> 1788669 DEBUG (simCloudManagerPool-2338-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated with properties: {event=nodeAdded, waitFor=2, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerAction}, {name=test1, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestDummyAction}], enabled=true}
[junit4] 2> 1788671 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 5
[junit4] 2> 1788672 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 5, lastZnodeVersion 5
[junit4] 2> 1788681 DEBUG (ScheduledTrigger-2339-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1788681 DEBUG (ScheduledTrigger-2339-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1788681 DEBUG (ScheduledTrigger-2339-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10002_solr at time 1954931348522773
[junit4] 2> 1788689 DEBUG (ScheduledTrigger-2339-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 1788702 DEBUG (ScheduledTrigger-2339-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1788702 DEBUG (ScheduledTrigger-2339-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1788709 DEBUG (ScheduledTrigger-2339-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 1788722 DEBUG (ScheduledTrigger-2339-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1788722 DEBUG (ScheduledTrigger-2339-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1788722 DEBUG (ScheduledTrigger-2339-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger firing registered processor for nodes: [127.0.0.1:10002_solr] added at times [1954931348522773], now=1954933417648323
[junit4] 2> 1788723 DEBUG (ScheduledTrigger-2339-thread-3) [ ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
[junit4] 2> "id":"6f1ffec96c715T1zkivunzsud6vok1fzwwen8z2",
[junit4] 2> "source":"node_added_trigger",
[junit4] 2> "eventTime":1954931348522773,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[1954931348522773],
[junit4] 2> "nodeNames":["127.0.0.1:10002_solr"]}}
[junit4] 2> 1788724 DEBUG (ScheduledTrigger-2339-thread-3) [ ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas, node_added_trigger]
[junit4] 2> 1788775 DEBUG (AutoscalingActionExecutor-2340-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
[junit4] 2> "id":"6f1ffec96c715T1zkivunzsud6vok1fzwwen8z2",
[junit4] 2> "source":"node_added_trigger",
[junit4] 2> "eventTime":1954931348522773,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[1954931348522773],
[junit4] 2> "_enqueue_time_":1954934180364923,
[junit4] 2> "nodeNames":["127.0.0.1:10002_solr"]}}
[junit4] 2> 1788812 DEBUG (AutoscalingActionExecutor-2340-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Resuming trigger: .auto_add_replicas after 100ms
[junit4] 2> 1788812 DEBUG (AutoscalingActionExecutor-2340-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_trigger after 100ms
[junit4] 2> 1788812 DEBUG (AutoscalingActionExecutor-2340-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing took 1879 ms for event id=6f1ffec96c715T1zkivunzsud6vok1fzwwen8z2
[junit4] 2> 1788924 DEBUG (ScheduledTrigger-2339-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 4
[junit4] 2> 1788924 DEBUG (ScheduledTrigger-2339-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1788924 DEBUG (ScheduledTrigger-2339-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
[junit4] 2> 1788924 DEBUG (ScheduledTrigger-2339-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10003_solr at time 1954943525538773
[junit4] 2> 1788945 DEBUG (ScheduledTrigger-2339-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 4
[junit4] 2> 1788945 DEBUG (ScheduledTrigger-2339-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1788945 DEBUG (ScheduledTrigger-2339-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
[junit4] 2> 1788965 DEBUG (ScheduledTrigger-2339-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 4
[junit4] 2> 1788965 DEBUG (ScheduledTrigger-2339-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1788965 DEBUG (ScheduledTrigger-2339-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
[junit4] 2> 1788965 DEBUG (ScheduledTrigger-2339-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger firing registered processor for nodes: [127.0.0.1:10003_solr] added at times [1954943525538773], now=1954945573514573
[junit4] 2> 1788965 DEBUG (ScheduledTrigger-2339-thread-4) [ ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
[junit4] 2> "id":"6f202c2654bd5T1zkivunzsud6vok1fzwwen8z5",
[junit4] 2> "source":"node_added_trigger",
[junit4] 2> "eventTime":1954943525538773,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[1954943525538773],
[junit4] 2> "nodeNames":["127.0.0.1:10003_solr"]}}
[junit4] 2> 1788965 DEBUG (ScheduledTrigger-2339-thread-4) [ ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas, node_added_trigger]
[junit4] 2> 1788997 DEBUG (AutoscalingActionExecutor-2340-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
[junit4] 2> "id":"6f202c2654bd5T1zkivunzsud6vok1fzwwen8z5",
[junit4] 2> "source":"node_added_trigger",
[junit4] 2> "eventTime":1954943525538773,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[1954943525538773],
[junit4] 2> "_enqueue_time_":1954945582757723,
[junit4] 2> "nodeNames":["127.0.0.1:10003_solr"]}}
[junit4] 2> 1789034 WARN (AutoscalingActionExecutor-2340-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Exception executing actions
[junit4] 2> java.lang.Exception: Error executing action: test1 for trigger event: {
[junit4] 2> "id":"6f202c2654bd5T1zkivunzsud6vok1fzwwen8z5",
[junit4] 2> "source":"node_added_trigger",
[junit4] 2> "eventTime":1954943525538773,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[1954943525538773],
[junit4] 2> "_enqueue_time_":1954945582757723,
[junit4] 2> "nodeNames":["127.0.0.1:10003_solr"]}}
[junit4] 2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:301)
[junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[junit4] 2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> Caused by: java.lang.RuntimeException: failure
[junit4] 2> at org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestDummyAction.process(TestTriggerIntegration.java:892)
[junit4] 2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:298)
[junit4] 2> ... 6 more
[junit4] 2> 1789034 DEBUG (AutoscalingActionExecutor-2340-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Resuming trigger: .auto_add_replicas after 100ms
[junit4] 2> 1789034 DEBUG (AutoscalingActionExecutor-2340-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_trigger after 100ms
[junit4] 2> 1789034 DEBUG (AutoscalingActionExecutor-2340-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing took 1856 ms for event id=6f202c2654bd5T1zkivunzsud6vok1fzwwen8z5
[junit4] 2> 1789054 INFO (TEST-TestTriggerIntegration.testListeners-seed#[6E8117FDF9225FF5]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testListeners
[junit4] 2> 1789054 INFO (TEST-TestTriggerIntegration.testListeners-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase
[junit4] 2>
[junit4] 2> 1789054 INFO (TEST-TestTriggerIntegration.testListeners-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase #############################################
[junit4] 2> 1789054 INFO (TEST-TestTriggerIntegration.testListeners-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ############ FINAL CLUSTER STATS ############
[junit4] 2> 1789054 INFO (TEST-TestTriggerIntegration.testListeners-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase #############################################
[junit4] 2>
[junit4] 2> 1789055 INFO (TEST-TestTriggerIntegration.testListeners-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Live nodes: 4
[junit4] 2> 1789055 INFO (TEST-TestTriggerIntegration.testListeners-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Empty nodes: 4
[junit4] 2> 1789055 INFO (TEST-TestTriggerIntegration.testListeners-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Dead nodes: 0
[junit4] 2> 1789056 INFO (TEST-TestTriggerIntegration.testListeners-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Collections: []
[junit4] 2> 1789056 INFO (TEST-TestTriggerIntegration.testListeners-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Max replicas per node: 0
[junit4] 2> 1789056 INFO (TEST-TestTriggerIntegration.testListeners-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Min replicas per node: 0
[junit4] 2> 1789056 INFO (TEST-TestTriggerIntegration.testListeners-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Total replicas: 0
[junit4] 2> 1789057 INFO (TEST-TestTriggerIntegration.testListeners-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ######### Final Solr op counts ##########
[junit4] 2> 1789057 INFO (TEST-TestTriggerIntegration.testListeners-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - autoscaling 3
[junit4] 2> 1789058 INFO (TEST-TestTriggerIntegration.testListeners-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - update 4
[junit4] 2> 1789058 INFO (TEST-TestTriggerIntegration.testListeners-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ######### Autoscaling event counts ###########
[junit4] 2> 1789092 INFO (TEST-TestTriggerIntegration.testListeners-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## * Trigger: node_added_trigger
[junit4] 2> 1789093 INFO (TEST-TestTriggerIntegration.testListeners-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - FAILED 1
[junit4] 2> 1789093 INFO (TEST-TestTriggerIntegration.testListeners-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - STARTED 2
[junit4] 2> 1789093 INFO (TEST-TestTriggerIntegration.testListeners-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - SUCCEEDED 1
[junit4] 2> 1789143 DEBUG (ScheduledTrigger-2339-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 4
[junit4] 2> 1789144 DEBUG (ScheduledTrigger-2339-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1789144 DEBUG (ScheduledTrigger-2339-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
[junit4] 2> 1789239 DEBUG (ScheduledTrigger-2339-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 4
[junit4] 2> 1789239 DEBUG (ScheduledTrigger-2339-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1789239 DEBUG (ScheduledTrigger-2339-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
[junit4] 2> 1789240 INFO (TEST-TestTriggerIntegration.testCooldown-seed#[6E8117FDF9225FF5]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testCooldown
[junit4] 2> 1789240 DEBUG (TEST-TestTriggerIntegration.testCooldown-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 6
[junit4] 2> 1789253 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 6
[junit4] 2> 1789253 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
[junit4] 2> 1789253 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 1789253 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 6, lastZnodeVersion 6
[junit4] 2> 1789254 INFO (TEST-TestTriggerIntegration.testCooldown-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
[junit4] 2> 1789254 WARN (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Interrupted
[junit4] 2> java.lang.InterruptedException
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048)
[junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:175)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 1789255 DEBUG (TEST-TestTriggerIntegration.testCooldown-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
[junit4] 2> 1789273 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
[junit4] 2> 1789273 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 7
[junit4] 2> 1789273 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10000_solr, 127.0.0.1:10002_solr]
[junit4] 2> 1789273 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 7, lastZnodeVersion -1
[junit4] 2> 1789273 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 7
[junit4] 2> 1789291 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 1789292 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 7, lastZnodeVersion 7
[junit4] 2> 1789307 DEBUG (ScheduledTrigger-2342-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1789344 DEBUG (ScheduledTrigger-2342-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1789365 DEBUG (ScheduledTrigger-2342-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1789417 DEBUG (simCloudManagerPool-2341-thread-1) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 1789417 DEBUG (simCloudManagerPool-2341-thread-1) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 8
[junit4] 2> 1789417 DEBUG (simCloudManagerPool-2341-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10000_solr, 127.0.0.1:10002_solr]
[junit4] 2> 1789417 DEBUG (simCloudManagerPool-2341-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_cooldown_trigger instantiated with properties: {event=nodeAdded, waitFor=1, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerAction}], enabled=true}
[junit4] 2> 1789417 DEBUG (simCloudManagerPool-2341-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10000_solr, 127.0.0.1:10002_solr]
[junit4] 2> 1789419 DEBUG (ScheduledTrigger-2342-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1789419 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 8
[junit4] 2> 1789419 INFO (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
[junit4] 2> 1789419 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 8, lastZnodeVersion 8
[junit4] 2> 1789428 DEBUG (ScheduledTrigger-2342-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_cooldown_trigger
[junit4] 2> 1789428 DEBUG (ScheduledTrigger-2342-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
[junit4] 2> 1789429 DEBUG (simCloudManagerPool-2341-thread-2) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 1789429 DEBUG (simCloudManagerPool-2341-thread-2) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 9
[junit4] 2> 1789429 DEBUG (simCloudManagerPool-2341-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10000_solr, 127.0.0.1:10002_solr]
[junit4] 2> 1789429 DEBUG (simCloudManagerPool-2341-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_cooldown_trigger instantiated with properties: {event=nodeAdded, waitFor=1, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerAction}], enabled=true}
[junit4] 2> 1789429 DEBUG (simCloudManagerPool-2341-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10000_solr, 127.0.0.1:10002_solr]
[junit4] 2> 1789429 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 9
[junit4] 2> 1789429 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 9, lastZnodeVersion 9
[junit4] 2> 1789444 DEBUG (ScheduledTrigger-2342-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 1789465 DEBUG (ScheduledTrigger-2342-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_cooldown_trigger
[junit4] 2> 1789465 DEBUG (ScheduledTrigger-2342-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1789465 DEBUG (ScheduledTrigger-2342-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 1789489 DEBUG (ScheduledTrigger-2342-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10004_solr at time 1954971767178123
[junit4] 2> 1789489 DEBUG (ScheduledTrigger-2342-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 1789518 DEBUG (ScheduledTrigger-2342-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 1789518 DEBUG (ScheduledTrigger-2342-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_cooldown_trigger
[junit4] 2> 1789518 DEBUG (ScheduledTrigger-2342-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1789518 DEBUG (ScheduledTrigger-2342-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_cooldown_trigger firing registered processor for nodes: [127.0.0.1:10004_solr] added at times [1954971767178123], now=1954973222251873
[junit4] 2> 1789518 DEBUG (ScheduledTrigger-2342-thread-2) [ ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
[junit4] 2> "id":"6f20955ba838bT1zkivunzsud6vok1fzwwen8z8",
[junit4] 2> "source":"node_added_cooldown_trigger",
[junit4] 2> "eventTime":1954971767178123,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[1954971767178123],
[junit4] 2> "nodeNames":["127.0.0.1:10004_solr"]}}
[junit4] 2> 1789527 DEBUG (ScheduledTrigger-2342-thread-2) [ ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas, node_added_cooldown_trigger]
[junit4] 2> 1789528 DEBUG (AutoscalingActionExecutor-2343-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
[junit4] 2> "id":"6f20955ba838bT1zkivunzsud6vok1fzwwen8z8",
[junit4] 2> "source":"node_added_cooldown_trigger",
[junit4] 2> "eventTime":1954971767178123,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[1954971767178123],
[junit4] 2> "_enqueue_time_":1954973654738973,
[junit4] 2> "nodeNames":["127.0.0.1:10004_solr"]}}
[junit4] 2> 1789529 DEBUG (AutoscalingActionExecutor-2343-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Resuming trigger: .auto_add_replicas after 100ms
[junit4] 2> 1789529 DEBUG (AutoscalingActionExecutor-2343-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_cooldown_trigger after 100ms
[junit4] 2> 1789529 DEBUG (AutoscalingActionExecutor-2343-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing took 47 ms for event id=6f20955ba838bT1zkivunzsud6vok1fzwwen8z8
[junit4] 2> 1789640 DEBUG (ScheduledTrigger-2342-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 4
[junit4] 2> 1789640 DEBUG (ScheduledTrigger-2342-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_cooldown_trigger
[junit4] 2> 1789640 DEBUG (ScheduledTrigger-2342-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
[junit4] 2> 1789640 DEBUG (ScheduledTrigger-2342-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10005_solr at time 1954979318057123
[junit4] 2> 1789660 DEBUG (ScheduledTrigger-2342-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 4
[junit4] 2> 1789660 DEBUG (ScheduledTrigger-2342-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_cooldown_trigger
[junit4] 2> 1789660 DEBUG (ScheduledTrigger-2342-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
[junit4] 2> 1789660 DEBUG (ScheduledTrigger-2342-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_cooldown_trigger firing registered processor for nodes: [127.0.0.1:10005_solr] added at times [1954979318057123], now=1954980330838123
[junit4] 2> 1789660 DEBUG (ScheduledTrigger-2342-thread-1) [ ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
[junit4] 2> "id":"6f20b17cbc8a3T1zkivunzsud6vok1fzwwen8zb",
[junit4] 2> "source":"node_added_cooldown_trigger",
[junit4] 2> "eventTime":1954979318057123,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[1954979318057123],
[junit4] 2> "nodeNames":["127.0.0.1:10005_solr"]}}
[junit4] 2> 1789660 DEBUG (ScheduledTrigger-2342-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas, node_added_cooldown_trigger]
[junit4] 2> 1789676 DEBUG (AutoscalingActionExecutor-2343-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
[junit4] 2> "id":"6f20b17cbc8a3T1zkivunzsud6vok1fzwwen8zb",
[junit4] 2> "source":"node_added_cooldown_trigger",
[junit4] 2> "eventTime":1954979318057123,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[1954979318057123],
[junit4] 2> "_enqueue_time_":1954980340676523,
[junit4] 2> "nodeNames":["127.0.0.1:10005_solr"]}}
[junit4] 2> 1789677 DEBUG (AutoscalingActionExecutor-2343-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Resuming trigger: .auto_add_replicas after 100ms
[junit4] 2> 1789677 DEBUG (AutoscalingActionExecutor-2343-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_cooldown_trigger after 100ms
[junit4] 2> 1789677 DEBUG (AutoscalingActionExecutor-2343-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing took 29 ms for event id=6f20b17cbc8a3T1zkivunzsud6vok1fzwwen8zb
[junit4] 2> 1789737 INFO (TEST-TestTriggerIntegration.testCooldown-seed#[6E8117FDF9225FF5]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testCooldown
[junit4] 2> 1789737 INFO (TEST-TestTriggerIntegration.testCooldown-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase
[junit4] 2>
[junit4] 2> 1789737 INFO (TEST-TestTriggerIntegration.testCooldown-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase #############################################
[junit4] 2> 1789737 INFO (TEST-TestTriggerIntegration.testCooldown-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ############ FINAL CLUSTER STATS ############
[junit4] 2> 1789737 INFO (TEST-TestTriggerIntegration.testCooldown-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase #############################################
[junit4] 2>
[junit4] 2> 1789737 INFO (TEST-TestTriggerIntegration.testCooldown-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Live nodes: 4
[junit4] 2> 1789737 INFO (TEST-TestTriggerIntegration.testCooldown-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Empty nodes: 4
[junit4] 2> 1789749 INFO (TEST-TestTriggerIntegration.testCooldown-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Dead nodes: 0
[junit4] 2> 1789749 INFO (TEST-TestTriggerIntegration.testCooldown-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Collections: []
[junit4] 2> 1789749 INFO (TEST-TestTriggerIntegration.testCooldown-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Max replicas per node: 0
[junit4] 2> 1789749 INFO (TEST-TestTriggerIntegration.testCooldown-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Min replicas per node: 0
[junit4] 2> 1789749 INFO (TEST-TestTriggerIntegration.testCooldown-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Total replicas: 0
[junit4] 2> 1789749 INFO (TEST-TestTriggerIntegration.testCooldown-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ######### Final Solr op counts ##########
[junit4] 2> 1789749 INFO (TEST-TestTriggerIntegration.testCooldown-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - autoscaling 2
[junit4] 2> 1789749 INFO (TEST-TestTriggerIntegration.testCooldown-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - update 4
[junit4] 2> 1789749 INFO (TEST-TestTriggerIntegration.testCooldown-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ######### Autoscaling event counts ###########
[junit4] 2> 1789749 INFO (TEST-TestTriggerIntegration.testCooldown-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## * Trigger: node_added_cooldown_trigger
[junit4] 2> 1789749 INFO (TEST-TestTriggerIntegration.testCooldown-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - STARTED 2
[junit4] 2> 1789749 INFO (TEST-TestTriggerIntegration.testCooldown-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - SUCCEEDED 2
[junit4] 2> 1789753 INFO (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[6E8117FDF9225FF5]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testNodeAddedTrigger
[junit4] 2> 1789753 DEBUG (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 10
[junit4] 2> 1789770 INFO (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
[junit4] 2> 1789770 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 10
[junit4] 2> 1789770 DEBUG (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
[junit4] 2> 1789770 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
[junit4] 2> 1789770 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 1789770 WARN (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Interrupted
[junit4] 2> java.lang.InterruptedException
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220)
[junit4] 2> at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335)
[junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:163)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 1789786 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
[junit4] 2> 1789786 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 11
[junit4] 2> 1789786 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10004_solr, 127.0.0.1:10002_solr]
[junit4] 2> 1789786 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 11, lastZnodeVersion -1
[junit4] 2> 1789786 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 11
[junit4] 2> 1789787 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 1789787 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 11, lastZnodeVersion 11
[junit4] 2> 1789788 DEBUG (ScheduledTrigger-2345-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1789816 DEBUG (ScheduledTrigger-2345-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1789852 DEBUG (ScheduledTrigger-2345-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1789881 DEBUG (simCloudManagerPool-2344-thread-1) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 1789881 DEBUG (simCloudManagerPool-2344-thread-1) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 12
[junit4] 2> 1789881 DEBUG (simCloudManagerPool-2344-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10004_solr, 127.0.0.1:10002_solr]
[junit4] 2> 1789881 DEBUG (simCloudManagerPool-2344-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10004_solr, 127.0.0.1:10002_solr]
[junit4] 2> 1789881 DEBUG (simCloudManagerPool-2344-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated with properties: {event=nodeAdded, waitFor=2, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerAction}], enabled=true}
[junit4] 2> 1789887 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 12
[junit4] 2> 1789888 DEBUG (ScheduledTrigger-2345-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1789888 INFO (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
[junit4] 2> 1789891 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 12, lastZnodeVersion 12
[junit4] 2> 1789891 DEBUG (ScheduledTrigger-2345-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1789904 DEBUG (ScheduledTrigger-2345-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1789904 DEBUG (ScheduledTrigger-2345-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10006_solr at time 1954992515625473
[junit4] 2> 1789916 DEBUG (ScheduledTrigger-2345-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 1789925 DEBUG (ScheduledTrigger-2345-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1789925 DEBUG (ScheduledTrigger-2345-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1789940 DEBUG (ScheduledTrigger-2345-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 1789961 DEBUG (ScheduledTrigger-2345-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 1789961 DEBUG (ScheduledTrigger-2345-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1789961 DEBUG (ScheduledTrigger-2345-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1789961 DEBUG (ScheduledTrigger-2345-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger firing registered processor for nodes: [127.0.0.1:10006_solr] added at times [1954992515625473], now=1954995368447273
[junit4] 2> 1789961 DEBUG (ScheduledTrigger-2345-thread-2) [ ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
[junit4] 2> "id":"6f20e2a6eb201T1zkivunzsud6vok1fzwwen8ze",
[junit4] 2> "source":"node_added_trigger",
[junit4] 2> "eventTime":1954992515625473,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[1954992515625473],
[junit4] 2> "nodeNames":["127.0.0.1:10006_solr"]}}
[junit4] 2> 1789961 DEBUG (ScheduledTrigger-2345-thread-2) [ ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas, node_added_trigger]
[junit4] 2> 1790062 DEBUG (AutoscalingActionExecutor-2346-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
[junit4] 2> "id":"6f20e2a6eb201T1zkivunzsud6vok1fzwwen8ze",
[junit4] 2> "source":"node_added_trigger",
[junit4] 2> "eventTime":1954992515625473,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[1954992515625473],
[junit4] 2> "_enqueue_time_":1954995389102973,
[junit4] 2> "nodeNames":["127.0.0.1:10006_solr"]}}
[junit4] 2> 1790108 DEBUG (simCloudManagerPool-2344-thread-4) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 1790108 DEBUG (simCloudManagerPool-2344-thread-4) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 13
[junit4] 2> 1790108 DEBUG (simCloudManagerPool-2344-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10004_solr, 127.0.0.1:10006_solr, 127.0.0.1:10002_solr]
[junit4] 2> 1790108 DEBUG (simCloudManagerPool-2344-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10004_solr, 127.0.0.1:10006_solr, 127.0.0.1:10002_solr]
[junit4] 2> 1790108 DEBUG (simCloudManagerPool-2344-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated with properties: {event=nodeAdded, waitFor=2, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerAction}], enabled=true}
[junit4] 2> 1790108 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 13
[junit4] 2> 1790115 DEBUG (AutoscalingActionExecutor-2346-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Resuming trigger: .auto_add_replicas after 100ms
[junit4] 2> 1790115 DEBUG (AutoscalingActionExecutor-2346-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_trigger after 100ms
[junit4] 2> 1790115 DEBUG (AutoscalingActionExecutor-2346-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing took 2642 ms for event id=6f20e2a6eb201T1zkivunzsud6vok1fzwwen8ze
[junit4] 2> 1790116 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 13, lastZnodeVersion 13
[junit4] 2> 1790160 INFO (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[6E8117FDF9225FF5]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testNodeAddedTrigger
[junit4] 2> 1790160 INFO (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase
[junit4] 2>
[junit4] 2> 1790160 INFO (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase #############################################
[junit4] 2> 1790160 INFO (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ############ FINAL CLUSTER STATS ############
[junit4] 2> 1790160 INFO (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase #############################################
[junit4] 2>
[junit4] 2> 1790160 INFO (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Live nodes: 3
[junit4] 2> 1790160 INFO (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Empty nodes: 3
[junit4] 2> 1790160 INFO (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Dead nodes: 0
[junit4] 2> 1790160 INFO (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Collections: []
[junit4] 2> 1790160 INFO (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Max replicas per node: 0
[junit4] 2> 1790160 INFO (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Min replicas per node: 0
[junit4] 2> 1790160 INFO (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Total replicas: 0
[junit4] 2> 1790160 INFO (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ######### Final Solr op counts ##########
[junit4] 2> 1790160 INFO (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - autoscaling 2
[junit4] 2> 1790160 INFO (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - update 2
[junit4] 2> 1790160 INFO (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ######### Autoscaling event counts ###########
[junit4] 2> 1790160 INFO (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## * Trigger: node_added_trigger
[junit4] 2> 1790160 INFO (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - STARTED 1
[junit4] 2> 1790160 INFO (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - SUCCEEDED 1
[junit4] 2> 1790182 INFO (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[6E8117FDF9225FF5]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testNodeAddedTriggerRestoreState
[junit4] 2> 1790182 DEBUG (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 14
[junit4] 2> 1790182 INFO (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
[junit4] 2> 1790183 DEBUG (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
[junit4] 2> 1790183 WARN (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
[junit4] 2> 1790209 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
[junit4] 2> 1790210 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 15
[junit4] 2> 1790210 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10004_solr, 127.0.0.1:10006_solr]
[junit4] 2> 1790210 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 15, lastZnodeVersion -1
[junit4] 2> 1790210 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 15
[junit4] 2> 1790233 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 1790233 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 15, lastZnodeVersion 15
[junit4] 2> 1790233 DEBUG (ScheduledTrigger-2348-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1790265 DEBUG (ScheduledTrigger-2348-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1790308 DEBUG (ScheduledTrigger-2348-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1790442 DEBUG (ScheduledTrigger-2348-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1790443 DEBUG (simCloudManagerPool-2347-thread-1) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 1790443 DEBUG (simCloudManagerPool-2347-thread-1) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 16
[junit4] 2> 1790443 DEBUG (simCloudManagerPool-2347-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10004_solr, 127.0.0.1:10006_solr]
[junit4] 2> 1790443 DEBUG (simCloudManagerPool-2347-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_restore_trigger instantiated with properties: {event=nodeAdded, waitFor=5, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerAction}], enabled=true}
[junit4] 2> 1790443 DEBUG (simCloudManagerPool-2347-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10004_solr, 127.0.0.1:10006_solr]
[junit4] 2> 1790444 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 16
[junit4] 2> 1790444 INFO (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
[junit4] 2> 1790444 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 16, lastZnodeVersion 16
[junit4] 2> 1790444 DEBUG (ScheduledTrigger-2348-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_restore_trigger
[junit4] 2> 1790444 DEBUG (ScheduledTrigger-2348-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1790444 DEBUG (ScheduledTrigger-2348-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10007_solr at time 1955019517980023
[junit4] 2> 1790462 DEBUG (ScheduledTrigger-2348-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 1790502 DEBUG (simCloudManagerPool-2347-thread-2) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 1790502 DEBUG (simCloudManagerPool-2347-thread-2) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 17
[junit4] 2> 1790502 DEBUG (ScheduledTrigger-2348-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_restore_trigger
[junit4] 2> 1790502 DEBUG (ScheduledTrigger-2348-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1790517 DEBUG (ScheduledTrigger-2348-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 1790538 DEBUG (ScheduledTrigger-2348-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_restore_trigger
[junit4] 2> 1790538 DEBUG (simCloudManagerPool-2347-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10004_solr, 127.0.0.1:10006_solr, 127.0.0.1:10007_solr]
[junit4] 2> 1790538 DEBUG (ScheduledTrigger-2348-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1790538 DEBUG (simCloudManagerPool-2347-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_restore_trigger instantiated with properties: {event=nodeAdded, waitFor=0, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerAction}], enabled=true}
[junit4] 2> 1790538 DEBUG (ScheduledTrigger-2348-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 1790538 DEBUG (simCloudManagerPool-2347-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10004_solr, 127.0.0.1:10006_solr, 127.0.0.1:10007_solr]
[junit4] 2> 1790538 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 17
[junit4] 2> 1790539 INFO (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
[junit4] 2> 1790539 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 17, lastZnodeVersion 17
[junit4] 2> 1790539 DEBUG (ScheduledTrigger-2348-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_restore_trigger
[junit4] 2> 1790539 DEBUG (ScheduledTrigger-2348-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1790539 DEBUG (ScheduledTrigger-2348-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_restore_trigger firing registered processor for nodes: [127.0.0.1:10007_solr] added at times [1955019517980023], now=1955024258597923
[junit4] 2> 1790539 DEBUG (ScheduledTrigger-2348-thread-3) [ ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
[junit4] 2> "id":"6f21473e5ed77T1zkivunzsud6vok1fzwwen8zh",
[junit4] 2> "source":"node_added_restore_trigger",
[junit4] 2> "eventTime":1955019517980023,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[1955019517980023],
[junit4] 2> "nodeNames":["127.0.0.1:10007_solr"]}}
[junit4] 2> 1790539 DEBUG (ScheduledTrigger-2348-thread-3) [ ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas, node_added_restore_trigger]
[junit4] 2> 1790552 DEBUG (AutoscalingActionExecutor-2349-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
[junit4] 2> "id":"6f21473e5ed77T1zkivunzsud6vok1fzwwen8zh",
[junit4] 2> "source":"node_added_restore_trigger",
[junit4] 2> "eventTime":1955019517980023,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[1955019517980023],
[junit4] 2> "_enqueue_time_":1955024277630423,
[junit4] 2> "nodeNames":["127.0.0.1:10007_solr"]}}
[junit4] 2> 1790552 DEBUG (AutoscalingActionExecutor-2349-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Resuming trigger: .auto_add_replicas after 100ms
[junit4] 2> 1790552 DEBUG (AutoscalingActionExecutor-2349-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_restore_trigger after 100ms
[junit4] 2> 1790552 DEBUG (AutoscalingActionExecutor-2349-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing took 24 ms for event id=6f21473e5ed77T1zkivunzsud6vok1fzwwen8zh
[junit4] 2> 1790553 INFO (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[6E8117FDF9225FF5]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testNodeAddedTriggerRestoreState
[junit4] 2> 1790553 INFO (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase
[junit4] 2>
[junit4] 2> 1790553 INFO (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase #############################################
[junit4] 2> 1790553 INFO (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ############ FINAL CLUSTER STATS ############
[junit4] 2> 1790553 INFO (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase #############################################
[junit4] 2>
[junit4] 2> 1790553 INFO (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Live nodes: 3
[junit4] 2> 1790553 INFO (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Empty nodes: 3
[junit4] 2> 1790553 INFO (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Dead nodes: 0
[junit4] 2> 1790553 INFO (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Collections: []
[junit4] 2> 1790553 INFO (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Max replicas per node: 0
[junit4] 2> 1790553 INFO (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Min replicas per node: 0
[junit4] 2> 1790553 INFO (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Total replicas: 0
[junit4] 2> 1790553 INFO (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ######### Final Solr op counts ##########
[junit4] 2> 1790553 INFO (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - autoscaling 2
[junit4] 2> 1790553 INFO (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - update 2
[junit4] 2> 1790553 INFO (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ######### Autoscaling event counts ###########
[junit4] 2> 1790553 INFO (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## * Trigger: node_added_restore_trigger
[junit4] 2> 1790553 INFO (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - STARTED 1
[junit4] 2> 1790553 INFO (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - SUCCEEDED 1
[junit4] 2> 1790556 INFO (TEST-TestTriggerIntegration.testEventFromRestoredState-seed#[6E8117FDF9225FF5]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testEventFromRestoredState
[junit4] 2> 1790557 DEBUG (TEST-TestTriggerIntegration.testEventFromRestoredState-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 18
[junit4] 2> 1790557 INFO (TEST-TestTriggerIntegration.testEventFromRestoredState-seed#[6E8117FDF9225FF5]) [ ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
[junit4] 2> 1790557 WARN (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTrigger
[...truncated too long message...]
] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:42507_solr
[junit4] 2> 20140 INFO (jetty-closer-2-thread-1) [ ] o.e.j.s.session Stopped scavenging
[junit4] 2> 20166 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x10075052ac4000c, likely client has closed socket
[junit4] 2> 20167 INFO (jetty-closer-2-thread-3) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@3dc1d1f1{/solr,null,UNAVAILABLE}
[junit4] 2> 20168 INFO (jetty-closer-2-thread-3) [ ] o.e.j.s.session Stopped scavenging
[junit4] 2> 21574 WARN (zkCallback-30-thread-2-processing-n:127.0.0.1:37722_solr) [n:127.0.0.1:37722_solr ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
[junit4] 2> 21574 INFO (jetty-closer-2-thread-2) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@70eb960d{/solr,null,UNAVAILABLE}
[junit4] 2> 21574 INFO (jetty-closer-2-thread-2) [ ] o.e.j.s.session Stopped scavenging
[junit4] 2> 50150 WARN (jetty-closer-2-thread-1) [ ] o.e.j.u.t.QueuedThreadPool QueuedThreadPool@qtp297481271{STOPPING,8<=8<=10000,i=0,q=1} Couldn't stop Thread[qtp297481271-56,5,TGRP-TestV2Request]
[junit4] 2> 50160 ERROR (SUITE-TestV2Request-seed#[9384056312F5B55]-worker) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 50180 INFO (SUITE-TestV2Request-seed#[9384056312F5B55]-worker) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:46636 46636
[junit4] 2> 50402 INFO (Thread-1) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:46636 46636
[junit4] 2> 50403 WARN (Thread-1) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 14 /solr/collections/test/terms/shard1
[junit4] 2> 12 /solr/configs/config
[junit4] 2> 12 /solr/collections/test/terms/shard2
[junit4] 2> 5 /solr/aliases.json
[junit4] 2> 4 /solr/security.json
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 68 /solr/collections/test/state.json
[junit4] 2> 5 /solr/clusterstate.json
[junit4] 2> 5 /solr/clusterprops.json
[junit4] 2> 2 /solr/overseer_elect/election/72186259090702345-127.0.0.1:45415_solr-n_0000000000
[junit4] 2> 2 /solr/collections/test/leader_elect/shard2/election/72186259090702348-core_node7-n_0000000000
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 5 /solr/live_nodes
[junit4] 2> 5 /solr/collections
[junit4] 2>
[junit4] 2> 50404 INFO (SUITE-TestV2Request-seed#[9384056312F5B55]-worker) [ ] o.a.s.SolrTestCaseJ4 ------------------------------------------------------- Done waiting for tracked resources to be released
[junit4] 2> Feb 26, 2018 6:18:41 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4] 2> WARNING: Will linger awaiting termination of 2 leaked thread(s).
[junit4] 2> Feb 26, 2018 6:19:01 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4] 2> SEVERE: 1 thread leaked from SUITE scope at org.apache.solr.client.solrj.request.TestV2Request:
[junit4] 2> 1) Thread[id=56, name=qtp297481271-56, state=TIMED_WAITING, group=TGRP-TestV2Request]
[junit4] 2> at sun.misc.Unsafe.park(Native Method)
[junit4] 2> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
[junit4] 2> at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
[junit4] 2> at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> Feb 26, 2018 6:19:01 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
[junit4] 2> INFO: Starting to interrupt leaked threads:
[junit4] 2> 1) Thread[id=56, name=qtp297481271-56, state=TIMED_WAITING, group=TGRP-TestV2Request]
[junit4] 2> Feb 26, 2018 6:19:04 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
[junit4] 2> SEVERE: There are still zombie threads that couldn't be terminated:
[junit4] 2> 1) Thread[id=56, name=qtp297481271-56, state=TIMED_WAITING, group=TGRP-TestV2Request]
[junit4] 2> at sun.misc.Unsafe.park(Native Method)
[junit4] 2> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
[junit4] 2> at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
[junit4] 2> at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene70), sim=RandomSimilarity(queryNorm=false): {}, locale=el-GR, timezone=America/Godthab
[junit4] 2> NOTE: Linux 4.4.0-112-generic amd64/Oracle Corporation 1.8.0_152 (64-bit)/cpus=4,threads=2,free=262489488,total=345505792
[junit4] 2> NOTE: All tests run in this JVM: [HyperbolicCosineEvaluatorTest, TestV2Request]
[junit4] 2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestV2Request -Dtests.seed=9384056312F5B55 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/test-data/enwiki.random.lines.txt -Dtests.locale=el-GR -Dtests.timezone=America/Godthab -Dtests.asserts=true -Dtests.file.encoding=UTF-8
[junit4] ERROR 0.00s J1 | TestV2Request (suite) <<<
[junit4] > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.client.solrj.request.TestV2Request:
[junit4] > 1) Thread[id=56, name=qtp297481271-56, state=TIMED_WAITING, group=TGRP-TestV2Request]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
[junit4] > at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
[junit4] > at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] > at __randomizedtesting.SeedInfo.seed([9384056312F5B55]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
[junit4] > 1) Thread[id=56, name=qtp297481271-56, state=TIMED_WAITING, group=TGRP-TestV2Request]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
[junit4] > at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
[junit4] > at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] > at __randomizedtesting.SeedInfo.seed([9384056312F5B55]:0)
[junit4] Completed [30/146 (1!)] on J1 in 74.13s, 3 tests, 2 errors <<< FAILURES!
[...truncated 8134 lines...]
BUILD FAILED
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/build.xml:626: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/build.xml:570: Some of the tests produced a heap dump, but did not fail. Maybe a suppressed OutOfMemoryError? Dumps created:
* java_pid27181.hprof
Total time: 525 minutes 20 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Lucene-Solr-NightlyTests-7.x - Build # 158 - Still
unstable
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.x/158/
3 tests failed.
FAILED: org.apache.solr.cloud.ReplaceNodeNoTargetTest.test
Error Message:
Stack Trace:
java.lang.AssertionError
at __randomizedtesting.SeedInfo.seed([2E3A421EDF6746B1:A66E7DC4719B2B49]:0)
at org.junit.Assert.fail(Assert.java:92)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertTrue(Assert.java:54)
at org.apache.solr.cloud.ReplaceNodeNoTargetTest.test(ReplaceNodeNoTargetTest.java:92)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: org.apache.solr.handler.admin.AutoscalingHistoryHandlerTest.testHistory
Error Message:
expected:<5> but was:<0>
Stack Trace:
java.lang.AssertionError: expected:<5> but was:<0>
at __randomizedtesting.SeedInfo.seed([2E3A421EDF6746B1:43C6E6E3652FB9B6]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at org.apache.solr.handler.admin.AutoscalingHistoryHandlerTest.testHistory(AutoscalingHistoryHandlerTest.java:311)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: org.apache.solr.core.HdfsDirectoryFactoryTest.testInitArgsOrSysPropConfig
Error Message:
The max direct memory is likely too low. Either increase it (by adding -XX:MaxDirectMemorySize=<size>g -XX:+UseLargePages to your containers startup args) or disable direct allocation using solr.hdfs.blockcache.direct.memory.allocation=false in solrconfig.xml. If you are putting the block cache on the heap, your java heap size might not be large enough. Failed allocating ~134.217728 MB.
Stack Trace:
java.lang.RuntimeException: The max direct memory is likely too low. Either increase it (by adding -XX:MaxDirectMemorySize=<size>g -XX:+UseLargePages to your containers startup args) or disable direct allocation using solr.hdfs.blockcache.direct.memory.allocation=false in solrconfig.xml. If you are putting the block cache on the heap, your java heap size might not be large enough. Failed allocating ~134.217728 MB.
at __randomizedtesting.SeedInfo.seed([2E3A421EDF6746B1:D9958B3522EEAC9A]:0)
at org.apache.solr.core.HdfsDirectoryFactory.createBlockCache(HdfsDirectoryFactory.java:312)
at org.apache.solr.core.HdfsDirectoryFactory.getBlockDirectoryCache(HdfsDirectoryFactory.java:288)
at org.apache.solr.core.HdfsDirectoryFactory.create(HdfsDirectoryFactory.java:228)
at org.apache.solr.core.HdfsDirectoryFactoryTest.testInitArgsOrSysPropConfig(HdfsDirectoryFactoryTest.java:111)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:694)
at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
at org.apache.solr.store.blockcache.BlockCache.<init>(BlockCache.java:71)
at org.apache.solr.core.HdfsDirectoryFactory.createBlockCache(HdfsDirectoryFactory.java:310)
... 42 more
Build Log:
[...truncated 12722 lines...]
[junit4] Suite: org.apache.solr.handler.admin.AutoscalingHistoryHandlerTest
[junit4] 2> 185990 INFO (SUITE-AutoscalingHistoryHandlerTest-seed#[2E3A421EDF6746B1]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
[junit4] 2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_2E3A421EDF6746B1-001/init-core-data-001
[junit4] 2> 185992 WARN (SUITE-AutoscalingHistoryHandlerTest-seed#[2E3A421EDF6746B1]-worker) [ ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=1 numCloses=1
[junit4] 2> 185993 INFO (SUITE-AutoscalingHistoryHandlerTest-seed#[2E3A421EDF6746B1]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
[junit4] 2> 186000 INFO (SUITE-AutoscalingHistoryHandlerTest-seed#[2E3A421EDF6746B1]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
[junit4] 2> 186001 INFO (SUITE-AutoscalingHistoryHandlerTest-seed#[2E3A421EDF6746B1]-worker) [ ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_2E3A421EDF6746B1-001/tempDir-001
[junit4] 2> 186001 INFO (SUITE-AutoscalingHistoryHandlerTest-seed#[2E3A421EDF6746B1]-worker) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 186011 INFO (Thread-228) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 186011 INFO (Thread-228) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 186019 ERROR (Thread-228) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 186104 INFO (SUITE-AutoscalingHistoryHandlerTest-seed#[2E3A421EDF6746B1]-worker) [ ] o.a.s.c.ZkTestServer start zk server on port:46339
[junit4] 2> 186108 INFO (zkConnectionManagerCallback-132-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 186127 INFO (jetty-launcher-129-thread-1) [ ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-22T07:57:37+10:30, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
[junit4] 2> 186128 INFO (jetty-launcher-129-thread-2) [ ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-22T07:57:37+10:30, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
[junit4] 2> 186136 INFO (jetty-launcher-129-thread-1) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 186136 INFO (jetty-launcher-129-thread-1) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 186136 INFO (jetty-launcher-129-thread-1) [ ] o.e.j.s.session Scavenging every 600000ms
[junit4] 2> 186136 INFO (jetty-launcher-129-thread-1) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4c267010{/solr,null,AVAILABLE}
[junit4] 2> 186138 INFO (jetty-launcher-129-thread-2) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 186138 INFO (jetty-launcher-129-thread-2) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 186138 INFO (jetty-launcher-129-thread-2) [ ] o.e.j.s.session Scavenging every 600000ms
[junit4] 2> 186148 INFO (jetty-launcher-129-thread-2) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@26c4b78e{/solr,null,AVAILABLE}
[junit4] 2> 186174 INFO (jetty-launcher-129-thread-2) [ ] o.e.j.s.AbstractConnector Started ServerConnector@69564b5d{SSL,[ssl, http/1.1]}{127.0.0.1:37694}
[junit4] 2> 186174 INFO (jetty-launcher-129-thread-2) [ ] o.e.j.s.Server Started @188426ms
[junit4] 2> 186174 INFO (jetty-launcher-129-thread-2) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=37694}
[junit4] 2> 186174 ERROR (jetty-launcher-129-thread-2) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 186174 INFO (jetty-launcher-129-thread-2) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 7.3.0
[junit4] 2> 186174 INFO (jetty-launcher-129-thread-2) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 186174 INFO (jetty-launcher-129-thread-2) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 186174 INFO (jetty-launcher-129-thread-2) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-02-25T01:03:51.971Z
[junit4] 2> 186180 INFO (jetty-launcher-129-thread-1) [ ] o.e.j.s.AbstractConnector Started ServerConnector@c9157d6{SSL,[ssl, http/1.1]}{127.0.0.1:54807}
[junit4] 2> 186180 INFO (jetty-launcher-129-thread-1) [ ] o.e.j.s.Server Started @188432ms
[junit4] 2> 186180 INFO (jetty-launcher-129-thread-1) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=54807}
[junit4] 2> 186180 ERROR (jetty-launcher-129-thread-1) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 186180 INFO (jetty-launcher-129-thread-1) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 7.3.0
[junit4] 2> 186180 INFO (jetty-launcher-129-thread-1) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 186180 INFO (jetty-launcher-129-thread-1) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 186180 INFO (jetty-launcher-129-thread-1) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-02-25T01:03:51.977Z
[junit4] 2> 186182 INFO (zkConnectionManagerCallback-134-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 186183 INFO (jetty-launcher-129-thread-2) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 186183 INFO (zkConnectionManagerCallback-136-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 186183 INFO (jetty-launcher-129-thread-1) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 186199 INFO (jetty-launcher-129-thread-2) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:46339/solr
[junit4] 2> 186202 INFO (jetty-launcher-129-thread-1) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:46339/solr
[junit4] 2> 186211 INFO (zkConnectionManagerCallback-140-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 186211 INFO (zkConnectionManagerCallback-144-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 186226 INFO (zkConnectionManagerCallback-148-thread-1-processing-n:127.0.0.1:54807_solr) [n:127.0.0.1:54807_solr ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 186227 INFO (zkConnectionManagerCallback-146-thread-1-processing-n:127.0.0.1:37694_solr) [n:127.0.0.1:37694_solr ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 186341 INFO (jetty-launcher-129-thread-1) [n:127.0.0.1:54807_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 186341 INFO (jetty-launcher-129-thread-1) [n:127.0.0.1:54807_solr ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:54807_solr
[junit4] 2> 186342 INFO (jetty-launcher-129-thread-1) [n:127.0.0.1:54807_solr ] o.a.s.c.Overseer Overseer (id=73353407156125701-127.0.0.1:54807_solr-n_0000000000) starting
[junit4] 2> 186354 DEBUG (OverseerStateUpdate-73353407156125701-127.0.0.1:54807_solr-n_0000000000) [n:127.0.0.1:54807_solr ] o.a.s.c.Overseer Starting to work on the main queue
[junit4] 2> 186354 INFO (jetty-launcher-129-thread-2) [n:127.0.0.1:37694_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 186355 INFO (jetty-launcher-129-thread-2) [n:127.0.0.1:37694_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37694_solr
[junit4] 2> 186359 INFO (OverseerStateUpdate-73353407156125701-127.0.0.1:54807_solr-n_0000000000) [n:127.0.0.1:54807_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 186361 INFO (jetty-launcher-129-thread-1) [n:127.0.0.1:54807_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54807_solr
[junit4] 2> 186375 INFO (zkCallback-145-thread-1-processing-n:127.0.0.1:37694_solr) [n:127.0.0.1:37694_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 186380 DEBUG (OverseerAutoScalingTriggerThread-73353407156125701-127.0.0.1:54807_solr-n_0000000000) [n:127.0.0.1:54807_solr ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
[junit4] 2> 186380 INFO (zkCallback-147-thread-1-processing-n:127.0.0.1:54807_solr) [n:127.0.0.1:54807_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 186380 DEBUG (OverseerStateUpdate-73353407156125701-127.0.0.1:54807_solr-n_0000000000) [n:127.0.0.1:54807_solr ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"downnode",
[junit4] 2> "node_name":"127.0.0.1:54807_solr"} current state version: 0
[junit4] 2> 186380 DEBUG (OverseerStateUpdate-73353407156125701-127.0.0.1:54807_solr-n_0000000000) [n:127.0.0.1:54807_solr ] o.a.s.c.o.NodeMutator DownNode state invoked for node: 127.0.0.1:54807_solr
[junit4] 2> 186386 DEBUG (OverseerAutoScalingTriggerThread-73353407156125701-127.0.0.1:54807_solr-n_0000000000) [n:127.0.0.1:54807_solr ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
[junit4] 2> 186386 DEBUG (OverseerAutoScalingTriggerThread-73353407156125701-127.0.0.1:54807_solr-n_0000000000) [n:127.0.0.1:54807_solr ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:54807_solr, 127.0.0.1:37694_solr]
[junit4] 2> 186386 DEBUG (OverseerAutoScalingTriggerThread-73353407156125701-127.0.0.1:54807_solr-n_0000000000) [n:127.0.0.1:54807_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion -1
[junit4] 2> 186386 DEBUG (OverseerAutoScalingTriggerThread-73353407156125701-127.0.0.1:54807_solr-n_0000000000) [n:127.0.0.1:54807_solr ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
[junit4] 2> 186393 DEBUG (OverseerAutoScalingTriggerThread-73353407156125701-127.0.0.1:54807_solr-n_0000000000) [n:127.0.0.1:54807_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 186393 INFO (zkCallback-145-thread-2-processing-n:127.0.0.1:37694_solr) [n:127.0.0.1:37694_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 186393 DEBUG (OverseerAutoScalingTriggerThread-73353407156125701-127.0.0.1:54807_solr-n_0000000000) [n:127.0.0.1:54807_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 1
[junit4] 2> 186404 DEBUG (ScheduledTrigger-230-thread-1) [n:127.0.0.1:54807_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 186484 INFO (jetty-launcher-129-thread-1) [n:127.0.0.1:54807_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_54807.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@126a9212
[junit4] 2> 186492 INFO (jetty-launcher-129-thread-2) [n:127.0.0.1:37694_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37694.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@126a9212
[junit4] 2> 186500 INFO (jetty-launcher-129-thread-1) [n:127.0.0.1:54807_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_54807.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@126a9212
[junit4] 2> 186500 INFO (jetty-launcher-129-thread-1) [n:127.0.0.1:54807_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_54807.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@126a9212
[junit4] 2> 186503 INFO (jetty-launcher-129-thread-1) [n:127.0.0.1:54807_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_2E3A421EDF6746B1-001/tempDir-001/node1/.
[junit4] 2> 186509 INFO (jetty-launcher-129-thread-2) [n:127.0.0.1:37694_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37694.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@126a9212
[junit4] 2> 186509 INFO (jetty-launcher-129-thread-2) [n:127.0.0.1:37694_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37694.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@126a9212
[junit4] 2> 186510 INFO (jetty-launcher-129-thread-2) [n:127.0.0.1:37694_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_2E3A421EDF6746B1-001/tempDir-001/node2/.
[junit4] 2> 186559 INFO (zkConnectionManagerCallback-154-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 186571 INFO (zkConnectionManagerCallback-158-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 186572 INFO (SUITE-AutoscalingHistoryHandlerTest-seed#[2E3A421EDF6746B1]-worker) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 186573 INFO (SUITE-AutoscalingHistoryHandlerTest-seed#[2E3A421EDF6746B1]-worker) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:46339/solr ready
[junit4] 2> 186764 INFO (qtp1097950914-794) [n:127.0.0.1:54807_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&name=.system&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:37694_solr&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 186774 INFO (OverseerThreadFactory-232-thread-1-processing-n:127.0.0.1:54807_solr) [n:127.0.0.1:54807_solr ] o.a.s.c.a.c.CreateCollectionCmd Create collection .system
[junit4] 2> 186781 DEBUG (OverseerStateUpdate-73353407156125701-127.0.0.1:54807_solr-n_0000000000) [n:127.0.0.1:54807_solr ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "name":".system",
[junit4] 2> "fromApi":"true",
[junit4] 2> "replicationFactor":"1",
[junit4] 2> "numShards":1,
[junit4] 2> "createNodeSet":"127.0.0.1:37694_solr",
[junit4] 2> "nrtReplicas":"1",
[junit4] 2> "stateFormat":"2",
[junit4] 2> "operation":"create"} current state version: 0
[junit4] 2> 186781 DEBUG (OverseerStateUpdate-73353407156125701-127.0.0.1:54807_solr-n_0000000000) [n:127.0.0.1:54807_solr ] o.a.s.c.o.ClusterStateMutator building a new cName: .system
[junit4] 2> 186781 DEBUG (OverseerStateUpdate-73353407156125701-127.0.0.1:54807_solr-n_0000000000) [n:127.0.0.1:54807_solr ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/.system/state.json
[junit4] 2> 186884 DEBUG (OverseerStateUpdate-73353407156125701-127.0.0.1:54807_solr-n_0000000000) [n:127.0.0.1:54807_solr ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":".system",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":".system_shard1_replica_n1",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:37694/solr",
[junit4] 2> "type":"NRT",
[junit4] 2> "waitForFinalState":"false"} current state version: 0
[junit4] 2> 186884 INFO (OverseerStateUpdate-73353407156125701-127.0.0.1:54807_solr-n_0000000000) [n:127.0.0.1:54807_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":".system",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":".system_shard1_replica_n1",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:37694/solr",
[junit4] 2> "type":"NRT",
[junit4] 2> "waitForFinalState":"false"}
[junit4] 2> 186887 DEBUG (OverseerStateUpdate-73353407156125701-127.0.0.1:54807_solr-n_0000000000) [n:127.0.0.1:54807_solr ] o.a.s.c.o.SliceMutator Old Slice: shard1:{
[junit4] 2> "range":"80000000-7fffffff",
[junit4] 2> "state":"active",
[junit4] 2> "replicas":{}}
[junit4] 2> 186887 DEBUG (OverseerStateUpdate-73353407156125701-127.0.0.1:54807_solr-n_0000000000) [n:127.0.0.1:54807_solr ] o.a.s.c.o.SliceMutator New Slice: shard1:{
[junit4] 2> "range":"80000000-7fffffff",
[junit4] 2> "state":"active",
[junit4] 2> "replicas":{"core_node2":{
[junit4] 2> "core":".system_shard1_replica_n1",
[junit4] 2> "base_url":"https://127.0.0.1:37694/solr",
[junit4] 2> "state":"down",
[junit4] 2> "node_name":null,
[junit4] 2> "type":"NRT"}}}
[junit4] 2> 186988 DEBUG (OverseerStateUpdate-73353407156125701-127.0.0.1:54807_solr-n_0000000000) [n:127.0.0.1:54807_solr ] o.a.s.c.o.ZkStateWriter going to update_collection /collections/.system/state.json version: 0
[junit4] 2> 187152 INFO (qtp148896360-804) [n:127.0.0.1:37694_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node2&collection.configName=.system&newCollection=true&name=.system_shard1_replica_n1&action=CREATE&numShards=1&collection=.system&shard=shard1&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 187152 INFO (qtp148896360-804) [n:127.0.0.1:37694_solr ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 187164 DEBUG (OverseerStateUpdate-73353407156125701-127.0.0.1:54807_solr-n_0000000000) [n:127.0.0.1:54807_solr ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "core":".system_shard1_replica_n1",
[junit4] 2> "core_node_name":"core_node2",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"https://127.0.0.1:37694/solr",
[junit4] 2> "node_name":"127.0.0.1:37694_solr",
[junit4] 2> "numShards":"1",
[junit4] 2> "state":"down",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":".system",
[junit4] 2> "type":"NRT",
[junit4] 2> "operation":"state"} current state version: 0
[junit4] 2> 187164 DEBUG (OverseerStateUpdate-73353407156125701-127.0.0.1:54807_solr-n_0000000000) [n:127.0.0.1:54807_solr ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
[junit4] 2> "core":".system_shard1_replica_n1",
[junit4] 2> "core_node_name":"core_node2",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"https://127.0.0.1:37694/solr",
[junit4] 2> "node_name":"127.0.0.1:37694_solr",
[junit4] 2> "numShards":"1",
[junit4] 2> "state":"down",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":".system",
[junit4] 2> "type":"NRT",
[junit4] 2> "operation":"state"}
[junit4] 2> 187164 DEBUG (OverseerStateUpdate-73353407156125701-127.0.0.1:54807_solr-n_0000000000) [n:127.0.0.1:54807_solr ] o.a.s.c.o.ReplicaMutator Will update state for replica: core_node2:{"core":".system_shard1_replica_n1","base_url":"https://127.0.0.1:37694/solr","node_name":"127.0.0.1:37694_solr","state":"down","type":"NRT"}
[junit4] 2> 187164 DEBUG (OverseerStateUpdate-73353407156125701-127.0.0.1:54807_solr-n_0000000000) [n:127.0.0.1:54807_solr ] o.a.s.c.o.ReplicaMutator Collection is now: DocCollection(.system//collections/.system/state.json/1)={
[junit4] 2> "pullReplicas":"0",
[junit4] 2> "replicationFactor":"1",
[junit4] 2> "router":{"name":"compositeId"},
[junit4] 2> "maxShardsPerNode":"1",
[junit4] 2> "autoAddReplicas":"false",
[junit4] 2> "nrtReplicas":"1",
[junit4] 2> "tlogReplicas":"0",
[junit4] 2> "shards":{"shard1":{
[junit4] 2> "range":"80000000-7fffffff",
[junit4] 2> "state":"active",
[junit4] 2> "replicas":{"core_node2":{
[junit4] 2> "core":".system_shard1_replica_n1",
[junit4] 2> "base_url":"https://127.0.0.1:37694/solr",
[junit4] 2> "node_name":"127.0.0.1:37694_solr",
[junit4] 2> "state":"down",
[junit4] 2> "type":"NRT"}}}}}
[junit4] 2> 187266 DEBUG (OverseerStateUpdate-73353407156125701-127.0.0.1:54807_solr-n_0000000000) [n:127.0.0.1:54807_solr ] o.a.s.c.o.ZkStateWriter going to update_collection /collections/.system/state.json version: 1
[junit4] 2> 187266 INFO (zkCallback-145-thread-2-processing-n:127.0.0.1:37694_solr) [n:127.0.0.1:37694_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [2])
[junit4] 2> 187266 INFO (zkCallback-145-thread-1-processing-n:127.0.0.1:37694_solr) [n:127.0.0.1:37694_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [2])
[junit4] 2> 187406 DEBUG (ScheduledTrigger-230-thread-1) [n:127.0.0.1:54807_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 188177 WARN (qtp148896360-804) [n:127.0.0.1:37694_solr c:.system s:shard1 r:core_node2 x:.system_shard1_replica_n1] o.a.s.c.Config You should not use LATEST as luceneMatchVersion property: if you use this setting, and then Solr upgrades to a newer release of Lucene, sizable changes may happen. If precise back compatibility is important then you should instead explicitly specify an actual Lucene version.
[junit4] 2> 188179 INFO (qtp148896360-804) [n:127.0.0.1:37694_solr c:.system s:shard1 r:core_node2 x:.system_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
[junit4] 2> 188207 INFO (qtp148896360-804) [n:127.0.0.1:37694_solr c:.system s:shard1 r:core_node2 x:.system_shard1_replica_n1] o.a.s.s.ManagedIndexSchemaFactory The schema is configured as managed, but managed schema resource managed-schema not found - loading non-managed schema schema.xml instead
[junit4] 2> 188213 INFO (qtp148896360-804) [n:127.0.0.1:37694_solr c:.system s:shard1 r:core_node2 x:.system_shard1_replica_n1] o.a.s.s.IndexSchema [.system_shard1_replica_n1] Schema name=_system collection or core
[junit4] 2> 188221 INFO (qtp148896360-804) [n:127.0.0.1:37694_solr c:.system s:shard1 r:core_node2 x:.system_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema _system collection or core/1.1 with uniqueid field id
[junit4] 2> 188233 INFO (qtp148896360-804) [n:127.0.0.1:37694_solr c:.system s:shard1 r:core_node2 x:.system_shard1_replica_n1] o.a.s.s.ManagedIndexSchema Created and persisted managed schema znode at /configs/.system/managed-schema
[junit4] 2> 188235 INFO (qtp148896360-804) [n:127.0.0.1:37694_solr c:.system s:shard1 r:core_node2 x:.system_shard1_replica_n1] o.a.s.s.ManagedIndexSchemaFactory After upgrading to managed schema in ZooKeeper, renamed the non-managed schema /configs/.system/schema.xml to /configs/.system/schema.xml.bak
[junit4] 2> 188235 INFO (qtp148896360-804) [n:127.0.0.1:37694_solr c:.system s:shard1 r:core_node2 x:.system_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore '.system_shard1_replica_n1' using configuration from collection .system, trusted=true
[junit4] 2> 188236 INFO (qtp148896360-804) [n:127.0.0.1:37694_solr c:.system s:shard1 r:core_node2 x:.system_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37694.solr.core..system.shard1.replica_n1' (registry 'solr.core..system.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@126a9212
[junit4] 2> 188236 INFO (qtp148896360-804) [n:127.0.0.1:37694_solr c:.system s:shard1 r:core_node2 x:.system_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 188236 INFO (qtp148896360-804) [n:127.0.0.1:37694_solr c:.system s:shard1 r:core_node2 x:.system_shard1_replica_n1] o.a.s.c.SolrCore [[.system_shard1_replica_n1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_2E3A421EDF6746B1-001/tempDir-001/node2/.system_shard1_replica_n1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_2E3A421EDF6746B1-001/tempDir-001/node2/./.system_shard1_replica_n1/data/]
[junit4] 2> 188296 INFO (qtp148896360-804) [n:127.0.0.1:37694_solr c:.system s:shard1 r:core_node2 x:.system_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 188296 INFO (qtp148896360-804) [n:127.0.0.1:37694_solr c:.system s:shard1 r:core_node2 x:.system_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 188297 INFO (qtp148896360-804) [n:127.0.0.1:37694_solr c:.system s:shard1 r:core_node2 x:.system_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: if 1 uncommited docs
[junit4] 2> 188297 INFO (qtp148896360-804) [n:127.0.0.1:37694_solr c:.system s:shard1 r:core_node2 x:.system_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 188299 INFO (qtp148896360-804) [n:127.0.0.1:37694_solr c:.system s:shard1 r:core_node2 x:.system_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@11fa0f2e[.system_shard1_replica_n1] main]
[junit4] 2> 188300 INFO (qtp148896360-804) [n:127.0.0.1:37694_solr c:.system s:shard1 r:core_node2 x:.system_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/.system
[junit4] 2> 188301 INFO (qtp148896360-804) [n:127.0.0.1:37694_solr c:.system s:shard1 r:core_node2 x:.system_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/.system
[junit4] 2> 188307 INFO (qtp148896360-804) [n:127.0.0.1:37694_solr c:.system s:shard1 r:core_node2 x:.system_shard1_replica_n1] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/.system/managed-schema
[junit4] 2> 188308 INFO (qtp148896360-804) [n:127.0.0.1:37694_solr c:.system s:shard1 r:core_node2 x:.system_shard1_replica_n1] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
[junit4] 2> 188309 INFO (qtp148896360-804) [n:127.0.0.1:37694_solr c:.system s:shard1 r:core_node2 x:.system_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 188309 INFO (qtp148896360-804) [n:127.0.0.1:37694_solr c:.system s:shard1 r:core_node2 x:.system_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1593332868428333056
[junit4] 2> 188313 INFO (searcherExecutor-237-thread-1-processing-n:127.0.0.1:37694_solr x:.system_shard1_replica_n1 s:shard1 c:.system r:core_node2) [n:127.0.0.1:37694_solr c:.system s:shard1 r:core_node2 x:.system_shard1_replica_n1] o.a.s.c.SolrCore [.system_shard1_replica_n1] Registered new searcher Searcher@11fa0f2e[.system_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 188331 INFO (qtp148896360-804) [n:127.0.0.1:37694_solr c:.system s:shard1 r:core_node2 x:.system_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 188331 INFO (qtp148896360-804) [n:127.0.0.1:37694_solr c:.system s:shard1 r:core_node2 x:.system_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 188331 INFO (qtp148896360-804) [n:127.0.0.1:37694_solr c:.system s:shard1 r:core_node2 x:.system_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:37694/solr/.system_shard1_replica_n1/
[junit4] 2> 188331 INFO (qtp148896360-804) [n:127.0.0.1:37694_solr c:.system s:shard1 r:core_node2 x:.system_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 188331 INFO (qtp148896360-804) [n:127.0.0.1:37694_solr c:.system s:shard1 r:core_node2 x:.system_shard1_replica_n1] o.a.s.c.SyncStrategy https://127.0.0.1:37694/solr/.system_shard1_replica_n1/ has no replicas
[junit4] 2> 188331 INFO (qtp148896360-804) [n:127.0.0.1:37694_solr c:.system s:shard1 r:core_node2 x:.system_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 188335 INFO (qtp148896360-804) [n:127.0.0.1:37694_solr c:.system s:shard1 r:core_node2 x:.system_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:37694/solr/.system_shard1_replica_n1/ shard1
[junit4] 2> 188335 DEBUG (OverseerStateUpdate-73353407156125701-127.0.0.1:54807_solr-n_0000000000) [n:127.0.0.1:54807_solr ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":".system",
[junit4] 2> "base_url":"https://127.0.0.1:37694/solr",
[junit4] 2> "core":".system_shard1_replica_n1",
[junit4] 2> "state":"active"} current state version: 0
[junit4] 2> 188406 DEBUG (ScheduledTrigger-230-thread-2) [n:127.0.0.1:54807_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 188436 DEBUG (OverseerStateUpdate-73353407156125701-127.0.0.1:54807_solr-n_0000000000) [n:127.0.0.1:54807_solr ] o.a.s.c.o.ZkStateWriter going to update_collection /collections/.system/state.json version: 2
[junit4] 2> 188436 INFO (zkCallback-145-thread-2-processing-n:127.0.0.1:37694_solr) [n:127.0.0.1:37694_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [2])
[junit4] 2> 188436 INFO (zkCallback-145-thread-1-processing-n:127.0.0.1:37694_solr) [n:127.0.0.1:37694_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [2])
[junit4] 2> 188486 INFO (qtp148896360-804) [n:127.0.0.1:37694_solr c:.system s:shard1 r:core_node2 x:.system_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 188488 INFO (qtp148896360-804) [n:127.0.0.1:37694_solr c:.system s:shard1 r:core_node2 x:.system_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node2&collection.configName=.system&newCollection=true&name=.system_shard1_replica_n1&action=CREATE&numShards=1&collection=.system&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1336
[junit4] 2> 188500 DEBUG (OverseerStateUpdate-73353407156125701-127.0.0.1:54807_solr-n_0000000000) [n:127.0.0.1:54807_solr ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "core":".system_shard1_replica_n1",
[junit4] 2> "core_node_name":"core_node2",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"https://127.0.0.1:37694/solr",
[junit4] 2> "node_name":"127.0.0.1:37694_solr",
[junit4] 2> "numShards":"1",
[junit4] 2> "state":"active",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":".system",
[junit4] 2> "type":"NRT",
[junit4] 2> "operation":"state"} current state version: 0
[junit4] 2> 188500 DEBUG (OverseerStateUpdate-73353407156125701-127.0.0.1:54807_solr-n_0000000000) [n:127.0.0.1:54807_solr ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
[junit4] 2> "core":".system_shard1_replica_n1",
[junit4] 2> "core_node_name":"core_node2",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"https://127.0.0.1:37694/solr",
[junit4] 2> "node_name":"127.0.0.1:37694_solr",
[junit4] 2> "numShards":"1",
[junit4] 2> "state":"active",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":".system",
[junit4] 2> "type":"NRT",
[junit4] 2> "operation":"state"}
[junit4] 2> 188500 DEBUG (OverseerStateUpdate-73353407156125701-127.0.0.1:54807_solr-n_0000000000) [n:127.0.0.1:54807_solr ] o.a.s.c.o.ReplicaMutator Will update state for replica: core_node2:{"core":".system_shard1_replica_n1","base_url":"https://127.0.0.1:37694/solr","node_name":"127.0.0.1:37694_solr","state":"active","type":"NRT","leader":"true"}
[junit4] 2> 188500 DEBUG (OverseerStateUpdate-73353407156125701-127.0.0.1:54807_solr-n_0000000000) [n:127.0.0.1:54807_solr ] o.a.s.c.o.ReplicaMutator Collection is now: DocCollection(.system//collections/.system/state.json/3)={
[junit4] 2> "pullReplicas":"0",
[junit4] 2> "replicationFactor":"1",
[junit4] 2> "router":{"name":"compositeId"},
[junit4] 2> "maxShardsPerNode":"1",
[junit4] 2> "autoAddReplicas":"false",
[junit4] 2> "nrtReplicas":"1",
[junit4] 2> "tlogReplicas":"0",
[junit4] 2> "shards":{"shard1":{
[junit4] 2> "range":"80000000-7fffffff",
[junit4] 2> "state":"active",
[junit4] 2> "replicas":{"core_node2":{
[junit4] 2> "core":".system_shard1_replica_n1",
[junit4] 2> "base_url":"https://127.0.0.1:37694/solr",
[junit4] 2> "node_name":"127.0.0.1:37694_solr",
[junit4] 2> "state":"active",
[junit4] 2> "type":"NRT",
[junit4] 2> "leader":"true"}}}}}
[junit4] 2> 188502 INFO (qtp1097950914-794) [n:127.0.0.1:54807_solr ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
[junit4] 2> 188601 DEBUG (OverseerStateUpdate-73353407156125701-127.0.0.1:54807_solr-n_0000000000) [n:127.0.0.1:54807_solr ] o.a.s.c.o.ZkStateWriter going to update_collection /collections/.system/state.json version: 3
[junit4] 2> 188602 INFO (zkCallback-145-thread-2-processing-n:127.0.0.1:37694_solr) [n:127.0.0.1:37694_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [2])
[junit4] 2> 188602 INFO (zkCallback-145-thread-1-processing-n:127.0.0.1:37694_solr) [n:127.0.0.1:37694_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [2])
[junit4] 2> 188775 INFO (OverseerCollectionConfigSetProcessor-73353407156125701-127.0.0.1:54807_solr-n_0000000000) [n:127.0.0.1:54807_solr ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 189407 DEBUG (ScheduledTrigger-230-thread-1) [n:127.0.0.1:54807_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 189503 INFO (qtp1097950914-794) [n:127.0.0.1:54807_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&name=.system&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:37694_solr&wt=javabin&version=2} status=0 QTime=2739
[junit4] 2> 189507 INFO (qtp1097950914-796) [n:127.0.0.1:54807_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=3&maxShardsPerNode=3&name=AutoscalingHistoryHandlerTest_collection&nrtReplicas=3&action=CREATE&numShards=1&createNodeSet=127.0.0.1:54807_solr&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 189510 INFO (OverseerThreadFactory-232-thread-2-processing-n:127.0.0.1:54807_solr) [n:127.0.0.1:54807_solr ] o.a.s.c.a.c.CreateCollectionCmd Create collection AutoscalingHistoryHandlerTest_collection
[junit4] 2> 189513 INFO (OverseerThreadFactory-232-thread-2-processing-n:127.0.0.1:54807_solr) [n:127.0.0.1:54807_solr ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/managed-schema to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/managed-schema
[junit4] 2> 189517 INFO (OverseerThreadFactory-232-thread-2-processing-n:127.0.0.1:54807_solr) [n:127.0.0.1:54807_solr ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/protwords.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/protwords.txt
[junit4] 2> 189518 INFO (OverseerThreadFactory-232-thread-2-processing-n:127.0.0.1:54807_solr) [n:127.0.0.1:54807_solr ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/solrconfig.xml to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/solrconfig.xml
[junit4] 2> 189520 INFO (OverseerThreadFactory-232-thread-2-processing-n:127.0.0.1:54807_solr) [n:127.0.0.1:54807_solr ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/synonyms.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/synonyms.txt
[junit4] 2> 189521 INFO (OverseerThreadFactory-232-thread-2-processing-n:127.0.0.1:54807_solr) [n:127.0.0.1:54807_solr ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/stopwords.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/stopwords.txt
[junit4] 2> 189523 INFO (OverseerThreadFactory-232-thread-2-processing-n:127.0.0.1:54807_solr) [n:127.0.0.1:54807_solr ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/contractions_it.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/contractions_it.txt
[junit4] 2> 189524 INFO (OverseerThreadFactory-232-thread-2-processing-n:127.0.0.1:54807_solr) [n:127.0.0.1:54807_solr ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/contractions_ca.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/contractions_ca.txt
[junit4] 2> 189526 INFO (OverseerThreadFactory-232-thread-2-processing-n:127.0.0.1:54807_solr) [n:127.0.0.1:54807_solr ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stemdict_nl.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/stemdict_nl.txt
[junit4] 2> 189527 INFO (OverseerThreadFactory-232-thread-2-processing-n:127.0.0.1:54807_solr) [n:127.0.0.1:54807_solr ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_no.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/stopwords_no.txt
[junit4] 2> 189528 INFO (OverseerThreadFactory-232-thread-2-processing-n:127.0.0.1:54807_solr) [n:127.0.0.1:54807_solr ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_hy.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/stopwords_hy.txt
[junit4] 2> 189532 INFO (OverseerThreadFactory-232-thread-2-processing-n:127.0.0.1:54807_solr) [n:127.0.0.1:54807_solr ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_id.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/stopwords_id.txt
[junit4] 2> 189534 INFO (OverseerThreadFactory-232-thread-2-processing-n:127.0.0.1:54807_solr) [n:127.0.0.1:54807_solr ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_hi.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/stopwords_hi.txt
[junit4] 2> 189535 INFO (OverseerThreadFactory-232-thread-2-processing-n:127.0.0.1:54807_solr) [n:127.0.0.1:54807_solr ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_ga.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/stopwords_ga.txt
[junit4] 2> 189536 INFO (OverseerThreadFactory-232-thread-2-processing-n:127.0.0.1:54807_solr) [n:127.0.0.1:54807_solr ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_da.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/stopwords_da.txt
[junit4] 2> 189537 INFO (OverseerThreadFactory-232-thread-2-processing-n:127.0.0.1:54807_solr) [n:127.0.0.1:54807_solr ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_ja.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/stopwords_ja.txt
[junit4] 2> 189539 INFO (OverseerThreadFactory-232-thread-2-processing-n:127.0.0.1:54807_solr) [n:127.0.0.1:54807_solr ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_pt.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/stopwords_pt.txt
[junit4] 2> 189541 INFO (OverseerThreadFactory-232-thread-2-processing-n:127.0.0.1:54807_solr) [n:127.0.0.1:54807_solr ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_hu.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/stopwords_hu.txt
[junit4] 2> 189542 INFO (OverseerThreadFactory-232-thread-2-processing-n:127.0.0.1:54807_solr) [n:127.0.0.1:54807_solr ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_el.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/stopwords_el.txt
[junit4] 2> 189543 INFO (OverseerThreadFactory-232-thread-2-processing-n:127.0.0.1:54807_solr) [n:127.0.0.1:54807_solr ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_ru.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/stopwords_ru.txt
[junit4] 2> 189544 INFO (OverseerThreadFactory-232-thread-2-processing-n:127.0.0.1:54807_solr) [n:127.0.0.1:54807_solr ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_tr.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/stopwords_tr.txt
[junit4] 2> 189545 INFO (OverseerThreadFactory-232-thread-2-processing-n:127.0.0.1:54807_solr) [n:127.0.0.1:54807_solr ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_ar.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/stopwords_ar.txt
[junit4] 2> 189546 INFO (OverseerThreadFactory-232-thread-2-processing-n:127.0.0.1:54807_solr) [n:127.0.0.1:54807_solr ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/userdict_ja.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/userdict_ja.txt
[junit4] 2> 189547 INFO (OverseerThreadFactory-232-thread-2-processing-n:127.0.0.1:54807_solr) [n:127.0.0.1:54807_solr ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_eu.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/stopwords_eu.txt
[junit4] 2> 189548 INFO (OverseerThreadFactory-232-thread-2-processing-n:127.0.0.1:54807_solr) [n:127.0.0.1:54807_solr ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_it.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/stopwords_it.txt
[junit4] 2> 189551 INFO (OverseerThreadFactory-232-thread-2-processing-n:127.0.0.1:54807_solr) [n:127.0.0.1:54807_solr ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_cz.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/stopwords_cz.txt
[junit4] 2> 189552 INFO (OverseerThreadFactory-232-thread-2-processing-n:127.0.0.1:54807_solr) [n:127.0.0.1:54807_solr ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stoptags_ja.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/stoptags_ja.txt
[junit4] 2> 189553 INFO (OverseerThreadFactory-232-thread-2-processing-n:127.0.0.1:54807_solr) [n:127.0.0.1:54807_solr ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/contractions_fr.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/contractions_fr.txt
[junit4] 2> 189557 INFO (OverseerThreadFactory-232-thread-2-processing-n:127.0.0.1:54807_solr) [n:127.0.0.1:54807_solr ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_de.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/stopwords_de.txt
[junit4] 2> 189558 INFO (OverseerThreadFactory-232-thread-2-processing-n:127.0.0.1:54807_solr) [n:127.0.0.1:54807_solr ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_fa.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/stopwords_fa.txt
[junit4] 2> 189559 INFO (OverseerThreadFactory-232-thread-2-processing-n:127.0.0.1:54807_solr) [n:127.0.0.1:54807_solr ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/contractions_ga.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/contractions_ga.txt
[junit4] 2> 189560 INFO (OverseerThreadFactory-232-thread-2-processing-n:127.0.0.1:54807_solr) [n:127.0.0.1:54807_solr ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_ca.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/stopwords_ca.txt
[junit4] 2> 189561 INFO (OverseerThreadFactory-232-thread-2-processing-n:127.0.0.1:54807_solr) [n:127.0.0.1:54807_solr ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_nl.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/stopwords_nl.txt
[junit4] 2> 189562 INFO (OverseerThreadFactory-232-thread-2-processing-n:127.0.0.1:54807_solr) [n:127.0.0.1:54807_solr ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_bg.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/stopwords_bg.txt
[junit4] 2> 189563 INFO (OverseerThreadFactory-232-thread-2-processing-n:127.0.0.1:54807_solr) [n:127.0.0.1:54807_solr ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_en.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/stopwords_en.txt
[junit4] 2> 189564 INFO (OverseerThreadFactory-232-thread-2-processing-n:127.0.0.1:54807_solr) [n:127.0.0.1:54807_solr ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_es.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/stopwords_es.txt
[junit4] 2> 189564 INFO (OverseerThreadFactory-232-thread-2-processing-n:127.0.0.1:54807_solr) [n:127.0.0.1:54807_solr ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_lv.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/stopwords_lv.txt
[junit4] 2> 189565 INFO (OverseerThreadFactory-232-thread-2-processing-n:127.0.0.1:54807_solr) [n:127.0.0.1:54807_solr ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_ro.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/stopwords_ro.txt
[junit4] 2> 189566 INFO (OverseerThreadFactory-232-thread-2-processing-n:127.0.0.1:54807_solr) [n:127.0.0.1:54807_solr ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_th.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/stopwords_th.txt
[junit4] 2> 189567 INFO (OverseerThreadFactory-232-thread-2-processing-n:127.0.0.1:54807_solr) [n:127.0.0.1:54807_solr ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_fr.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/stopwords_fr.txt
[junit4] 2> 189568 INFO (OverseerThreadFactory-232-thread-2-processing-n:127.0.0.1:54807_solr) [n:127.0.0.1:54807_solr ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_sv.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/stopwords_sv.txt
[junit4] 2> 189569 INFO (OverseerThreadFactory-232-thread-2-processing-n:127.0.0.1:54807_solr) [n:127.0.0.1:54807_solr ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_fi.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/stopwords_fi.txt
[junit4] 2> 189571 INFO (OverseerThreadFactory-232-thread-2-processing-n:127.0.0.1:54807_solr) [n:127.0.0.1:54807_solr ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/hyphenations_ga.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/hyphenations_ga.txt
[junit4] 2> 189572 INFO (OverseerThreadFactory-232-thread-2-processing-n:127.0.0.1:54807_solr) [n:127.0.0.1:54807_solr ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_gl.txt to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/lang/stopwords_gl.txt
[junit4] 2> 189573 INFO (OverseerThreadFactory-232-thread-2-processing-n:127.0.0.1:54807_solr) [n:127.0.0.1:54807_solr ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/params.json to /configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/params.json
[junit4] 2> 189574 WARN (OverseerThreadFactory-232-thread-2-processing-n:127.0.0.1:54807_solr) [n:127.0.0.1:54807_solr ] o.a.s.c.a.c.CreateCollectionCmd Specified number of replicas of 3 on collection AutoscalingHistoryHandlerTest_collection is higher than the number of Solr instances currently live or live and part of your createNodeSet(1). It's unusual to run two replica of the same slice on the same Solr-instance.
[junit4] 2> 189577 DEBUG (OverseerStateUpdate-73353407156125701-127.0.0.1:54807_solr-n_0000000000) [n:127.0.0.1:54807_solr ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "name":"AutoscalingHistoryHandlerTest_collection",
[junit4] 2> "fromApi":"true",
[junit4] 2> "replicationFactor":"3",
[junit4] 2> "numShards":"1",
[junit4] 2> "maxShardsPerNode":"3",
[junit4] 2> "createNodeSet":"127.0.0.1:54807_solr",
[junit4] 2> "nrtReplicas":"3",
[junit4] 2> "stateFormat":"2",
[junit4] 2> "operation":"create"} current state version: 0
[junit4] 2> 189578 DEBUG (OverseerStateUpdate-73353407156125701-127.0.0.1:54807_solr-n_0000000000) [n:127.0.0.1:54807_solr ] o.a.s.c.o.ClusterStateMutator building a new cName: AutoscalingHistoryHandlerTest_collection
[junit4] 2> 189678 DEBUG (OverseerStateUpdate-73353407156125701-127.0.0.1:54807_solr-n_0000000000) [n:127.0.0.1:54807_solr ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/AutoscalingHistoryHandlerTest_collection/state.json
[junit4] 2> 189782 DEBUG (OverseerStateUpdate-73353407156125701-127.0.0.1:54807_solr-n_0000000000) [n:127.0.0.1:54807_solr ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"AutoscalingHistoryHandlerTest_collection",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n1",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:54807/solr",
[junit4] 2> "type":"NRT",
[junit4] 2> "waitForFinalState":"false"} current state version: 0
[junit4] 2> 189782 INFO (OverseerStateUpdate-73353407156125701-127.0.0.1:54807_solr-n_0000000000) [n:127.0.0.1:54807_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"AutoscalingHistoryHandlerTest_collection",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n1",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:54807/solr",
[junit4] 2> "type":"NRT",
[junit4] 2> "waitForFinalState":"false"}
[junit4] 2> 189784 DEBUG (OverseerStateUpdate-73353407156125701-127.0.0.1:54807_solr-n_0000000000) [n:127.0.0.1:54807_solr ] o.a.s.c.o.SliceMutator Old Slice: shard1:{
[junit4] 2> "range":"80000000-7fffffff",
[junit4] 2> "state":"active",
[junit4] 2> "replicas":{}}
[junit4] 2> 189784 DEBUG (OverseerStateUpdate-73353407156125701-127.0.0.1:54807_solr-n_0000000000) [n:127.0.0.1:54807_solr ] o.a.s.c.o.SliceMutator New Slice: shard1:{
[junit4] 2> "range":"80000000-7fffffff",
[junit4] 2> "state":"active",
[junit4] 2> "replicas":{"core_node3":{
[junit4] 2> "core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n1",
[junit4] 2> "base_url":"https://127.0.0.1:54807/solr",
[junit4] 2> "state":"down",
[junit4] 2> "node_name":null,
[junit4] 2> "type":"NRT"}}}
[junit4] 2> 189785 DEBUG (OverseerStateUpdate-73353407156125701-127.0.0.1:54807_solr-n_0000000000) [n:127.0.0.1:54807_solr ] o.a.s.c.Overseer processMessage: queueSize: 2, message = {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"AutoscalingHistoryHandlerTest_collection",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n2",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:54807/solr",
[junit4] 2> "type":"NRT",
[junit4] 2> "waitForFinalState":"false"} current state version: 0
[junit4] 2> 189785 INFO (OverseerStateUpdate-73353407156125701-127.0.0.1:54807_solr-n_0000000000) [n:127.0.0.1:54807_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"AutoscalingHistoryHandlerTest_collection",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n2",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:54807/solr",
[junit4] 2> "type":"NRT",
[junit4] 2> "waitForFinalState":"false"}
[junit4] 2> 189786 DEBUG (OverseerStateUpdate-73353407156125701-127.0.0.1:54807_solr-n_0000000000) [n:127.0.0.1:54807_solr ] o.a.s.c.o.SliceMutator Old Slice: shard1:{
[junit4] 2> "range":"80000000-7fffffff",
[junit4] 2> "state":"active",
[junit4] 2> "replicas":{"core_node3":{
[junit4] 2> "core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n1",
[junit4] 2> "base_url":"https://127.0.0.1:54807/solr",
[junit4] 2> "state":"down",
[junit4] 2> "node_name":null,
[junit4] 2> "type":"NRT"}}}
[junit4] 2> 189786 DEBUG (OverseerStateUpdate-73353407156125701-127.0.0.1:54807_solr-n_0000000000) [n:127.0.0.1:54807_solr ] o.a.s.c.o.SliceMutator New Slice: shard1:{
[junit4] 2> "range":"80000000-7fffffff",
[junit4] 2> "state":"active",
[junit4] 2> "replicas":{
[junit4] 2> "core_node3":{
[junit4] 2> "core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n1",
[junit4] 2> "base_url":"https://127.0.0.1:54807/solr",
[junit4] 2> "state":"down",
[junit4] 2> "node_name":null,
[junit4] 2> "type":"NRT"},
[junit4] 2> "core_node5":{
[junit4] 2> "core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n2",
[junit4] 2> "base_url":"https://127.0.0.1:54807/solr",
[junit4] 2> "state":"down",
[junit4] 2> "node_name":null,
[junit4] 2> "type":"NRT"}}}
[junit4] 2> 189787 DEBUG (OverseerStateUpdate-73353407156125701-127.0.0.1:54807_solr-n_0000000000) [n:127.0.0.1:54807_solr ] o.a.s.c.Overseer processMessage: queueSize: 3, message = {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"AutoscalingHistoryHandlerTest_collection",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n4",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:54807/solr",
[junit4] 2> "type":"NRT",
[junit4] 2> "waitForFinalState":"false"} current state version: 0
[junit4] 2> 189787 INFO (OverseerStateUpdate-73353407156125701-127.0.0.1:54807_solr-n_0000000000) [n:127.0.0.1:54807_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"AutoscalingHistoryHandlerTest_collection",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n4",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:54807/solr",
[junit4] 2> "type":"NRT",
[junit4] 2> "waitForFinalState":"false"}
[junit4] 2> 189788 DEBUG (OverseerStateUpdate-73353407156125701-127.0.0.1:54807_solr-n_0000000000) [n:127.0.0.1:54807_solr ] o.a.s.c.o.SliceMutator Old Slice: shard1:{
[junit4] 2> "range":"80000000-7fffffff",
[junit4] 2> "state":"active",
[junit4] 2> "replicas":{
[junit4] 2> "core_node3":{
[junit4] 2> "core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n1",
[junit4] 2> "base_url":"https://127.0.0.1:54807/solr",
[junit4] 2> "state":"down",
[junit4] 2> "node_name":null,
[junit4] 2> "type":"NRT"},
[junit4] 2> "core_node5":{
[junit4] 2> "core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n2",
[junit4] 2> "base_url":"https://127.0.0.1:54807/solr",
[junit4] 2> "state":"down",
[junit4] 2> "node_name":null,
[junit4] 2> "type":"NRT"}}}
[junit4] 2> 189788 DEBUG (OverseerStateUpdate-73353407156125701-127.0.0.1:54807_solr-n_0000000000) [n:127.0.0.1:54807_solr ] o.a.s.c.o.SliceMutator New Slice: shard1:{
[junit4] 2> "range":"80000000-7fffffff",
[junit4] 2> "state":"active",
[junit4] 2> "replicas":{
[junit4] 2> "core_node3":{
[junit4] 2> "core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n1",
[junit4] 2> "base_url":"https://127.0.0.1:54807/solr",
[junit4] 2> "state":"down",
[junit4] 2> "node_name":null,
[junit4] 2> "type":"NRT"},
[junit4] 2> "core_node5":{
[junit4] 2> "core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n2",
[junit4] 2> "base_url":"https://127.0.0.1:54807/solr",
[junit4] 2> "state":"down",
[junit4] 2> "node_name":null,
[junit4] 2> "type":"NRT"},
[junit4] 2> "core_node6":{
[junit4] 2> "core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n4",
[junit4] 2> "base_url":"https://127.0.0.1:54807/solr",
[junit4] 2> "state":"down",
[junit4] 2> "node_name":null,
[junit4] 2> "type":"NRT"}}}
[junit4] 2> 189888 DEBUG (OverseerStateUpdate-73353407156125701-127.0.0.1:54807_solr-n_0000000000) [n:127.0.0.1:54807_solr ] o.a.s.c.o.ZkStateWriter going to update_collection /collections/AutoscalingHistoryHandlerTest_collection/state.json version: 0
[junit4] 2> 190008 INFO (qtp1097950914-791) [n:127.0.0.1:54807_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=AutoscalingHistoryHandlerTest_collection.AUTOCREATED&newCollection=true&name=AutoscalingHistoryHandlerTest_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=AutoscalingHistoryHandlerTest_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 190009 INFO (qtp1097950914-791) [n:127.0.0.1:54807_solr ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 190018 DEBUG (OverseerStateUpdate-73353407156125701-127.0.0.1:54807_solr-n_0000000000) [n:127.0.0.1:54807_solr ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n1",
[junit4] 2> "core_node_name":"core_node3",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"https://127.0.0.1:54807/solr",
[junit4] 2> "node_name":"127.0.0.1:54807_solr",
[junit4] 2> "numShards":"1",
[junit4] 2> "state":"down",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"AutoscalingHistoryHandlerTest_collection",
[junit4] 2> "type":"NRT",
[junit4] 2> "operation":"state"} current state version: 0
[junit4] 2> 190018 DEBUG (OverseerStateUpdate-73353407156125701-127.0.0.1:54807_solr-n_0000000000) [n:127.0.0.1:54807_solr ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
[junit4] 2> "core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n1",
[junit4] 2> "core_node_name":"core_node3",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"https://127.0.0.1:54807/solr",
[junit4] 2> "node_name":"127.0.0.1:54807_solr",
[junit4] 2> "numShards":"1",
[junit4] 2> "state":"down",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"AutoscalingHistoryHandlerTest_collection",
[junit4] 2> "type":"NRT",
[junit4] 2> "operation":"state"}
[junit4] 2> 190018 DEBUG (OverseerStateUpdate-73353407156125701-127.0.0.1:54807_solr-n_0000000000) [n:127.0.0.1:54807_solr ] o.a.s.c.o.ReplicaMutator Will update state for replica: core_node3:{"core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n1","base_url":"https://127.0.0.1:54807/solr","node_name":"127.0.0.1:54807_solr","state":"down","type":"NRT"}
[junit4] 2> 190018 DEBUG (OverseerStateUpdate-73353407156125701-127.0.0.1:54807_solr-n_0000000000) [n:127.0.0.1:54807_solr ] o.a.s.c.o.ReplicaMutator Collection is now: DocCollection(AutoscalingHistoryHandlerTest_collection//collections/AutoscalingHistoryHandlerTest_collection/state.json/1)={
[junit4] 2> "pullReplicas":"0",
[junit4] 2> "replicationFactor":"3",
[junit4] 2> "router":{"name":"compositeId"},
[junit4] 2> "maxShardsPerNode":"3",
[junit4] 2> "autoAddReplicas":"false",
[junit4] 2> "nrtReplicas":"3",
[junit4] 2> "tlogReplicas":"0",
[junit4] 2> "shards":{"shard1":{
[junit4] 2> "range":"80000000-7fffffff",
[junit4] 2> "state":"active",
[junit4] 2> "replicas":{
[junit4] 2> "core_node3":{
[junit4] 2> "core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n1",
[junit4] 2> "base_url":"https://127.0.0.1:54807/solr",
[junit4] 2> "node_name":"127.0.0.1:54807_solr",
[junit4] 2> "state":"down",
[junit4] 2> "type":"NRT"},
[junit4] 2> "core_node5":{
[junit4] 2> "core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n2",
[junit4] 2> "base_url":"https://127.0.0.1:54807/solr",
[junit4] 2> "state":"down",
[junit4] 2> "node_name":null,
[junit4] 2> "type":"NRT"},
[junit4] 2> "core_node6":{
[junit4] 2> "core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n4",
[junit4] 2> "base_url":"https://127.0.0.1:54807/solr",
[junit4] 2> "state":"down",
[junit4] 2> "node_name":null,
[junit4] 2> "type":"NRT"}}}}}
[junit4] 2> 190050 INFO (qtp1097950914-868) [n:127.0.0.1:54807_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node6&collection.configName=AutoscalingHistoryHandlerTest_collection.AUTOCREATED&newCollection=true&name=AutoscalingHistoryHandlerTest_collection_shard1_replica_n4&action=CREATE&numShards=1&collection=AutoscalingHistoryHandlerTest_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 190050 INFO (qtp1097950914-794) [n:127.0.0.1:54807_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node5&collection.configName=AutoscalingHistoryHandlerTest_collection.AUTOCREATED&newCollection=true&name=AutoscalingHistoryHandlerTest_collection_shard1_replica_n2&action=CREATE&numShards=1&collection=AutoscalingHistoryHandlerTest_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 190052 DEBUG (OverseerStateUpdate-73353407156125701-127.0.0.1:54807_solr-n_0000000000) [n:127.0.0.1:54807_solr ] o.a.s.c.Overseer processMessage: queueSize: 2, message = {
[junit4] 2> "core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n4",
[junit4] 2> "core_node_name":"core_node6",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"https://127.0.0.1:54807/solr",
[junit4] 2> "node_name":"127.0.0.1:54807_solr",
[junit4] 2> "numShards":"1",
[junit4] 2> "state":"down",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"AutoscalingHistoryHandlerTest_collection",
[junit4] 2> "type":"NRT",
[junit4] 2> "operation":"state"} current state version: 0
[junit4] 2> 190052 DEBUG (OverseerStateUpdate-73353407156125701-127.0.0.1:54807_solr-n_0000000000) [n:127.0.0.1:54807_solr ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
[junit4] 2> "core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n4",
[junit4] 2> "core_node_name":"core_node6",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"https://127.0.0.1:54807/solr",
[junit4] 2> "node_name":"127.0.0.1:54807_solr",
[junit4] 2> "numShards":"1",
[junit4] 2> "state":"down",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"AutoscalingHistoryHandlerTest_collection",
[junit4] 2> "type":"NRT",
[junit4] 2> "operation":"state"}
[junit4] 2> 190052 DEBUG (OverseerStateUpdate-73353407156125701-127.0.0.1:54807_solr-n_0000000000) [n:127.0.0.1:54807_solr ] o.a.s.c.o.ReplicaMutator Will update state for replica: core_node6:{"core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n4","base_url":"https://127.0.0.1:54807/solr","node_name":"127.0.0.1:54807_solr","state":"down","type":"NRT"}
[junit4] 2> 190052 DEBUG (OverseerStateUpdate-73353407156125701-127.0.0.1:54807_solr-n_0000000000) [n:127.0.0.1:54807_solr ] o.a.s.c.o.ReplicaMutator Collection is now: DocCollection(AutoscalingHistoryHandlerTest_collection//collections/AutoscalingHistoryHandlerTest_collection/state.json/1)={
[junit4] 2> "pullReplicas":"0",
[junit4] 2> "replicationFactor":"3",
[junit4] 2> "router":{"name":"compositeId"},
[junit4] 2> "maxShardsPerNode":"3",
[junit4] 2> "autoAddReplicas":"false",
[junit4] 2> "nrtReplicas":"3",
[junit4] 2> "tlogReplicas":"0",
[junit4] 2> "shards":{"shard1":{
[junit4] 2> "range":"80000000-7fffffff",
[junit4] 2> "state":"active",
[junit4] 2> "replicas":{
[junit4] 2> "core_node3":{
[junit4] 2> "core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n1",
[junit4] 2> "base_url":"https://127.0.0.1:54807/solr",
[junit4] 2> "node_name":"127.0.0.1:54807_solr",
[junit4] 2> "state":"down",
[junit4] 2> "type":"NRT"},
[junit4] 2> "core_node5":{
[junit4] 2> "core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n2",
[junit4] 2> "base_url":"https://127.0.0.1:54807/solr",
[junit4] 2> "state":"down",
[junit4] 2> "node_name":null,
[junit4] 2> "type":"NRT"},
[junit4] 2> "core_node6":{
[junit4] 2> "core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n4",
[junit4] 2> "base_url":"https://127.0.0.1:54807/solr",
[junit4] 2> "node_name":"127.0.0.1:54807_solr",
[junit4] 2> "state":"down",
[junit4] 2> "type":"NRT"}}}}}
[junit4] 2> 190053 DEBUG (OverseerStateUpdate-73353407156125701-127.0.0.1:54807_solr-n_0000000000) [n:127.0.0.1:54807_solr ] o.a.s.c.Overseer processMessage: queueSize: 3, message = {
[junit4] 2> "core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n2",
[junit4] 2> "core_node_name":"core_node5",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"https://127.0.0.1:54807/solr",
[junit4] 2> "node_name":"127.0.0.1:54807_solr",
[junit4] 2> "numShards":"1",
[junit4] 2> "state":"down",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"AutoscalingHistoryHandlerTest_collection",
[junit4] 2> "type":"NRT",
[junit4] 2> "operation":"state"} current state version: 0
[junit4] 2> 190053 DEBUG (OverseerStateUpdate-73353407156125701-127.0.0.1:54807_solr-n_0000000000) [n:127.0.0.1:54807_solr ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
[junit4] 2> "core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n2",
[junit4] 2> "core_node_name":"core_node5",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"https://127.0.0.1:54807/solr",
[junit4] 2> "node_name":"127.0.0.1:54807_solr",
[junit4] 2> "numShards":"1",
[junit4] 2> "state":"down",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"AutoscalingHistoryHandlerTest_collection",
[junit4] 2> "type":"NRT",
[junit4] 2> "operation":"state"}
[junit4] 2> 190053 DEBUG (OverseerStateUpdate-73353407156125701-127.0.0.1:54807_solr-n_0000000000) [n:127.0.0.1:54807_solr ] o.a.s.c.o.ReplicaMutator Will update state for replica: core_node5:{"core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n2","base_url":"https://127.0.0.1:54807/solr","node_name":"127.0.0.1:54807_solr","state":"down","type":"NRT"}
[junit4] 2> 190053 DEBUG (OverseerStateUpdate-73353407156125701-127.0.0.1:54807_solr-n_0000000000) [n:127.0.0.1:54807_solr ] o.a.s.c.o.ReplicaMutator Collection is now: DocCollection(AutoscalingHistoryHandlerTest_collection//collections/AutoscalingHistoryHandlerTest_collection/state.json/1)={
[junit4] 2> "pullReplicas":"0",
[junit4] 2> "replicationFactor":"3",
[junit4] 2> "router":{"name":"compositeId"},
[junit4] 2> "maxShardsPerNode":"3",
[junit4] 2> "autoAddReplicas":"false",
[junit4] 2> "nrtReplicas":"3",
[junit4] 2> "tlogReplicas":"0",
[junit4] 2> "shards":{"shard1":{
[junit4] 2> "range":"80000000-7fffffff",
[junit4] 2> "state":"active",
[junit4] 2> "replicas":{
[junit4] 2> "core_node3":{
[junit4] 2> "core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n1",
[junit4] 2> "base_url":"https://127.0.0.1:54807/solr",
[junit4] 2> "node_name":"127.0.0.1:54807_solr",
[junit4] 2> "state":"down",
[junit4] 2> "type":"NRT"},
[junit4] 2> "core_node5":{
[junit4] 2> "core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n2",
[junit4] 2> "base_url":"https://127.0.0.1:54807/solr",
[junit4] 2> "node_name":"127.0.0.1:54807_solr",
[junit4] 2> "state":"down",
[junit4] 2> "type":"NRT"},
[junit4] 2> "core_node6":{
[junit4] 2> "core":"AutoscalingHistoryHandlerTest_collection_shard1_replica_n4",
[junit4] 2> "base_url":"https://127.0.0.1:54807/solr",
[junit4] 2> "node_name":"127.0.0.1:54807_solr",
[junit4] 2> "state":"down",
[junit4] 2> "type":"NRT"}}}}}
[junit4] 2> 190154 DEBUG (OverseerStateUpdate-73353407156125701-127.0.0.1:54807_solr-n_0000000000) [n:127.0.0.1:54807_solr ] o.a.s.c.o.ZkStateWriter going to update_collection /collections/AutoscalingHistoryHandlerTest_collection/state.json version: 1
[junit4] 2> 190155 INFO (zkCallback-147-thread-1-processing-n:127.0.0.1:54807_solr) [n:127.0.0.1:54807_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/AutoscalingHistoryHandlerTest_collection/state.json] for collection [AutoscalingHistoryHandlerTest_collection] has occurred - updating... (live nodes size: [2])
[junit4] 2> 190155 INFO (zkCallback-147-thread-2-processing-n:127.0.0.1:54807_solr) [n:127.0.0.1:54807_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/AutoscalingHistoryHandlerTest_collection/state.json] for collection [AutoscalingHistoryHandlerTest_collection] has occurred - updating... (live nodes size: [2])
[junit4] 2> 190407 DEBUG (ScheduledTrigger-230-thread-3) [n:127.0.0.1:54807_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 191042 INFO (qtp1097950914-791) [n:127.0.0.1:54807_solr c:AutoscalingHistoryHandlerTest_collection s:shard1 r:core_node3 x:AutoscalingHistoryHandlerTest_collection_shard1_replica_n1] o.a.s.c.RequestParams conf resource params.json loaded . version : 0
[junit4] 2> 191044 INFO (qtp1097950914-791) [n:127.0.0.1:54807_solr c:AutoscalingHistoryHandlerTest_collection s:shard1 r:core_node3 x:AutoscalingHistoryHandlerTest_collection_shard1_replica_n1] o.a.s.c.RequestParams request params refreshed to version 0
[junit4] 2> 191046 WARN (qtp1097950914-791) [n:127.0.0.1:54807_solr c:AutoscalingHistoryHandlerTest_collection s:shard1 r:core_node3 x:AutoscalingHistoryHandlerTest_collection_shard1_replica_n1] o.a.s.c.SolrConfig Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_2E3A421EDF6746B1-001/tempDir-001/node1/AutoscalingHistoryHandler
[...truncated too long message...]
at org.apache.solr.core.HdfsDirectoryFactory.create(HdfsDirectoryFactory.java:228)
[junit4] > at org.apache.solr.core.HdfsDirectoryFactoryTest.testInitArgsOrSysPropConfig(HdfsDirectoryFactoryTest.java:111)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] > Caused by: java.lang.OutOfMemoryError: Direct buffer memory
[junit4] > at java.nio.Bits.reserveMemory(Bits.java:694)
[junit4] > at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
[junit4] > at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
[junit4] > at org.apache.solr.store.blockcache.BlockCache.<init>(BlockCache.java:71)
[junit4] > at org.apache.solr.core.HdfsDirectoryFactory.createBlockCache(HdfsDirectoryFactory.java:310)
[junit4] > ... 42 more
[junit4] 2> 4777425 WARN (SUITE-HdfsDirectoryFactoryTest-seed#[2E3A421EDF6746B1]-worker) [ ] o.a.h.h.s.d.DirectoryScanner DirectoryScanner: shutdown has been called
[junit4] 2> 4777434 INFO (SUITE-HdfsDirectoryFactoryTest-seed#[2E3A421EDF6746B1]-worker) [ ] o.m.log Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0
[junit4] 2> 4777435 WARN (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.core.HdfsDirectoryFactoryTest_2E3A421EDF6746B1-001/tempDir-001/hdfsBaseDir/data/data3/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.core.HdfsDirectoryFactoryTest_2E3A421EDF6746B1-001/tempDir-001/hdfsBaseDir/data/data4/]] heartbeating to localhost/127.0.0.1:46895) [ ] o.a.h.h.s.d.IncrementalBlockReportManager IncrementalBlockReportManager interrupted
[junit4] 2> 4777435 WARN (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.core.HdfsDirectoryFactoryTest_2E3A421EDF6746B1-001/tempDir-001/hdfsBaseDir/data/data3/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.core.HdfsDirectoryFactoryTest_2E3A421EDF6746B1-001/tempDir-001/hdfsBaseDir/data/data4/]] heartbeating to localhost/127.0.0.1:46895) [ ] o.a.h.h.s.d.DataNode Ending block pool service for: Block pool BP-1977328832-10.41.0.5-1519525220033 (Datanode Uuid d9c29f59-d4f1-4ab2-b0e4-f3bcd6b9d9eb) service to localhost/127.0.0.1:46895
[junit4] 2> 4777437 WARN (SUITE-HdfsDirectoryFactoryTest-seed#[2E3A421EDF6746B1]-worker) [ ] o.a.h.h.s.d.DirectoryScanner DirectoryScanner: shutdown has been called
[junit4] 2> 4777446 INFO (SUITE-HdfsDirectoryFactoryTest-seed#[2E3A421EDF6746B1]-worker) [ ] o.m.log Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0
[junit4] 2> 4777548 WARN (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.core.HdfsDirectoryFactoryTest_2E3A421EDF6746B1-001/tempDir-001/hdfsBaseDir/data/data1/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.core.HdfsDirectoryFactoryTest_2E3A421EDF6746B1-001/tempDir-001/hdfsBaseDir/data/data2/]] heartbeating to localhost/127.0.0.1:46895) [ ] o.a.h.h.s.d.IncrementalBlockReportManager IncrementalBlockReportManager interrupted
[junit4] 2> 4777548 WARN (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.core.HdfsDirectoryFactoryTest_2E3A421EDF6746B1-001/tempDir-001/hdfsBaseDir/data/data1/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.core.HdfsDirectoryFactoryTest_2E3A421EDF6746B1-001/tempDir-001/hdfsBaseDir/data/data2/]] heartbeating to localhost/127.0.0.1:46895) [ ] o.a.h.h.s.d.DataNode Ending block pool service for: Block pool BP-1977328832-10.41.0.5-1519525220033 (Datanode Uuid bee333fa-434c-44cd-9472-c86d030bcdb7) service to localhost/127.0.0.1:46895
[junit4] 2> 4777561 INFO (SUITE-HdfsDirectoryFactoryTest-seed#[2E3A421EDF6746B1]-worker) [ ] o.m.log Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0
[junit4] 2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J2/temp/solr.core.HdfsDirectoryFactoryTest_2E3A421EDF6746B1-001
[junit4] 2> Feb 25, 2018 2:20:23 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4] 2> WARNING: Will linger awaiting termination of 34 leaked thread(s).
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene70), sim=RandomSimilarity(queryNorm=false): {}, locale=sq, timezone=Asia/Calcutta
[junit4] 2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=4,threads=2,free=352284984,total=523239424
[junit4] 2> NOTE: All tests run in this JVM: [TestFaceting, SolrGraphiteReporterTest, TestSegmentSorting, CleanupOldIndexTest, V2ApiIntegrationTest, ConvertedLegacyTest, StatelessScriptUpdateProcessorFactoryTest, TestCollectionsAPIViaSolrCloudCluster, DistributedSpellCheckComponentTest, NotRequiredUniqueKeyTest, LeaderFailureAfterFreshStartTest, TestCoreAdminApis, TestDynamicFieldResource, ReturnFieldsTest, TestMacros, CollectionsAPISolrJTest, TestCloudManagedSchema, TestCloudInspectUtil, DateRangeFieldTest, TestRandomDVFaceting, SpellPossibilityIteratorTest, BlockDirectoryTest, TestBulkSchemaAPI, ZkStateWriterTest, ConnectionManagerTest, TestAtomicUpdateErrorCases, TestSchemalessBufferedUpdates, OverseerStatusTest, TestLargeCluster, TestComplexPhraseQParserPlugin, MoreLikeThisHandlerTest, TestSolrConfigHandlerCloud, ClusterStateTest, TestAnalyzeInfixSuggestions, TestSuggestSpellingConverter, ExplicitHLLTest, CachingDirectoryFactoryTest, TestMultiValuedNumericRangeQuery, TestInitParams, TestExclusionRuleCollectionAccess, SpatialFilterTest, TestConfigSetImmutable, TestLazyCores, DistributedExpandComponentTest, TestDistributedMissingSort, TestClusterProperties, TestSolrXml, DistanceUnitsTest, SolrTestCaseJ4Test, TestJsonFacetRefinement, TestFreeTextSuggestions, TestManagedSchemaThreadSafety, DistributedFacetPivotWhiteBoxTest, ZkShardTermsTest, TestConfigSetsAPIZkFailure, TestEmbeddedSolrServerConstructors, TestUnifiedSolrHighlighter, TestSolr4Spatial2, TestLMJelinekMercerSimilarityFactory, DirectSolrSpellCheckerTest, RecoveryAfterSoftCommitTest, ClassificationUpdateProcessorTest, TestAddFieldRealTimeGet, TestBinaryResponseWriter, DeleteInactiveReplicaTest, RegexBoostProcessorTest, TestSerializedLuceneMatchVersion, AtomicUpdatesTest, HLLUtilTest, TestSubQueryTransformer, TermVectorComponentTest, SolrCmdDistributorTest, CircularListTest, AnalysisAfterCoreReloadTest, TestSolrCloudWithHadoopAuthPlugin, TestFastWriter, TestObjectReleaseTracker, TriggerIntegrationTest, RulesTest, TestCorePropertiesReload, TestLegacyFieldReuse, SolrMetricReporterTest, RegexBytesRefFilterTest, ExecutePlanActionTest, TestStressReorder, TestOverriddenPrefixQueryForCustomFieldType, TestHalfAndHalfDocValues, CurrencyRangeFacetCloudTest, TestTolerantSearch, TestRealTimeGet, ChaosMonkeyNothingIsSafeWithPullReplicasTest, TestScoreJoinQPNoScore, AutoAddReplicasPlanActionTest, DistributedSuggestComponentTest, TestJmxIntegration, TestSchemaResource, DocValuesTest, TestOnReconnectListenerSupport, DocValuesMultiTest, TestTolerantUpdateProcessorRandomCloud, TestRetrieveFieldsOptimizer, MetricTriggerTest, SolrXmlInZkTest, BooleanFieldTest, QueryResultKeyTest, TestDynamicLoading, TestBadConfig, IgnoreCommitOptimizeUpdateProcessorFactoryTest, AliasIntegrationTest, TestSolrCoreProperties, CollectionTooManyReplicasTest, TestFieldTypeCollectionResource, ResponseLogComponentTest, TestManagedSynonymFilterFactory, PreAnalyzedFieldTest, TestFiltering, DeleteReplicaTest, ManagedSchemaRoundRobinCloudTest, TestSha256AuthenticationProvider, TestSkipOverseerOperations, StressHdfsTest, ShardSplitTest, IndexSchemaRuntimeFieldTest, RollingRestartTest, CacheHeaderTest, TestManagedResource, TestReRankQParserPlugin, DateFieldTest, SpellingQueryConverterTest, RAMDirectoryFactoryTest, TestLRUCache, TestUtils, SliceStateTest, SystemInfoHandlerTest, UUIDFieldTest, TestRTGBase, TestCursorMarkWithoutUniqueKey, TestSimpleTrackingShardHandler, TestLuceneIndexBackCompat, TestEmbeddedSolrServerAdminHandler, TestJettySolrRunner, ConnectionReuseTest, AddReplicaTest, AssignBackwardCompatibilityTest, DeleteLastCustomShardedReplicaTest, DeleteShardTest, DistribCursorPagingTest, LIRRollingUpdatesTest, LeaderElectionContextKeyTest, LeaderInitiatedRecoveryOnShardRestartTest, LegacyCloudClusterPropTest, MigrateRouteKeyTest, MissingSegmentRecoveryTest, MoveReplicaHDFSFailoverTest, MultiThreadedOCPTest, NodeMutatorTest, OverriddenZkACLAndCredentialsProvidersTest, OverseerRolesTest, ReplaceNodeNoTargetTest, SaslZkACLProviderTest, SharedFSAutoReplicaFailoverTest, TestCloudPseudoReturnFields, TestCloudRecovery, TestCryptoKeys, TestDistribDocBasedVersion, TestLockTree, TestMiniSolrCloudClusterSSL, TestPullReplicaErrorHandling, TestRebalanceLeaders, TestShortCircuitedRequests, TestSolrCloudWithDelegationTokens, TestSolrCloudWithSecureImpersonation, TestStressLiveNodes, TestTolerantUpdateProcessorCloud, TestUtilizeNode, TriLevelCompositeIdRoutingTest, VMParamsZkACLAndCredentialsProvidersTest, ZkFailoverTest, HdfsCollectionsAPIDistributedZkTest, TestPolicyCloud, TestComputePlanAction, TestPolicyCloud, TestSimDistributedQueue, BaseCdcrDistributedZkTest, CdcrBootstrapTest, HdfsBasicDistributedZkTest, TestClusterStateMutator, RuleEngineTest, BlobRepositoryCloudTest, CoreSorterTest, DirectoryFactoryTest, ExitableDirectoryReaderTest, HdfsDirectoryFactoryTest]
[junit4] Completed [618/777 (3!)] on J2 in 13.18s, 3 tests, 1 error <<< FAILURES!
[...truncated 49006 lines...]
[JENKINS] Lucene-Solr-NightlyTests-7.x - Build # 157 - Failure
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.x/157/
4 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest
Error Message:
9 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest: 1) Thread[id=60442, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest] at java.lang.Thread.sleep(Native Method) at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.lang.Thread.run(Thread.java:748) 2) Thread[id=60447, name=zkCallback-4078-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 3) Thread[id=60445, name=zkConnectionManagerCallback-4079-thread-1, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 4) Thread[id=60986, name=zkCallback-4078-thread-4, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 5) Thread[id=60443, name=TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]-SendThread(127.0.0.1:35305), state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest] at java.lang.Thread.sleep(Native Method) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1054) 6) Thread[id=60954, name=zkCallback-4078-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 7) Thread[id=60955, name=zkCallback-4078-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 8) Thread[id=60444, name=TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]-EventThread, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502) 9) Thread[id=61054, name=zkCallback-4078-thread-5, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 9 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest:
1) Thread[id=60442, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.lang.Thread.run(Thread.java:748)
2) Thread[id=60447, name=zkCallback-4078-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
3) Thread[id=60445, name=zkConnectionManagerCallback-4079-thread-1, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
4) Thread[id=60986, name=zkCallback-4078-thread-4, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
5) Thread[id=60443, name=TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]-SendThread(127.0.0.1:35305), state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1054)
6) Thread[id=60954, name=zkCallback-4078-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
7) Thread[id=60955, name=zkCallback-4078-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
8) Thread[id=60444, name=TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]-EventThread, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
9) Thread[id=61054, name=zkCallback-4078-thread-5, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
at __randomizedtesting.SeedInfo.seed([2C07BB5E6A2C13C1]:0)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest
Error Message:
There are still zombie threads that couldn't be terminated: 1) Thread[id=60447, name=zkCallback-4078-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2) Thread[id=60445, name=zkConnectionManagerCallback-4079-thread-1, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 3) Thread[id=60986, name=zkCallback-4078-thread-4, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 4) Thread[id=61054, name=zkCallback-4078-thread-5, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 5) Thread[id=60443, name=TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]-SendThread(127.0.0.1:35305), state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest] at java.lang.Thread.sleep(Native Method) at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063) 6) Thread[id=60954, name=zkCallback-4078-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 7) Thread[id=60955, name=zkCallback-4078-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
1) Thread[id=60447, name=zkCallback-4078-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2) Thread[id=60445, name=zkConnectionManagerCallback-4079-thread-1, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
3) Thread[id=60986, name=zkCallback-4078-thread-4, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
4) Thread[id=61054, name=zkCallback-4078-thread-5, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
5) Thread[id=60443, name=TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]-SendThread(127.0.0.1:35305), state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063)
6) Thread[id=60954, name=zkCallback-4078-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
7) Thread[id=60955, name=zkCallback-4078-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
at __randomizedtesting.SeedInfo.seed([2C07BB5E6A2C13C1]:0)
FAILED: junit.framework.TestSuite.org.apache.solr.search.TestRandomCollapseQParserPlugin
Error Message:
Stack Trace:
java.lang.AssertionError
at __randomizedtesting.SeedInfo.seed([2C07BB5E6A2C13C1]:0)
at org.apache.lucene.util.TestRuleRestoreSystemProperties.before(TestRuleRestoreSystemProperties.java:52)
at com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.before(TestRuleAdapter.java:26)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:35)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.test
Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:37182/k_k/collection1
Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:37182/k_k/collection1
at __randomizedtesting.SeedInfo.seed([2C07BB5E6A2C13C1:A4538484C4D07E39]:0)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1104)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:484)
at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:463)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.commit(AbstractFullDistribZkTestBase.java:1591)
at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.test(ChaosMonkeyNothingIsSafeTest.java:212)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
... 53 more
Build Log:
[...truncated 13158 lines...]
[junit4] Suite: org.apache.solr.uninverting.TestDocTermOrds
[junit4] 2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
[junit4] 2> NOTE: test params are: codec=FastDecompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST_DECOMPRESSION, chunkSize=20328, maxDocsPerChunk=5, blockSize=52), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST_DECOMPRESSION, chunkSize=20328, blockSize=52)), sim=RandomSimilarity(queryNorm=true): {field=DFR I(n)Z(0.3)}, locale=es-EC, timezone=US/Alaska
[junit4] 2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=4,threads=1,free=422872512,total=472907776
[junit4] 2> NOTE: All tests run in this JVM: [SimpleFacetsTest, TestDistributedMap, DeleteShardTest, OpenExchangeRatesOrgProviderTest, CursorPagingTest, ShufflingReplicaListTransformerTest, SuggesterWFSTTest, SolrMetricsIntegrationTest, DebugComponentTest, ChaosMonkeySafeLeaderWithPullReplicasTest, TestSort, ClassificationUpdateProcessorFactoryTest, BadIndexSchemaTest, UUIDFieldTest, TestRecoveryHdfs, ActionThrottleTest, TestFieldCollectionResource, TermVectorComponentDistributedTest, TestGraphTermsQParserPlugin, SpellCheckCollatorTest, TestCryptoKeys, TestConfigSetProperties, TestSQLHandler, TestLegacyNumericRangeQueryBuilder, BlobRepositoryCloudTest, CollectionsAPIAsyncDistributedZkTest, TestCSVResponseWriter, TestNRTOpen, ShardRoutingTest, TestCrossCoreJoin, CheckHdfsIndexTest, HighlighterConfigTest, TestSolrQueryResponse, DocumentAnalysisRequestHandlerTest, JSONWriterTest, DistributedFacetPivotSmallAdvancedTest, TestPointFields, SimpleMLTQParserTest, HdfsChaosMonkeyNothingIsSafeTest, ImplicitSnitchTest, TestSubQueryTransformerCrossCore, TestCustomSort, TestStressLucene, DeleteLastCustomShardedReplicaTest, ExitableDirectoryReaderTest, TestIBSimilarityFactory, TestFieldCacheSort, SolrPluginUtilsTest, TestCollationField, CoreAdminRequestStatusTest, TestUniqueKeyFieldResource, TestNumericTerms64, HdfsRecoverLeaseTest, TestRTimerTree, ConcurrentCreateRoutedAliasTest, TestRandomFaceting, TestZkAclsWithHadoopAuth, TestConfig, TestFastOutputStream, TestIndexSearcher, ParsingFieldUpdateProcessorsTest, TestTlogReplica, TestSolrQueryParser, FacetPivotSmallTest, TestSortableTextField, TestMergePolicyConfig, SolrIndexSplitterTest, OpenCloseCoreStressTest, TestDocSet, TestImplicitCoreProperties, CloneFieldUpdateProcessorFactoryTest, TestPHPSerializedResponseWriter, HDFSCollectionsAPITest, TestRequestForwarding, SolrJmxReporterCloudTest, TestExecutePlanAction, PluginInfoTest, SolrCloudReportersTest, OverseerModifyCollectionTest, TestCloudJSONFacetJoinDomain, TestSolrCloudSnapshots, TestConfigSetsAPI, ZkNodePropsTest, TestDocTermOrds]
[junit4] ERROR 0.00s J0 | TestDocTermOrds (suite) <<<
[junit4] > Throwable #1: java.lang.OutOfMemoryError: GC overhead limit exceededThrowable #2: java.lang.OutOfMemoryError: GC overhead limit exceededThrowable #3: java.lang.OutOfMemoryError: GC overhead limit exceeded
[junit4] Completed [201/777 (1!)] on J0 in 41.02s, 7 tests, 3 errors <<< FAILURES!
[...truncated 2 lines...]
[junit4] Suite: org.apache.solr.search.TestRandomCollapseQParserPlugin
[junit4] 2> NOTE: test params are: codec=FastDecompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST_DECOMPRESSION, chunkSize=20328, maxDocsPerChunk=5, blockSize=52), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST_DECOMPRESSION, chunkSize=20328, blockSize=52)), sim=RandomSimilarity(queryNorm=true): {field=DFR I(n)Z(0.3)}, locale=es-EC, timezone=US/Alaska
[junit4] 2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=4,threads=1,free=401542264,total=472907776
[junit4] 2> NOTE: All tests run in this JVM: [SimpleFacetsTest, TestDistributedMap, DeleteShardTest, OpenExchangeRatesOrgProviderTest, CursorPagingTest, ShufflingReplicaListTransformerTest, SuggesterWFSTTest, SolrMetricsIntegrationTest, DebugComponentTest, ChaosMonkeySafeLeaderWithPullReplicasTest, TestSort, ClassificationUpdateProcessorFactoryTest, BadIndexSchemaTest, UUIDFieldTest, TestRecoveryHdfs, ActionThrottleTest, TestFieldCollectionResource, TermVectorComponentDistributedTest, TestGraphTermsQParserPlugin, SpellCheckCollatorTest, TestCryptoKeys, TestConfigSetProperties, TestSQLHandler, TestLegacyNumericRangeQueryBuilder, BlobRepositoryCloudTest, CollectionsAPIAsyncDistributedZkTest, TestCSVResponseWriter, TestNRTOpen, ShardRoutingTest, TestCrossCoreJoin, CheckHdfsIndexTest, HighlighterConfigTest, TestSolrQueryResponse, DocumentAnalysisRequestHandlerTest, JSONWriterTest, DistributedFacetPivotSmallAdvancedTest, TestPointFields, SimpleMLTQParserTest, HdfsChaosMonkeyNothingIsSafeTest, ImplicitSnitchTest, TestSubQueryTransformerCrossCore, TestCustomSort, TestStressLucene, DeleteLastCustomShardedReplicaTest, ExitableDirectoryReaderTest, TestIBSimilarityFactory, TestFieldCacheSort, SolrPluginUtilsTest, TestCollationField, CoreAdminRequestStatusTest, TestUniqueKeyFieldResource, TestNumericTerms64, HdfsRecoverLeaseTest, TestRTimerTree, ConcurrentCreateRoutedAliasTest, TestRandomFaceting, TestZkAclsWithHadoopAuth, TestConfig, TestFastOutputStream, TestIndexSearcher, ParsingFieldUpdateProcessorsTest, TestTlogReplica, TestSolrQueryParser, FacetPivotSmallTest, TestSortableTextField, TestMergePolicyConfig, SolrIndexSplitterTest, OpenCloseCoreStressTest, TestDocSet, TestImplicitCoreProperties, CloneFieldUpdateProcessorFactoryTest, TestPHPSerializedResponseWriter, HDFSCollectionsAPITest, TestRequestForwarding, SolrJmxReporterCloudTest, TestExecutePlanAction, PluginInfoTest, SolrCloudReportersTest, OverseerModifyCollectionTest, TestCloudJSONFacetJoinDomain, TestSolrCloudSnapshots, TestConfigSetsAPI, ZkNodePropsTest, TestDocTermOrds, TestRandomCollapseQParserPlugin]
[junit4] 2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestRandomCollapseQParserPlugin -Dtests.seed=2C07BB5E6A2C13C1 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/test-data/enwiki.random.lines.txt -Dtests.locale=es-EC -Dtests.timezone=US/Alaska -Dtests.asserts=true -Dtests.file.encoding=UTF-8
[junit4] ERROR 0.00s J0 | TestRandomCollapseQParserPlugin (suite) <<<
[junit4] > Throwable #1: java.lang.AssertionError
[junit4] > at __randomizedtesting.SeedInfo.seed([2C07BB5E6A2C13C1]:0)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] Completed [202/777 (2!)] on J0 in 0.01s, 0 tests, 1 failure <<< FAILURES!
[...truncated 782 lines...]
[junit4] Suite: org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest
[junit4] 2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest_2C07BB5E6A2C13C1-001/init-core-data-001
[junit4] 2> 2976619 INFO (SUITE-ChaosMonkeyNothingIsSafeTest-seed#[2C07BB5E6A2C13C1]-worker) [ ] o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false) w/NUMERIC_DOCVALUES_SYSPROP=true
[junit4] 2> 2976620 INFO (SUITE-ChaosMonkeyNothingIsSafeTest-seed#[2C07BB5E6A2C13C1]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776)
[junit4] 2> 2976620 INFO (SUITE-ChaosMonkeyNothingIsSafeTest-seed#[2C07BB5E6A2C13C1]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
[junit4] 2> 2976620 INFO (SUITE-ChaosMonkeyNothingIsSafeTest-seed#[2C07BB5E6A2C13C1]-worker) [ ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /k_k/
[junit4] 2> 2976623 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 2976624 INFO (Thread-45255) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 2976624 INFO (Thread-45255) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 2976626 ERROR (Thread-45255) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 2976724 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.a.s.c.ZkTestServer start zk server on port:35305
[junit4] 2> 2976728 INFO (zkConnectionManagerCallback-4059-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 2976731 INFO (zkConnectionManagerCallback-4061-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 2976736 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 2976737 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
[junit4] 2> 2976739 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 2976740 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 2976741 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 2976742 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 2976743 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 2976744 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 2976746 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 2976747 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 2976748 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 2976749 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.a.s.c.AbstractFullDistribZkTestBase Will use TLOG replicas unless explicitly asked otherwise
[junit4] 2> 2976850 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T23:27:37+02:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
[junit4] 2> 2976851 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 2976851 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 2976851 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.e.j.s.session Scavenging every 660000ms
[junit4] 2> 2976851 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4503b13{/k_k,null,AVAILABLE}
[junit4] 2> 2976853 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@55dbc1d5{HTTP/1.1,[http/1.1]}{127.0.0.1:59147}
[junit4] 2> 2976853 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.e.j.s.Server Started @2979179ms
[junit4] 2> 2976853 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest_2C07BB5E6A2C13C1-001/tempDir-001/control/data, hostContext=/k_k, hostPort=59147, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest_2C07BB5E6A2C13C1-001/control-001/cores}
[junit4] 2> 2976853 ERROR (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 2976853 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 7.3.0
[junit4] 2> 2976853 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 2976853 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 2976853 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-02-24T01:47:00.040Z
[junit4] 2> 2976857 INFO (zkConnectionManagerCallback-4063-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 2976858 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 2976858 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest_2C07BB5E6A2C13C1-001/control-001/solr.xml
[junit4] 2> 2976861 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
[junit4] 2> 2976861 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
[junit4] 2> 2976863 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@2ad5bf25, but no JMX reporters were configured - adding default JMX reporter.
[junit4] 2> 2976872 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:35305/solr
[junit4] 2> 2976873 INFO (zkConnectionManagerCallback-4067-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 2976878 INFO (zkConnectionManagerCallback-4069-thread-1-processing-n:127.0.0.1:59147_k_k) [n:127.0.0.1:59147_k_k ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 2976941 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [n:127.0.0.1:59147_k_k ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 2976941 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [n:127.0.0.1:59147_k_k ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:59147_k_k
[junit4] 2> 2976942 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [n:127.0.0.1:59147_k_k ] o.a.s.c.Overseer Overseer (id=73347914452631556-127.0.0.1:59147_k_k-n_0000000000) starting
[junit4] 2> 2976949 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [n:127.0.0.1:59147_k_k ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59147_k_k
[junit4] 2> 2976950 INFO (OverseerStateUpdate-73347914452631556-127.0.0.1:59147_k_k-n_0000000000) [n:127.0.0.1:59147_k_k ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 2976999 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [n:127.0.0.1:59147_k_k ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2ad5bf25
[junit4] 2> 2977007 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [n:127.0.0.1:59147_k_k ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2ad5bf25
[junit4] 2> 2977007 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [n:127.0.0.1:59147_k_k ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2ad5bf25
[junit4] 2> 2977008 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [n:127.0.0.1:59147_k_k ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest_2C07BB5E6A2C13C1-001/control-001/cores
[junit4] 2> 2977026 INFO (zkConnectionManagerCallback-4075-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 2977026 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 2977027 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:35305/solr ready
[junit4] 2> 2977030 INFO (qtp1573230012-60400) [n:127.0.0.1:59147_k_k ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:59147_k_k&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 2977034 INFO (OverseerThreadFactory-9364-thread-1-processing-n:127.0.0.1:59147_k_k) [n:127.0.0.1:59147_k_k ] o.a.s.c.a.c.CreateCollectionCmd Create collection control_collection
[junit4] 2> 2977141 INFO (qtp1573230012-60404) [n:127.0.0.1:59147_k_k ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 2977142 INFO (qtp1573230012-60404) [n:127.0.0.1:59147_k_k ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
[junit4] 2> 2977245 INFO (zkCallback-4068-thread-1-processing-n:127.0.0.1:59147_k_k) [n:127.0.0.1:59147_k_k ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
[junit4] 2> 2978165 INFO (qtp1573230012-60404) [n:127.0.0.1:59147_k_k c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
[junit4] 2> 2978186 INFO (qtp1573230012-60404) [n:127.0.0.1:59147_k_k c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
[junit4] 2> 2978263 INFO (qtp1573230012-60404) [n:127.0.0.1:59147_k_k c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
[junit4] 2> 2978276 INFO (qtp1573230012-60404) [n:127.0.0.1:59147_k_k c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
[junit4] 2> 2978277 INFO (qtp1573230012-60404) [n:127.0.0.1:59147_k_k c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2ad5bf25
[junit4] 2> 2978277 INFO (qtp1573230012-60404) [n:127.0.0.1:59147_k_k c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 2978277 INFO (qtp1573230012-60404) [n:127.0.0.1:59147_k_k c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest_2C07BB5E6A2C13C1-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest_2C07BB5E6A2C13C1-001/control-001/cores/control_collection_shard1_replica_n1/data/]
[junit4] 2> 2978280 INFO (qtp1573230012-60404) [n:127.0.0.1:59147_k_k c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1104712507, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
[junit4] 2> 2978289 WARN (qtp1573230012-60404) [n:127.0.0.1:59147_k_k c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 2978333 INFO (qtp1573230012-60404) [n:127.0.0.1:59147_k_k c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 2978333 INFO (qtp1573230012-60404) [n:127.0.0.1:59147_k_k c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 2978334 INFO (qtp1573230012-60404) [n:127.0.0.1:59147_k_k c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: if uncommited for 15000ms;
[junit4] 2> 2978334 INFO (qtp1573230012-60404) [n:127.0.0.1:59147_k_k c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 2978335 INFO (qtp1573230012-60404) [n:127.0.0.1:59147_k_k c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1252772171, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
[junit4] 2> 2978336 INFO (qtp1573230012-60404) [n:127.0.0.1:59147_k_k c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@695847a4[control_collection_shard1_replica_n1] main]
[junit4] 2> 2978337 INFO (qtp1573230012-60404) [n:127.0.0.1:59147_k_k c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 2978337 INFO (qtp1573230012-60404) [n:127.0.0.1:59147_k_k c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 2978338 INFO (qtp1573230012-60404) [n:127.0.0.1:59147_k_k c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 2978339 INFO (searcherExecutor-9367-thread-1-processing-n:127.0.0.1:59147_k_k x:control_collection_shard1_replica_n1 s:shard1 c:control_collection) [n:127.0.0.1:59147_k_k c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@695847a4[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 2978339 INFO (qtp1573230012-60404) [n:127.0.0.1:59147_k_k c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1593244984568446976
[junit4] 2> 2978345 INFO (qtp1573230012-60404) [n:127.0.0.1:59147_k_k c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 2978345 INFO (qtp1573230012-60404) [n:127.0.0.1:59147_k_k c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 2978345 INFO (qtp1573230012-60404) [n:127.0.0.1:59147_k_k c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:59147/k_k/control_collection_shard1_replica_n1/
[junit4] 2> 2978345 INFO (qtp1573230012-60404) [n:127.0.0.1:59147_k_k c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 2978345 INFO (qtp1573230012-60404) [n:127.0.0.1:59147_k_k c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:59147/k_k/control_collection_shard1_replica_n1/ has no replicas
[junit4] 2> 2978345 INFO (qtp1573230012-60404) [n:127.0.0.1:59147_k_k c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 2978347 INFO (qtp1573230012-60404) [n:127.0.0.1:59147_k_k c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:59147/k_k/control_collection_shard1_replica_n1/ shard1
[junit4] 2> 2978448 INFO (zkCallback-4068-thread-1-processing-n:127.0.0.1:59147_k_k) [n:127.0.0.1:59147_k_k ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
[junit4] 2> 2978498 INFO (qtp1573230012-60404) [n:127.0.0.1:59147_k_k c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 2978500 INFO (qtp1573230012-60404) [n:127.0.0.1:59147_k_k c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1358
[junit4] 2> 2978506 INFO (qtp1573230012-60400) [n:127.0.0.1:59147_k_k ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
[junit4] 2> 2978600 INFO (zkCallback-4068-thread-2-processing-n:127.0.0.1:59147_k_k) [n:127.0.0.1:59147_k_k ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
[junit4] 2> 2979036 INFO (OverseerCollectionConfigSetProcessor-73347914452631556-127.0.0.1:59147_k_k-n_0000000000) [n:127.0.0.1:59147_k_k ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 2979506 INFO (qtp1573230012-60400) [n:127.0.0.1:59147_k_k ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:59147_k_k&wt=javabin&version=2} status=0 QTime=2476
[junit4] 2> 2979510 INFO (zkConnectionManagerCallback-4079-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 2979511 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 2979512 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:35305/solr ready
[junit4] 2> 2979512 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 2979512 INFO (qtp1573230012-60404) [n:127.0.0.1:59147_k_k ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=1&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 2979514 INFO (OverseerThreadFactory-9364-thread-2-processing-n:127.0.0.1:59147_k_k) [n:127.0.0.1:59147_k_k ] o.a.s.c.a.c.CreateCollectionCmd Create collection collection1
[junit4] 2> 2979515 WARN (OverseerThreadFactory-9364-thread-2-processing-n:127.0.0.1:59147_k_k) [n:127.0.0.1:59147_k_k ] o.a.s.c.a.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
[junit4] 2> 2979718 INFO (qtp1573230012-60404) [n:127.0.0.1:59147_k_k ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
[junit4] 2> 2979718 INFO (qtp1573230012-60404) [n:127.0.0.1:59147_k_k ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=1&wt=javabin&version=2} status=0 QTime=205
[junit4] 2> 2979808 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest_2C07BB5E6A2C13C1-001/shard-1-001 of type TLOG
[junit4] 2> 2979808 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T23:27:37+02:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
[junit4] 2> 2979809 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 2979809 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 2979809 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.e.j.s.session Scavenging every 660000ms
[junit4] 2> 2979810 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@dc23020{/k_k,null,AVAILABLE}
[junit4] 2> 2979810 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@22a36083{HTTP/1.1,[http/1.1]}{127.0.0.1:43836}
[junit4] 2> 2979810 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.e.j.s.Server Started @2982136ms
[junit4] 2> 2979810 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest_2C07BB5E6A2C13C1-001/tempDir-001/jetty1, replicaType=TLOG, solrconfig=solrconfig.xml, hostContext=/k_k, hostPort=43836, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest_2C07BB5E6A2C13C1-001/shard-1-001/cores}
[junit4] 2> 2979810 ERROR (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 2979810 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 7.3.0
[junit4] 2> 2979811 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 2979811 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 2979811 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-02-24T01:47:02.998Z
[junit4] 2> 2979812 INFO (zkConnectionManagerCallback-4081-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 2979813 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 2979813 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest_2C07BB5E6A2C13C1-001/shard-1-001/solr.xml
[junit4] 2> 2979817 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
[junit4] 2> 2979817 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
[junit4] 2> 2979818 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@2ad5bf25, but no JMX reporters were configured - adding default JMX reporter.
[junit4] 2> 2979820 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:35305/solr
[junit4] 2> 2979821 INFO (zkConnectionManagerCallback-4085-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 2979823 INFO (zkConnectionManagerCallback-4087-thread-1-processing-n:127.0.0.1:43836_k_k) [n:127.0.0.1:43836_k_k ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 2979830 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [n:127.0.0.1:43836_k_k ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 2979833 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [n:127.0.0.1:43836_k_k ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 2979835 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [n:127.0.0.1:43836_k_k ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43836_k_k
[junit4] 2> 2979835 INFO (zkCallback-4068-thread-2-processing-n:127.0.0.1:59147_k_k) [n:127.0.0.1:59147_k_k ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 2979841 INFO (zkCallback-4086-thread-1-processing-n:127.0.0.1:43836_k_k) [n:127.0.0.1:43836_k_k ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 2979841 INFO (zkCallback-4078-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 2979973 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [n:127.0.0.1:43836_k_k ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2ad5bf25
[junit4] 2> 2979982 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [n:127.0.0.1:43836_k_k ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2ad5bf25
[junit4] 2> 2979982 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [n:127.0.0.1:43836_k_k ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2ad5bf25
[junit4] 2> 2979984 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [n:127.0.0.1:43836_k_k ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest_2C07BB5E6A2C13C1-001/shard-1-001/cores
[junit4] 2> 2980009 INFO (qtp847376362-60449) [n:127.0.0.1:43836_k_k ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:43836_k_k&action=ADDREPLICA&collection=collection1&shard=shard2&type=TLOG&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 2980011 INFO (OverseerCollectionConfigSetProcessor-73347914452631556-127.0.0.1:59147_k_k-n_0000000000) [n:127.0.0.1:59147_k_k ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 2980012 INFO (OverseerThreadFactory-9364-thread-3-processing-n:127.0.0.1:59147_k_k) [n:127.0.0.1:59147_k_k ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:43836_k_k for creating new replica
[junit4] 2> 2980014 INFO (qtp847376362-60453) [n:127.0.0.1:43836_k_k ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_t41&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=TLOG
[junit4] 2> 2980014 INFO (qtp847376362-60453) [n:127.0.0.1:43836_k_k ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
[junit4] 2> 2981038 INFO (qtp847376362-60453) [n:127.0.0.1:43836_k_k c:collection1 s:shard2 x:collection1_shard2_replica_t41] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
[junit4] 2> 2981053 INFO (qtp847376362-60453) [n:127.0.0.1:43836_k_k c:collection1 s:shard2 x:collection1_shard2_replica_t41] o.a.s.s.IndexSchema [collection1_shard2_replica_t41] Schema name=test
[junit4] 2> 2981143 INFO (qtp847376362-60453) [n:127.0.0.1:43836_k_k c:collection1 s:shard2 x:collection1_shard2_replica_t41] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
[junit4] 2> 2981154 INFO (qtp847376362-60453) [n:127.0.0.1:43836_k_k c:collection1 s:shard2 x:collection1_shard2_replica_t41] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_t41' using configuration from collection collection1, trusted=true
[junit4] 2> 2981155 INFO (qtp847376362-60453) [n:127.0.0.1:43836_k_k c:collection1 s:shard2 x:collection1_shard2_replica_t41] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard2.replica_t41' (registry 'solr.core.collection1.shard2.replica_t41') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2ad5bf25
[junit4] 2> 2981155 INFO (qtp847376362-60453) [n:127.0.0.1:43836_k_k c:collection1 s:shard2 x:collection1_shard2_replica_t41] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 2981155 INFO (qtp847376362-60453) [n:127.0.0.1:43836_k_k c:collection1 s:shard2 x:collection1_shard2_replica_t41] o.a.s.c.SolrCore [[collection1_shard2_replica_t41] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest_2C07BB5E6A2C13C1-001/shard-1-001/cores/collection1_shard2_replica_t41], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest_2C07BB5E6A2C13C1-001/shard-1-001/cores/collection1_shard2_replica_t41/data/]
[junit4] 2> 2981159 INFO (qtp847376362-60453) [n:127.0.0.1:43836_k_k c:collection1 s:shard2 x:collection1_shard2_replica_t41] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1104712507, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
[junit4] 2> 2981168 WARN (qtp847376362-60453) [n:127.0.0.1:43836_k_k c:collection1 s:shard2 x:collection1_shard2_replica_t41] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 2981205 INFO (qtp847376362-60453) [n:127.0.0.1:43836_k_k c:collection1 s:shard2 x:collection1_shard2_replica_t41] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 2981205 INFO (qtp847376362-60453) [n:127.0.0.1:43836_k_k c:collection1 s:shard2 x:collection1_shard2_replica_t41] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 2981206 INFO (qtp847376362-60453) [n:127.0.0.1:43836_k_k c:collection1 s:shard2 x:collection1_shard2_replica_t41] o.a.s.u.CommitTracker Hard AutoCommit: if uncommited for 15000ms;
[junit4] 2> 2981206 INFO (qtp847376362-60453) [n:127.0.0.1:43836_k_k c:collection1 s:shard2 x:collection1_shard2_replica_t41] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 2981208 INFO (qtp847376362-60453) [n:127.0.0.1:43836_k_k c:collection1 s:shard2 x:collection1_shard2_replica_t41] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1252772171, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
[junit4] 2> 2981208 INFO (qtp847376362-60453) [n:127.0.0.1:43836_k_k c:collection1 s:shard2 x:collection1_shard2_replica_t41] o.a.s.s.SolrIndexSearcher Opening [Searcher@448620ba[collection1_shard2_replica_t41] main]
[junit4] 2> 2981209 INFO (qtp847376362-60453) [n:127.0.0.1:43836_k_k c:collection1 s:shard2 x:collection1_shard2_replica_t41] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 2981209 INFO (qtp847376362-60453) [n:127.0.0.1:43836_k_k c:collection1 s:shard2 x:collection1_shard2_replica_t41] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 2981210 INFO (qtp847376362-60453) [n:127.0.0.1:43836_k_k c:collection1 s:shard2 x:collection1_shard2_replica_t41] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 2981210 INFO (qtp847376362-60453) [n:127.0.0.1:43836_k_k c:collection1 s:shard2 x:collection1_shard2_replica_t41] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1593244987578908672
[junit4] 2> 2981211 INFO (searcherExecutor-9378-thread-1-processing-n:127.0.0.1:43836_k_k x:collection1_shard2_replica_t41 s:shard2 c:collection1) [n:127.0.0.1:43836_k_k c:collection1 s:shard2 x:collection1_shard2_replica_t41] o.a.s.c.SolrCore [collection1_shard2_replica_t41] Registered new searcher Searcher@448620ba[collection1_shard2_replica_t41] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 2981216 INFO (qtp847376362-60453) [n:127.0.0.1:43836_k_k c:collection1 s:shard2 x:collection1_shard2_replica_t41] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 2981216 INFO (qtp847376362-60453) [n:127.0.0.1:43836_k_k c:collection1 s:shard2 x:collection1_shard2_replica_t41] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 2981216 INFO (qtp847376362-60453) [n:127.0.0.1:43836_k_k c:collection1 s:shard2 x:collection1_shard2_replica_t41] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:43836/k_k/collection1_shard2_replica_t41/
[junit4] 2> 2981216 INFO (qtp847376362-60453) [n:127.0.0.1:43836_k_k c:collection1 s:shard2 x:collection1_shard2_replica_t41] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 2981216 INFO (qtp847376362-60453) [n:127.0.0.1:43836_k_k c:collection1 s:shard2 x:collection1_shard2_replica_t41] o.a.s.c.SyncStrategy http://127.0.0.1:43836/k_k/collection1_shard2_replica_t41/ has no replicas
[junit4] 2> 2981216 INFO (qtp847376362-60453) [n:127.0.0.1:43836_k_k c:collection1 s:shard2 x:collection1_shard2_replica_t41] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 2981216 INFO (qtp847376362-60453) [n:127.0.0.1:43836_k_k c:collection1 s:shard2 x:collection1_shard2_replica_t41] o.a.s.c.ZkController collection1_shard2_replica_t41 stopping background replication from leader
[junit4] 2> 2981219 INFO (qtp847376362-60453) [n:127.0.0.1:43836_k_k c:collection1 s:shard2 x:collection1_shard2_replica_t41] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:43836/k_k/collection1_shard2_replica_t41/ shard2
[junit4] 2> 2981370 INFO (qtp847376362-60453) [n:127.0.0.1:43836_k_k c:collection1 s:shard2 x:collection1_shard2_replica_t41] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 2981371 INFO (qtp847376362-60453) [n:127.0.0.1:43836_k_k c:collection1 s:shard2 x:collection1_shard2_replica_t41] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_t41&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=TLOG} status=0 QTime=1357
[junit4] 2> 2981373 INFO (qtp847376362-60449) [n:127.0.0.1:43836_k_k ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:43836_k_k&action=ADDREPLICA&collection=collection1&shard=shard2&type=TLOG&wt=javabin&version=2} status=0 QTime=1363
[junit4] 2> 2981463 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest_2C07BB5E6A2C13C1-001/shard-2-001 of type TLOG
[junit4] 2> 2981464 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T23:27:37+02:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
[junit4] 2> 2981465 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 2981465 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 2981465 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.e.j.s.session Scavenging every 660000ms
[junit4] 2> 2981465 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@64d993df{/k_k,null,AVAILABLE}
[junit4] 2> 2981466 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@6fd8c71c{HTTP/1.1,[http/1.1]}{127.0.0.1:34756}
[junit4] 2> 2981466 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.e.j.s.Server Started @2983792ms
[junit4] 2> 2981466 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest_2C07BB5E6A2C13C1-001/tempDir-001/jetty2, replicaType=TLOG, solrconfig=solrconfig.xml, hostContext=/k_k, hostPort=34756, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest_2C07BB5E6A2C13C1-001/shard-2-001/cores}
[junit4] 2> 2981466 ERROR (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 2981466 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 7.3.0
[junit4] 2> 2981466 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 2981466 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 2981466 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-02-24T01:47:04.653Z
[junit4] 2> 2981468 INFO (zkConnectionManagerCallback-4091-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 2981469 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 2981469 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest_2C07BB5E6A2C13C1-001/shard-2-001/solr.xml
[junit4] 2> 2981472 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
[junit4] 2> 2981472 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
[junit4] 2> 2981474 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@2ad5bf25, but no JMX reporters were configured - adding default JMX reporter.
[junit4] 2> 2981476 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:35305/solr
[junit4] 2> 2981477 INFO (zkConnectionManagerCallback-4095-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 2981478 INFO (zkConnectionManagerCallback-4097-thread-1-processing-n:127.0.0.1:34756_k_k) [n:127.0.0.1:34756_k_k ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 2981483 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [n:127.0.0.1:34756_k_k ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 2981484 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [n:127.0.0.1:34756_k_k ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 2981486 INFO (TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]) [n:127.0.0.1:34756_k_k ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:3
[...truncated too long message...]
nit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
[junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
[junit4] > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] > at __randomizedtesting.SeedInfo.seed([2C07BB5E6A2C13C1]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
[junit4] > 1) Thread[id=60447, name=zkCallback-4078-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
[junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
[junit4] > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] > 2) Thread[id=60445, name=zkConnectionManagerCallback-4079-thread-1, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
[junit4] > at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] > 3) Thread[id=60986, name=zkCallback-4078-thread-4, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
[junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
[junit4] > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] > 4) Thread[id=61054, name=zkCallback-4078-thread-5, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
[junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
[junit4] > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] > 5) Thread[id=60443, name=TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[2C07BB5E6A2C13C1]-SendThread(127.0.0.1:35305), state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
[junit4] > at java.lang.Thread.sleep(Native Method)
[junit4] > at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105)
[junit4] > at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000)
[junit4] > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063)
[junit4] > 6) Thread[id=60954, name=zkCallback-4078-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
[junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
[junit4] > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] > 7) Thread[id=60955, name=zkCallback-4078-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
[junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
[junit4] > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] > at __randomizedtesting.SeedInfo.seed([2C07BB5E6A2C13C1]:0)
[junit4] Completed [432/777 (3!)] on J1 in 94.59s, 1 test, 3 errors <<< FAILURES!
[...truncated 943 lines...]
[junit4] JVM J0: stdout was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/temp/junit4-J0-20180224_005720_8124072268164021975440.sysout
[junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
[junit4] java.lang.OutOfMemoryError: Java heap space
[junit4] Dumping heap to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/heapdumps/java_pid8633.hprof ...
[junit4] Heap dump file created [452732546 bytes in 0.850 secs]
[junit4] <<< JVM J0: EOF ----
[junit4] JVM J0: stderr was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/temp/junit4-J0-20180224_005720_8128178310285512366921.syserr
[junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
[junit4] WARN: Event serializer exception. -> java.lang.OutOfMemoryError: GC overhead limit exceeded
[junit4] at java.lang.StringBuilder.toString(StringBuilder.java:407)
[junit4] at java.lang.StackTraceElement.toString(StackTraceElement.java:173)
[junit4] at java.lang.String.valueOf(String.java:2994)
[junit4] at java.lang.StringBuilder.append(StringBuilder.java:131)
[junit4] at java.lang.Throwable.printStackTrace(Throwable.java:658)
[junit4] at java.lang.Throwable.printStackTrace(Throwable.java:721)
[junit4] at org.junit.runner.notification.Failure.getTrace(Failure.java:67)
[junit4] java.lang.OutOfMemoryError: GC overhead limit exceeded (stack unavailable; OOM)
[junit4] <<< JVM J0: EOF ----
[...truncated 9403 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/build.xml:626: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/build.xml:570: Some of the tests produced a heap dump, but did not fail. Maybe a suppressed OutOfMemoryError? Dumps created:
* java_pid8633.hprof
Total time: 238 minutes 58 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any