You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Steve Rowe (JIRA)" <ji...@apache.org> on 2018/10/01 18:53:00 UTC
[jira] [Commented] (SOLR-12709) Simulate a 1 bln docs scaling-up
scenario
[ https://issues.apache.org/jira/browse/SOLR-12709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16634477#comment-16634477 ]
Steve Rowe commented on SOLR-12709:
-----------------------------------
{{TestSimExtremeIndexing.testScaleUp()}} failure from [https://builds.apache.org/job/Lucene-Solr-SmokeRelease-master/1135/], the seed only reproduced 1/5 iterations for me locally:
{noformat}
Checking out Revision 2369c8963412773592098475bdd8af1da81e3ac5 (refs/remotes/origin/master)
[...]
[smoker] [junit4] 2> 1221947 DEBUG (TEST-TestSimExtremeIndexing.testScaleUp-seed#[4D17B79C7402215C]) [ ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 218
[smoker] [junit4] 2> 1222001 INFO (TEST-TestSimExtremeIndexing.testScaleUp-seed#[4D17B79C7402215C]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase #######################################
[smoker] [junit4] 2> ############ CLUSTER STATE ############
[smoker] [junit4] 2> #######################################
[smoker] [junit4] 2> ## Live nodes: 200
[smoker] [junit4] 2> ## Empty nodes: 191
[smoker] [junit4] 2> ## Dead nodes: 0
[smoker] [junit4] 2> ## Collections:
[smoker] [junit4] 2> ## * .system
[smoker] [junit4] 2> ## shardsTotal 1
[smoker] [junit4] 2> ## shardsState {active=1}
[smoker] [junit4] 2> ## shardsWithoutLeader 0
[smoker] [junit4] 2> ## totalReplicas 1
[smoker] [junit4] 2> ## activeReplicas 1
[smoker] [junit4] 2> ## inactiveReplicas 0
[smoker] [junit4] 2> ## totalActiveDocs 34
[smoker] [junit4] 2> ## bufferedDocs 0
[smoker] [junit4] 2> ## maxActiveSliceDocs 34
[smoker] [junit4] 2> ## minActiveSliceDocs 34
[smoker] [junit4] 2> ## avgActiveSliceDocs 34
[smoker] [junit4] 2> ## totalInactiveDocs 0
[smoker] [junit4] 2> ## maxInactiveSliceDocs 0
[smoker] [junit4] 2> ## minInactiveSliceDocs 0
[smoker] [junit4] 2> ## avgInactiveSliceDocs NaN
[smoker] [junit4] 2> ## totalActiveBytes 27,240
[smoker] [junit4] 2> ## maxActiveSliceBytes 27,240
[smoker] [junit4] 2> ## minActiveSliceBytes 27,240
[smoker] [junit4] 2> ## avgActiveSliceBytes 27,240
[smoker] [junit4] 2> ## totalInactiveBytes 0
[smoker] [junit4] 2> ## maxInactiveSliceBytes 0
[smoker] [junit4] 2> ## minInactiveSliceBytes 0
[smoker] [junit4] 2> ## avgInactiveSliceBytes NaN
[smoker] [junit4] 2> ## totalActiveDeletedDocs 0
[smoker] [junit4] 2> ## * testScaleUp_collection
[smoker] [junit4] 2> ## shardsTotal 92
[smoker] [junit4] 2> ## shardsState {active=46, construction=2, inactive=44}
[smoker] [junit4] 2> ## shardsWithoutLeader 0
[smoker] [junit4] 2> ## totalReplicas 184
[smoker] [junit4] 2> ## activeReplicas 92
[smoker] [junit4] 2> ## inactiveReplicas 92
[smoker] [junit4] 2> ## totalActiveDocs 1,000,000,000
[smoker] [junit4] 2> ## bufferedDocs 204,400,000
[smoker] [junit4] 2> ## maxActiveSliceDocs 45,600,000
[smoker] [junit4] 2> ## minActiveSliceDocs 15,625,000
[smoker] [junit4] 2> ## avgActiveSliceDocs 17,295,652
[smoker] [junit4] 2> ## totalInactiveDocs 1,634,193,750
[smoker] [junit4] 2> ## maxInactiveSliceDocs 58,650,000
[smoker] [junit4] 2> ## minInactiveSliceDocs 22,800,000
[smoker] [junit4] 2> ## avgInactiveSliceDocs 35,525,951
[smoker] [junit4] 2> ## totalActiveBytes 390,916,321,320
[smoker] [junit4] 2> ## maxActiveSliceBytes 19,875,010,240
[smoker] [junit4] 2> ## minActiveSliceBytes 7,673,447,740
[smoker] [junit4] 2> ## avgActiveSliceBytes 8,498,180,898
[smoker] [junit4] 2> ## totalInactiveBytes 787,600,706,560
[smoker] [junit4] 2> ## maxInactiveSliceBytes 28,125,010,240
[smoker] [junit4] 2> ## minInactiveSliceBytes 11,400,010,240
[smoker] [junit4] 2> ## avgInactiveSliceBytes 17,121,754,490
[smoker] [junit4] 2> ## totalActiveDeletedDocs 0
[smoker] [junit4] 2> ## Max replicas per node: 45
[smoker] [junit4] 2> ## Min replicas per node: 0
[smoker] [junit4] 2> ## Total replicas: 185
[smoker] [junit4] 2> ## * .system 1
[smoker] [junit4] 2> ## - active 1
[smoker] [junit4] 2> ## * testScaleUp_collection 184
[smoker] [junit4] 2> ## - active 184
[smoker] [junit4] 2> ######### Solr op counts ##########
[smoker] [junit4] 2> ## - CREATE 1
[smoker] [junit4] 2> ## - DELETESTATUS 51
[smoker] [junit4] 2> ## - REQUESTSTATUS 51
[smoker] [junit4] 2> ## - SPLITSHARD 52
[smoker] [junit4] 2> ## - autoscaling 1
[smoker] [junit4] 2> ## - query 1
[smoker] [junit4] 2> ## - update 5042
[smoker] [junit4] 2> ######### Autoscaling event counts ###########
[smoker] [junit4] 2> ## * Trigger: scaleUpTrigger
[smoker] [junit4] 2> ## - FAILED 1
[smoker] [junit4] 2> ## - STARTED 8
[smoker] [junit4] 2> ## - SUCCEEDED 1
[smoker] [junit4] 2>
[smoker] [junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestSimExtremeIndexing -Dtests.method=testScaleUp -Dtests.seed=4D17B79C7402215C -Dtests.multiplier=2 -Dtests.locale=ro -Dtests.timezone=America/Catamarca -Dtests.asserts=true -Dtests.file.encoding=UTF-8
[smoker] [junit4] FAILURE 4.88s J1 | TestSimExtremeIndexing.testScaleUp <<<
[smoker] [junit4] > Throwable #1: java.lang.AssertionError: {numFound=795600000,start=0,docs=[]} expected:<1000000000> but was:<795600000>
[smoker] [junit4] > at __randomizedtesting.SeedInfo.seed([4D17B79C7402215C:6C49F13E782CFFFD]:0)
[smoker] [junit4] > at org.apache.solr.cloud.autoscaling.sim.TestSimExtremeIndexing.testScaleUp(TestSimExtremeIndexing.java:121)
[smoker] [junit4] > at java.lang.Thread.run(Thread.java:748)
[smoker] [junit4] 2> 1222003 DEBUG (SUITE-TestSimExtremeIndexing-seed#[4D17B79C7402215C]-worker) [ ] o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool executor now
[smoker] [junit4] 2> 1222003 DEBUG (SUITE-TestSimExtremeIndexing-seed#[4D17B79C7402215C]-worker) [ ] o.a.s.c.a.ScheduledTriggers Shutting down action executor now
[smoker] [junit4] 2> 1222003 WARN (SolrRrdBackendFactory-5969-thread-1) [ ] o.a.s.m.r.SolrRrdBackendFactory Error updating RRD data for solr.jvm
[smoker] [junit4] 2> java.io.IOException: java.lang.InterruptedException
[smoker] [junit4] 2> at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:653) ~[test/:?]
[smoker] [junit4] 2> at org.apache.solr.cloud.autoscaling.sim.SimCloudManager$1.request(SimCloudManager.java:224) ~[test/:?]
[smoker] [junit4] 2> at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194) ~[java/:?]
[smoker] [junit4] 2> at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:173) ~[java/:?]
[smoker] [junit4] 2> at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:138) ~[java/:?]
[smoker] [junit4] 2> at org.apache.solr.metrics.rrd.SolrRrdBackendFactory.lambda$maybeSyncBackends$5(SolrRrdBackendFactory.java:357) ~[java/:?]
[smoker] [junit4] 2> at java.util.HashMap.forEach(HashMap.java:1289) ~[?:1.8.0_172]
[smoker] [junit4] 2> at org.apache.solr.metrics.rrd.SolrRrdBackendFactory.maybeSyncBackends(SolrRrdBackendFactory.java:350) ~[java/:?]
[smoker] [junit4] 2> at org.apache.solr.metrics.rrd.SolrRrdBackendFactory.lambda$new$0(SolrRrdBackendFactory.java:110) ~[java/:?]
[smoker] [junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_172]
[smoker] [junit4] 2> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_172]
[smoker] [junit4] 2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_172]
[smoker] [junit4] 2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_172]
[smoker] [junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_172]
[smoker] [junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_172]
[smoker] [junit4] 2> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
[smoker] [junit4] 2> Caused by: java.lang.InterruptedException
[smoker] [junit4] 2> at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:404) ~[?:1.8.0_172]
[smoker] [junit4] 2> at java.util.concurrent.FutureTask.get(FutureTask.java:191) ~[?:1.8.0_172]
[smoker] [junit4] 2> at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:651) ~[test/:?]
[smoker] [junit4] 2> ... 15 more
[smoker] [junit4] 2> 1222004 WARN (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
[smoker] [junit4] 2> 1222004 DEBUG (simCloudManagerPool-5971-thread-33) [ ] o.a.s.c.a.s.SimClusterStateProvider ** creating new collection states
[smoker] [junit4] 2> 1222005 WARN (SolrRrdBackendFactory-5969-thread-1) [ ] o.a.s.m.r.SolrRrdBackendFactory Error updating RRD data for solr.collection.testScaleUp_collection
[smoker] [junit4] 2> java.io.IOException: java.util.concurrent.ExecutionException: java.io.IOException: already closed
[smoker] [junit4] 2> at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:653) ~[test/:?]
[smoker] [junit4] 2> at org.apache.solr.cloud.autoscaling.sim.SimCloudManager$1.request(SimCloudManager.java:224) ~[test/:?]
[smoker] [junit4] 2> at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194) ~[java/:?]
[smoker] [junit4] 2> at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:173) ~[java/:?]
[smoker] [junit4] 2> at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:138) ~[java/:?]
[smoker] [junit4] 2> at org.apache.solr.metrics.rrd.SolrRrdBackendFactory.lambda$maybeSyncBackends$5(SolrRrdBackendFactory.java:357) ~[java/:?]
[smoker] [junit4] 2> at java.util.HashMap.forEach(HashMap.java:1289) ~[?:1.8.0_172]
[smoker] [junit4] 2> at org.apache.solr.metrics.rrd.SolrRrdBackendFactory.maybeSyncBackends(SolrRrdBackendFactory.java:350) ~[java/:?]
[smoker] [junit4] 2> at org.apache.solr.metrics.rrd.SolrRrdBackendFactory.lambda$new$0(SolrRrdBackendFactory.java:110) ~[java/:?]
[smoker] [junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_172]
[smoker] [junit4] 2> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_172]
[smoker] [junit4] 2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_172]
[smoker] [junit4] 2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_172]
[smoker] [junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_172]
[smoker] [junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_172]
[smoker] [junit4] 2> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
[smoker] [junit4] 2> Caused by: java.util.concurrent.ExecutionException: java.io.IOException: already closed
[smoker] [junit4] 2> at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:1.8.0_172]
[smoker] [junit4] 2> at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[?:1.8.0_172]
[smoker] [junit4] 2> at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:651) ~[test/:?]
[smoker] [junit4] 2> ... 15 more
[smoker] [junit4] 2> Caused by: java.io.IOException: already closed
[smoker] [junit4] 2> at org.apache.solr.cloud.autoscaling.sim.SimClusterStateProvider.ensureNotClosed(SimClusterStateProvider.java:2138) ~[test/:?]
[smoker] [junit4] 2> at org.apache.solr.cloud.autoscaling.sim.SimClusterStateProvider.simUpdate(SimClusterStateProvider.java:1367) ~[test/:?]
[smoker] [junit4] 2> at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.simHandleSolrRequest(SimCloudManager.java:738) ~[test/:?]
[smoker] [junit4] 2> at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.lambda$request$15(SimCloudManager.java:650) ~[test/:?]
[smoker] [junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_172]
[smoker] [junit4] 2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209) ~[java/:?]
[smoker] [junit4] 2> ... 3 more
[smoker] [junit4] 2> 1222006 WARN (AutoscalingActionExecutor-5973-thread-1) [ ] o.a.s.c.a.SystemLogListener Exception sending event to collection .system
[smoker] [junit4] 2> java.io.IOException: java.util.concurrent.ExecutionException: java.io.IOException: already closed
[smoker] [junit4] 2> at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:653) ~[test/:?]
[smoker] [junit4] 2> at org.apache.solr.cloud.autoscaling.SystemLogListener.onEvent(SystemLogListener.java:118) ~[java/:?]
[smoker] [junit4] 2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers$TriggerListeners.fireListeners(ScheduledTriggers.java:792) ~[java/:?]
[smoker] [junit4] 2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:310) ~[java/:?]
[smoker] [junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_172]
[smoker] [junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_172]
[smoker] [junit4] 2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209) ~[java/:?]
[smoker] [junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_172]
[smoker] [junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_172]
[smoker] [junit4] 2> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
[smoker] [junit4] 2> Caused by: java.util.concurrent.ExecutionException: java.io.IOException: already closed
[smoker] [junit4] 2> at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:1.8.0_172]
[smoker] [junit4] 2> at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[?:1.8.0_172]
[smoker] [junit4] 2> at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:651) ~[test/:?]
[smoker] [junit4] 2> ... 9 more
[smoker] [junit4] 2> Caused by: java.io.IOException: already closed
[smoker] [junit4] 2> at org.apache.solr.cloud.autoscaling.sim.SimClusterStateProvider.ensureNotClosed(SimClusterStateProvider.java:2138) ~[test/:?]
[smoker] [junit4] 2> at org.apache.solr.cloud.autoscaling.sim.SimClusterStateProvider.simUpdate(SimClusterStateProvider.java:1367) ~[test/:?]
[smoker] [junit4] 2> at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.simHandleSolrRequest(SimCloudManager.java:738) ~[test/:?]
[smoker] [junit4] 2> at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.lambda$request$15(SimCloudManager.java:650) ~[test/:?]
[smoker] [junit4] 2> ... 5 more
[smoker] [junit4] 2> 1222009 DEBUG (simCloudManagerPool-5971-thread-33) [ ] o.a.s.c.a.s.SimClusterStateProvider ** saved cluster state version 219
[smoker] [junit4] 2> 1222009 WARN (AutoscalingActionExecutor-5973-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Exception executing actions
[smoker] [junit4] 2> org.apache.solr.cloud.autoscaling.TriggerActionException: Error processing action for trigger event: {
[smoker] [junit4] 2> "id":"1310b6c3b09c3cT31vg5ry49ycvekjxnv9ajjpzp",
[smoker] [junit4] 2> "source":"scaleUpTrigger",
[smoker] [junit4] 2> "eventTime":5366401710726204,
[smoker] [junit4] 2> "eventType":"INDEXSIZE",
[smoker] [junit4] 2> "properties":{
[smoker] [junit4] 2> "__start__":25,
[smoker] [junit4] 2> "aboveSize":{
[smoker] [junit4] 2> "testScaleUp_collection_shard1_0_0_0_0_replica_n217":"docs=24712500, bytes=12075020480",
[smoker] [junit4] 2> "testScaleUp_collection_shard1_0_0_0_1_replica_n223":"docs=24712500, bytes=12075010240",
[smoker] [junit4] 2> "testScaleUp_collection_shard1_0_0_1_0_replica_n127":"docs=24712500, bytes=12015645480",
[smoker] [junit4] 2> "testScaleUp_collection_shard1_0_0_1_1_replica_n133":"docs=24712500, bytes=12015635240",
[smoker] [junit4] 2> "testScaleUp_collection_shard1_0_1_0_0_replica_n199":"docs=24712500, bytes=12065645480",
[smoker] [junit4] 2> "testScaleUp_collection_shard1_0_1_0_1_replica_n203":"docs=24712500, bytes=12065635240",
[smoker] [junit4] 2> "testScaleUp_collection_shard1_0_1_1_0_replica_n179":"docs=24712500, bytes=12103145480",
[smoker] [junit4] 2> "testScaleUp_collection_shard1_0_1_1_1_replica_n183":"docs=24712500, bytes=12103135240",
[smoker] [junit4] 2> "testScaleUp_collection_shard1_1_replica_n11":"docs=45600000, bytes=19875010240",
[smoker] [junit4] 2> "testScaleUp_collection_shard2_0_0_0_0_replica_n137":"docs=24712500, bytes=11990645480",
[smoker] [junit4] 2> "testScaleUp_collection_shard2_0_0_0_1_replica_n143":"docs=24712500, bytes=11990635240",
[smoker] [junit4] 2> "testScaleUp_collection_shard2_0_0_1_0_replica_n237":"docs=24712500, bytes=12075020480",
[smoker] [junit4] 2> "testScaleUp_collection_shard2_0_0_1_1_replica_n243":"docs=24712500, bytes=12075010240",
[smoker] [junit4] 2> "testScaleUp_collection_shard2_0_1_0_0_replica_n227":"docs=24712500, bytes=12081270480",
[smoker] [junit4] 2> "testScaleUp_collection_shard2_0_1_0_1_replica_n233":"docs=24712500, bytes=12081260240",
[smoker] [junit4] 2> "testScaleUp_collection_shard2_0_1_1_0_replica_n167":"docs=24712500, bytes=12053145480",
[smoker] [junit4] 2> "testScaleUp_collection_shard2_0_1_1_1_replica_n173":"docs=24712500, bytes=12053135240",
[smoker] [junit4] 2> "testScaleUp_collection_shard2_1_0_0_0_replica_n147":"docs=24712500, bytes=12003145480",
[smoker] [junit4] 2> "testScaleUp_collection_shard2_1_0_0_1_replica_n153":"docs=24712500, bytes=12003135240",
[smoker] [junit4] 2> "testScaleUp_collection_shard2_1_0_1_0_replica_n209":"docs=24712500, bytes=12156270480",
[smoker] [junit4] 2> "testScaleUp_collection_shard2_1_0_1_1_replica_n213":"docs=24712500, bytes=12156260240",
[smoker] [junit4] 2> "testScaleUp_collection_shard2_1_1_0_0_replica_n189":"docs=24712500, bytes=12200020480",
[smoker] [junit4] 2> "testScaleUp_collection_shard2_1_1_0_1_replica_n193":"docs=24712500, bytes=12200010240",
[smoker] [junit4] 2> "testScaleUp_collection_shard2_1_1_1_0_replica_n157":"docs=24712500, bytes=12068770480",
[smoker] [junit4] 2> "testScaleUp_collection_shard2_1_1_1_1_replica_n163":"docs=24712500, bytes=12068760240"},
[smoker] [junit4] 2> "belowSize":{},
[smoker] [junit4] 2> "_enqueue_time_":5366407403469004,
[smoker] [junit4] 2> "requestedOps":[
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard2_0_0_1_0"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard1_0_1_0_1"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard2_0_1_1_1"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard2_1_0_0_0"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard2_1_1_1_0"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard2_0_0_0_0"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard2_0_0_0_1"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard1_0_1_1_1"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard2_0_0_1_1"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard2_1_1_0_0"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard2_1_0_1_0"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard2_1_1_1_1"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard1_0_0_1_1"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard1_0_1_0_0"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard2_0_1_0_0"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard2_1_0_0_1"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard1_1"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard1_0_0_0_0"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard2_0_1_1_0"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard2_1_1_0_1"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard1_0_1_1_0"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard1_0_0_0_1"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard2_0_1_0_1"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard1_0_0_1_0"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard2_1_0_1_1"}]}}]}}
[smoker] [junit4] 2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:311) ~[java/:?]
[smoker] [junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_172]
[smoker] [junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_172]
[smoker] [junit4] 2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209) ~[java/:?]
[smoker] [junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_172]
[smoker] [junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_172]
[smoker] [junit4] 2> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
[smoker] [junit4] 2> Caused by: org.apache.solr.common.SolrException: Unexpected exception while processing event: {
[smoker] [junit4] 2> "id":"1310b6c3b09c3cT31vg5ry49ycvekjxnv9ajjpzp",
[smoker] [junit4] 2> "source":"scaleUpTrigger",
[smoker] [junit4] 2> "eventTime":5366401710726204,
[smoker] [junit4] 2> "eventType":"INDEXSIZE",
[smoker] [junit4] 2> "properties":{
[smoker] [junit4] 2> "__start__":25,
[smoker] [junit4] 2> "aboveSize":{
[smoker] [junit4] 2> "testScaleUp_collection_shard1_0_0_0_0_replica_n217":"docs=24712500, bytes=12075020480",
[smoker] [junit4] 2> "testScaleUp_collection_shard1_0_0_0_1_replica_n223":"docs=24712500, bytes=12075010240",
[smoker] [junit4] 2> "testScaleUp_collection_shard1_0_0_1_0_replica_n127":"docs=24712500, bytes=12015645480",
[smoker] [junit4] 2> "testScaleUp_collection_shard1_0_0_1_1_replica_n133":"docs=24712500, bytes=12015635240",
[smoker] [junit4] 2> "testScaleUp_collection_shard1_0_1_0_0_replica_n199":"docs=24712500, bytes=12065645480",
[smoker] [junit4] 2> "testScaleUp_collection_shard1_0_1_0_1_replica_n203":"docs=24712500, bytes=12065635240",
[smoker] [junit4] 2> "testScaleUp_collection_shard1_0_1_1_0_replica_n179":"docs=24712500, bytes=12103145480",
[smoker] [junit4] 2> "testScaleUp_collection_shard1_0_1_1_1_replica_n183":"docs=24712500, bytes=12103135240",
[smoker] [junit4] 2> "testScaleUp_collection_shard1_1_replica_n11":"docs=45600000, bytes=19875010240",
[smoker] [junit4] 2> "testScaleUp_collection_shard2_0_0_0_0_replica_n137":"docs=24712500, bytes=11990645480",
[smoker] [junit4] 2> "testScaleUp_collection_shard2_0_0_0_1_replica_n143":"docs=24712500, bytes=11990635240",
[smoker] [junit4] 2> "testScaleUp_collection_shard2_0_0_1_0_replica_n237":"docs=24712500, bytes=12075020480",
[smoker] [junit4] 2> "testScaleUp_collection_shard2_0_0_1_1_replica_n243":"docs=24712500, bytes=12075010240",
[smoker] [junit4] 2> "testScaleUp_collection_shard2_0_1_0_0_replica_n227":"docs=24712500, bytes=12081270480",
[smoker] [junit4] 2> "testScaleUp_collection_shard2_0_1_0_1_replica_n233":"docs=24712500, bytes=12081260240",
[smoker] [junit4] 2> "testScaleUp_collection_shard2_0_1_1_0_replica_n167":"docs=24712500, bytes=12053145480",
[smoker] [junit4] 2> "testScaleUp_collection_shard2_0_1_1_1_replica_n173":"docs=24712500, bytes=12053135240",
[smoker] [junit4] 2> "testScaleUp_collection_shard2_1_0_0_0_replica_n147":"docs=24712500, bytes=12003145480",
[smoker] [junit4] 2> "testScaleUp_collection_shard2_1_0_0_1_replica_n153":"docs=24712500, bytes=12003135240",
[smoker] [junit4] 2> "testScaleUp_collection_shard2_1_0_1_0_replica_n209":"docs=24712500, bytes=12156270480",
[smoker] [junit4] 2> "testScaleUp_collection_shard2_1_0_1_1_replica_n213":"docs=24712500, bytes=12156260240",
[smoker] [junit4] 2> "testScaleUp_collection_shard2_1_1_0_0_replica_n189":"docs=24712500, bytes=12200020480",
[smoker] [junit4] 2> "testScaleUp_collection_shard2_1_1_0_1_replica_n193":"docs=24712500, bytes=12200010240",
[smoker] [junit4] 2> "testScaleUp_collection_shard2_1_1_1_0_replica_n157":"docs=24712500, bytes=12068770480",
[smoker] [junit4] 2> "testScaleUp_collection_shard2_1_1_1_1_replica_n163":"docs=24712500, bytes=12068760240"},
[smoker] [junit4] 2> "belowSize":{},
[smoker] [junit4] 2> "_enqueue_time_":5366407403469004,
[smoker] [junit4] 2> "requestedOps":[
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard2_0_0_1_0"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard1_0_1_0_1"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard2_0_1_1_1"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard2_1_0_0_0"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard2_1_1_1_0"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard2_0_0_0_0"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard2_0_0_0_1"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard1_0_1_1_1"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard2_0_0_1_1"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard2_1_1_0_0"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard2_1_0_1_0"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard2_1_1_1_1"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard1_0_0_1_1"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard1_0_1_0_0"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard2_0_1_0_0"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard2_1_0_0_1"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard1_1"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard1_0_0_0_0"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard2_0_1_1_0"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard2_1_1_0_1"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard1_0_1_1_0"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard1_0_0_0_1"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard2_0_1_0_1"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard1_0_0_1_0"}]}},
[smoker] [junit4] 2> {
[smoker] [junit4] 2> "action":"SPLITSHARD",
[smoker] [junit4] 2> "hints":{"COLL_SHARD":[{
[smoker] [junit4] 2> "first":"testScaleUp_collection",
[smoker] [junit4] 2> "second":"shard2_1_0_1_1"}]}}]}}
[smoker] [junit4] 2> at org.apache.solr.cloud.autoscaling.ExecutePlanAction.process(ExecutePlanAction.java:120) ~[java/:?]
[smoker] [junit4] 2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:308) ~[java/:?]
[smoker] [junit4] 2> ... 6 more
[smoker] [junit4] 2> Caused by: org.apache.solr.common.SolrException: Unexpected exception executing operation: action=SPLITSHARD&async=scaleUpTrigger/1310b6c3b09c3cT31vg5ry49ycvekjxnv9ajjpzp/0&waitForFinalState=true&collection=testScaleUp_collection&shard=shard2_0_1_0_1
[smoker] [junit4] 2> at org.apache.solr.cloud.autoscaling.ExecutePlanAction.process(ExecutePlanAction.java:110) ~[java/:?]
[smoker] [junit4] 2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:308) ~[java/:?]
[smoker] [junit4] 2> ... 6 more
[smoker] [junit4] 2> Caused by: java.io.IOException: java.lang.InterruptedException
[smoker] [junit4] 2> at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:653) ~[test/:?]
[smoker] [junit4] 2> at org.apache.solr.cloud.autoscaling.ExecutePlanAction.process(ExecutePlanAction.java:80) ~[java/:?]
[smoker] [junit4] 2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:308) ~[java/:?]
[smoker] [junit4] 2> ... 6 more
[smoker] [junit4] 2> Caused by: java.lang.InterruptedException
[smoker] [junit4] 2> at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:404) ~[?:1.8.0_172]
[smoker] [junit4] 2> at java.util.concurrent.FutureTask.get(FutureTask.java:191) ~[?:1.8.0_172]
[smoker] [junit4] 2> at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:651) ~[test/:?]
[smoker] [junit4] 2> at org.apache.solr.cloud.autoscaling.ExecutePlanAction.process(ExecutePlanAction.java:80) ~[java/:?]
[smoker] [junit4] 2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:308) ~[java/:?]
[smoker] [junit4] 2> ... 6 more
[smoker] [junit4] 2> 1222010 DEBUG (AutoscalingActionExecutor-5973-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing took 176888 ms for event id=1310b6c3b09c3cT31vg5ry49ycvekjxnv9ajjpzp
[smoker] [junit4] 2> 1222007 DEBUG (SUITE-TestSimExtremeIndexing-seed#[4D17B79C7402215C]-worker) [ ] o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
[smoker] [junit4] 2> 1222010 DEBUG (SUITE-TestSimExtremeIndexing-seed#[4D17B79C7402215C]-worker) [ ] o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread pool executor
[smoker] [junit4] 2> 1222010 DEBUG (SUITE-TestSimExtremeIndexing-seed#[4D17B79C7402215C]-worker) [ ] o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
[smoker] [junit4] 2> 1222011 DEBUG (SUITE-TestSimExtremeIndexing-seed#[4D17B79C7402215C]-worker) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
[smoker] [junit4] 2> 1222011 WARN (SolrRrdBackendFactory-5969-thread-1) [ ] o.a.s.m.r.SolrRrdBackendFactory Error updating RRD data for solr.node
[smoker] [junit4] 2> java.io.IOException: java.util.concurrent.ExecutionException: java.io.IOException: already closed
[smoker] [junit4] 2> at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:653) ~[test/:?]
[smoker] [junit4] 2> at org.apache.solr.cloud.autoscaling.sim.SimCloudManager$1.request(SimCloudManager.java:224) ~[test/:?]
[smoker] [junit4] 2> at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194) ~[java/:?]
[smoker] [junit4] 2> at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:173) ~[java/:?]
[smoker] [junit4] 2> at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:138) ~[java/:?]
[smoker] [junit4] 2> at org.apache.solr.metrics.rrd.SolrRrdBackendFactory.lambda$maybeSyncBackends$5(SolrRrdBackendFactory.java:357) ~[java/:?]
[smoker] [junit4] 2> at java.util.HashMap.forEach(HashMap.java:1289) ~[?:1.8.0_172]
[smoker] [junit4] 2> at org.apache.solr.metrics.rrd.SolrRrdBackendFactory.maybeSyncBackends(SolrRrdBackendFactory.java:350) ~[java/:?]
[smoker] [junit4] 2> at org.apache.solr.metrics.rrd.SolrRrdBackendFactory.lambda$new$0(SolrRrdBackendFactory.java:110) ~[java/:?]
[smoker] [junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_172]
[smoker] [junit4] 2> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_172]
[smoker] [junit4] 2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_172]
[smoker] [junit4] 2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_172]
[smoker] [junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_172]
[smoker] [junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_172]
[smoker] [junit4] 2> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
[smoker] [junit4] 2> Caused by: java.util.concurrent.ExecutionException: java.io.IOException: already closed
[smoker] [junit4] 2> at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:1.8.0_172]
[smoker] [junit4] 2> at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[?:1.8.0_172]
[smoker] [junit4] 2> at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:651) ~[test/:?]
[smoker] [junit4] 2> ... 15 more
[smoker] [junit4] 2> Caused by: java.io.IOException: already closed
[smoker] [junit4] 2> at org.apache.solr.cloud.autoscaling.sim.SimClusterStateProvider.ensureNotClosed(SimClusterStateProvider.java:2138) ~[test/:?]
[smoker] [junit4] 2> at org.apache.solr.cloud.autoscaling.sim.SimClusterStateProvider.simUpdate(SimClusterStateProvider.java:1367) ~[test/:?]
[smoker] [junit4] 2> at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.simHandleSolrRequest(SimCloudManager.java:738) ~[test/:?]
[smoker] [junit4] 2> at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.lambda$request$15(SimCloudManager.java:650) ~[test/:?]
[smoker] [junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_172]
[smoker] [junit4] 2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209) ~[java/:?]
[smoker] [junit4] 2> ... 3 more
[smoker] [junit4] 2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-master/lucene/build/smokeTestRelease/tmp/unpack/solr-8.0.0/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.sim.TestSimExtremeIndexing_4D17B79C7402215C-001
[smoker] [junit4] 2> 1222029 WARN (SolrRrdBackendFactory-5969-thread-1) [ ] o.a.s.m.r.SolrRrdBackendFactory Error sending RRD data updates
[smoker] [junit4] 2> java.io.IOException: java.util.concurrent.RejectedExecutionException: Task org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$129/198524200@1817d8df rejected from org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor@2a463d6b[Shutting down, pool size = 142, active threads = 0, queued tasks = 0, completed tasks = 5300]
[smoker] [junit4] 2> at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:653) ~[test/:?]
[smoker] [junit4] 2> at org.apache.solr.cloud.autoscaling.sim.SimCloudManager$1.request(SimCloudManager.java:224) ~[test/:?]
[smoker] [junit4] 2> at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194) ~[java/:?]
[smoker] [junit4] 2> at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:484) ~[java/:?]
[smoker] [junit4] 2> at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:448) ~[java/:?]
[smoker] [junit4] 2> at org.apache.solr.metrics.rrd.SolrRrdBackendFactory.maybeSyncBackends(SolrRrdBackendFactory.java:366) ~[java/:?]
[smoker] [junit4] 2> at org.apache.solr.metrics.rrd.SolrRrdBackendFactory.lambda$new$0(SolrRrdBackendFactory.java:110) ~[java/:?]
[smoker] [junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_172]
[smoker] [junit4] 2> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_172]
[smoker] [junit4] 2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_172]
[smoker] [junit4] 2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_172]
[smoker] [junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_172]
[smoker] [junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_172]
[smoker] [junit4] 2> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
[smoker] [junit4] 2> Caused by: java.util.concurrent.RejectedExecutionException: Task org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$129/198524200@1817d8df rejected from org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor@2a463d6b[Shutting down, pool size = 142, active threads = 0, queued tasks = 0, completed tasks = 5300]
[smoker] [junit4] 2> at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) ~[?:1.8.0_172]
[smoker] [junit4] 2> at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) ~[?:1.8.0_172]
[smoker] [junit4] 2> at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) ~[?:1.8.0_172]
[smoker] [junit4] 2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:194) ~[java/:?]
[smoker] [junit4] 2> at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134) ~[?:1.8.0_172]
[smoker] [junit4] 2> at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.submit(SimCloudManager.java:569) ~[test/:?]
[smoker] [junit4] 2> at org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:650) ~[test/:?]
[smoker] [junit4] 2> ... 13 more
[smoker] [junit4] 2> Sep 27, 2018 7:42:57 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[smoker] [junit4] 2> WARNING: Will linger awaiting termination of 18 leaked thread(s).
[smoker] [junit4] 2> NOTE: test params are: codec=Asserting(Lucene80), sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@2a3300f8), locale=ro, timezone=America/Catamarca
[smoker] [junit4] 2> NOTE: Linux 4.4.0-130-generic amd64/Oracle Corporation 1.8.0_172 (64-bit)/cpus=4,threads=1,free=223113456,total=518520832
[smoker] [junit4] 2> NOTE: All tests run in this JVM: [SearchHandlerTest, AutoScalingHandlerTest, TestRangeQuery, TestPerFieldSimilarity, TestBlobHandler, TestManagedSchema, ParsingFieldUpdateProcessorsTest, MetricsHistoryHandlerTest, TestLegacyTerms, DirectSolrSpellCheckerTest, ConfigSetsAPITest, TestConfig, TestNumericRangeQuery64, TestQueryUtils, TestValueSourceCache, SolrInfoBeanTest, TestMiniSolrCloudClusterSSL, SolrIndexSplitterTest, ProtectedTermFilterFactoryTest, DefaultValueUpdateProcessorTest, TestClusterProperties, TestAuthorizationFramework, TriggerSetPropertiesIntegrationTest, TestMacroExpander, TestSSLRandomization, TestDistribDocBasedVersion, FacetPivotSmallTest, HLLUtilTest, CdcrRequestHandlerTest, TestWithCollection, DateRangeFieldTest, V2StandaloneTest, TestCoreDiscovery, BasicFunctionalityTest, TestSolrIndexConfig, CachingDirectoryFactoryTest, NodeMarkersRegistrationTest, TestReload, DeleteStatusTest, DisMaxRequestHandlerTest, TestJavabinTupleStreamParser, TestManagedResource, TestCustomDocTransformer, TestCloudJSONFacetSKG, AnalysisErrorHandlingTest, JavabinLoaderTest, TestReversedWildcardFilterFactory, TestSimNodeAddedTrigger, SimpleMLTQParserTest, HealthCheckHandlerTest, JvmMetricsTest, TestBlendedInfixSuggestions, TestLMDirichletSimilarityFactory, TestFaceting, TestDirectoryFactory, HdfsRecoverLeaseTest, TestExceedMaxTermLength, TestDynamicFieldResource, TestNRTOpen, TestHdfsCloudBackupRestore, SpellCheckCollatorTest, TestUpdate, TestTolerantSearch, UtilsToolTest, TestFastLRUCache, BasicAuthIntegrationTest, TestOmitPositions, QueryElevationComponentTest, TestDistributedSearch, TestPayloadScoreQParserPlugin, PeerSyncTest, SyncSliceTest, HdfsDirectoryTest, ActionThrottleTest, DistributedSpellCheckComponentTest, HdfsChaosMonkeyNothingIsSafeTest, TriggerCooldownIntegrationTest, LukeRequestHandlerTest, TestNumericTerms64, TestImplicitCoreProperties, LargeFieldTest, TestDistributedStatsComponentCardinality, DistributedQueryComponentCustomSortTest, TestJsonFacetRefinement, WrapperMergePolicyFactoryTest, CdcrBootstrapTest, TestExclusionRuleCollectionAccess, PeerSyncReplicationTest, BasicDistributedZkTest, ImplicitSnitchTest, DirectUpdateHandlerTest, TestClusterStateMutator, CollectionsAPIAsyncDistributedZkTest, LeaderFailureAfterFreshStartTest, ScriptEngineTest, DistributedFacetSimpleRefinementLongTailTest, SimplePostToolTest, TestSolr4Spatial, SuggesterFSTTest, SchemaApiFailureTest, TestNoOpRegenerator, DirectoryFactoryTest, TestBulkSchemaConcurrent, ChaosMonkeyNothingIsSafeWithPullReplicasTest, OpenCloseCoreStressTest, ReplicationFactorTest, TestSolrCloudWithHadoopAuthPlugin, TestReplicationHandlerBackup, TestUseDocValuesAsStored2, DistributedDebugComponentTest, TestTestInjection, MultiTermTest, ZkNodePropsTest, NodeAddedTriggerTest, Tagger2Test, DocValuesNotIndexedTest, TestMinMaxOnMultiValuedField, QueryParsingTest, TestSolrQueryResponse, FieldMutatingUpdateProcessorTest, AssignBackwardCompatibilityTest, TermVectorComponentDistributedTest, TestFieldTypeCollectionResource, TestUnifiedSolrHighlighter, StatsComponentTest, TestSort, TestCollationField, TestConfigSetsAPIZkFailure, TestSimDistribStateManager, PreAnalyzedFieldTest, BadComponentTest, NumberUtilsTest, SpatialRPTFieldTypeTest, TestSolrCloudSnapshots, DeleteInactiveReplicaTest, DeleteNodeTest, DistribJoinFromCollectionTest, DistributedQueueTest, DistributedVersionInfoTest, LIROnShardRestartTest, LIRRollingUpdatesTest, MoveReplicaTest, OverseerStatusTest, RecoveryZkTest, RemoteQueryErrorTest, ReplaceNodeNoTargetTest, SSLMigrationTest, SaslZkACLProviderTest, ShardRoutingTest, SharedFSAutoReplicaFailoverTest, SolrCLIZkUtilsTest, SolrCloudExampleTest, SolrXmlInZkTest, TestCloudConsistency, TestCloudDeleteByQuery, TestCloudPseudoReturnFields, TestDeleteCollectionOnDownNodes, TestLeaderElectionWithEmptyReplica, TestLeaderElectionZkExpiry, TestLeaderInitiatedRecoveryThread, TestPullReplica, TestPullReplicaErrorHandling, CollectionReloadTest, CustomCollectionTest, AutoAddReplicasPlanActionTest, ExecutePlanActionTest, MetricTriggerIntegrationTest, ScheduledTriggerTest, SearchRateTriggerIntegrationTest, SystemLogListenerTest, TestSimExtremeIndexing]
[smoker] [junit4] Completed [539/835 (1!)] on J1 in 5.23s, 1 test, 1 failure <<< FAILURES!
{noformat}
> Simulate a 1 bln docs scaling-up scenario
> -----------------------------------------
>
> Key: SOLR-12709
> URL: https://issues.apache.org/jira/browse/SOLR-12709
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
> Reporter: Andrzej Bialecki
> Assignee: Andrzej Bialecki
> Priority: Major
> Fix For: 7.5, master (8.0)
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org